基于TPS的STN模块-Robust Scene Text Recognition with Automatic Rectification

TPS：薄板样条插值(Thin plate splines)

STN：Spatial Transformer Network

RARE：Robust Scene Text Recognition with Automatic Rectification

本文详细解读RARE论文中的第三节，基于TPS的STN模块。

简介

场景文字检测的难点有很多，仿射变换是其中一种，Jaderberg[2]等人提出的STN通过预测仿射变换矩阵的方式对输入图像进行矫正。但是真实场景的不规则文本要复杂的多，可能包括扭曲，弧形排列等情况，这种方式的变换是传统的STN解决不了的，因此作者提出了基于TPS的STN。TPS非常强大的一点在于其可以近似所有和生物有关的形变。流程如下：

localization network: 预测TPS矫正所需要的K个基准点（fiducial point）
Grid Generator：基于基准点进行TPS变换，生成输出Feature Map的采样窗格（Grid）
Sampler：每个Grid执行双线性插值

image.png

localization network

就是一个简单的CNN网络，最后两层使用全连接，最后一层全连接的输出为K*2，表示K个点的(x,y)坐标。最后一层的初始化作者探索了四种初始化方法，最后使用了(a)。其中(b)、(c)初始化性能较差，随机初始化不收敛。将最后一层全连接的data初始化为0，bias初始化为(a)图中的坐标。论文中K取了20。

image.png

class LocalizationNetwork(nn.Module):
    """ Localization Network of RARE, which predicts C' (K x 2) from I (I_width x I_height) """

    def __init__(self, F, I_channel_num):
        super(LocalizationNetwork, self).__init__()
        self.F = F
        self.I_channel_num = I_channel_num
        self.conv = nn.Sequential(
            nn.Conv2d(in_channels=self.I_channel_num, out_channels=64, kernel_size=3, stride=1, padding=1,
                      bias=False), nn.BatchNorm2d(64), nn.ReLU(True),
            nn.MaxPool2d(2, 2),  # batch_size x 64 x I_height/2 x I_width/2
            nn.Conv2d(64, 128, 3, 1, 1, bias=False), nn.BatchNorm2d(128), nn.ReLU(True),
            nn.MaxPool2d(2, 2),  # batch_size x 128 x I_height/4 x I_width/4
            nn.Conv2d(128, 256, 3, 1, 1, bias=False), nn.BatchNorm2d(256), nn.ReLU(True),
            nn.MaxPool2d(2, 2),  # batch_size x 256 x I_height/8 x I_width/8
            nn.Conv2d(256, 512, 3, 1, 1, bias=False), nn.BatchNorm2d(512), nn.ReLU(True),
            nn.AdaptiveAvgPool2d(1)  # batch_size x 512
        )

        self.localization_fc1 = nn.Sequential(nn.Linear(512, 256), nn.ReLU(True))
        self.localization_fc2 = nn.Linear(256, self.F * 2)

        # Init fc2 in LocalizationNetwork
        self.localization_fc2.weight.data.fill_(0)
        """ see RARE paper Fig. 6 (a) """
        ctrl_pts_x = np.linspace(-1.0, 1.0, int(F / 2))
        ctrl_pts_y_top = np.linspace(0.0, -1.0, num=int(F / 2))
        ctrl_pts_y_bottom = np.linspace(1.0, 0.0, num=int(F / 2))
        ctrl_pts_top = np.stack([ctrl_pts_x, ctrl_pts_y_top], axis=1)
        ctrl_pts_bottom = np.stack([ctrl_pts_x, ctrl_pts_y_bottom], axis=1)
        initial_bias = np.concatenate([ctrl_pts_top, ctrl_pts_bottom], axis=0)
        self.localization_fc2.bias.data = torch.from_numpy(initial_bias).float().view(-1)

Grid Generator

以下先不加证明给出TPS的插值函数（https://blog.csdn.net/caoniyadeniniang/article/details/78107057?isappinstalled=0&from=singlemessage）：

已知点集 ${\{X_1,X_2,X_3,X_4,...X_n}\}$ ，其中 $X_i=\{x_i,y_i\}$ ，求插入点的值
$\phi(x,y)=\sum_{j=1}^n A_jd_{j}^2logd_{j}+a+bx+cy \\ 也可以写成矩阵形式：\Delta{C'}T=Z$
其中 $a,b,c$ 均为标量， $d_j$ 表示当前插入的与点集中第 $j$ 个点的欧式距离， $x,y$ 表示待插入点的坐标

创建基准点 $C'$ （base fiducial points），下图中右图的蓝色坐标点，归一化到（-1,1），由_build_C函数创建，也是上文中提到的点集 ${\{X_1,X_2,X_3,X_4,...X_n}\}$ ，是一个常量。localization network最后一层全连接层输出的点即为下图中左图中的绿色的点。

image.png

求解 $T$ ，需要先求得 $\Delta{C'}$ ，再求 $\Delta{C'}$ 的逆矩阵（由_build_inv_delta_C求得），batch_C_prime_with_zeros变量表示 $Z$ 矩阵，batch_T变量表示 $T$ 矩阵

$Z = \left( \begin{matrix} C^T \\ 0^{3\times2} \end{matrix} \right)\\\\ T = \Delta{C'^{-1}}Z\\ \Delta{C'} = \left( \begin{matrix} 1^{K\times1} &C'^T & R \\0 &0 & 1^{K\times1} \\0 &0 & C' \end{matrix} \right) \\ 其中R=d^2logd，d_{ij}表示C'中c'_i和c'_j之间的欧式距离，即点集中X_i和X_j之间的欧式距离\\ 源码中实现的\Delta{C'}=\left( \begin{matrix} 1^{K\times1} &C'^T & R \\0 &0 & C' \\0 &0 & 1^{K\times1} \end{matrix} \right)$

得到 $T$ 矩阵后，我们已经可以将点集 $C$ 映射到点集 $C'$ ，因此可以应用 $T$ 矩阵和点集 $C'$ 将Input Image $I$ 转换为Rectified Image $I'$ 。创建图像坐标 $P$ ，对于 $I'$ 中的每一个点 $p'_{ij}$ 都可以在 $I$ 上找到一个与其对应的 $p_{ij}$ ，使用 $\phi(x,y)=\sum_{j=1}^n A_jd_{j}^2logd_{j}+a+bx+cy$ 即可求得。
- 创建 $P^{h\times w}$ ，函数_build_P
- 计算 $P'^{h\times w\times(K+3)}$ ，函数_build_P_hat
$r'_{i,k}=d^2_{i,k}logd^2_{i,k} \\ p' = [1, x'_i, y'_i, r'_{i,1},r'_{i,2},...,r'_{i,k}]\\ p_i = Tp'_i$

计算 $P$ ，变量batch_P_prime

class GridGenerator(nn.Module):
    """ Grid Generator of RARE, which produces P_prime by multipling T with P """

    def __init__(self, F, I_r_size):
        """ Generate P_hat and inv_delta_C for later """
        super(GridGenerator, self).__init__()
        self.eps = 1e-6
        self.I_r_height, self.I_r_width = I_r_size
        self.F = F
        self.C = self._build_C(self.F)  # F x 2
        self.P = self._build_P(self.I_r_width, self.I_r_height)
        ## for multi-gpu, you need register buffer
        self.register_buffer("inv_delta_C", torch.tensor(self._build_inv_delta_C(self.F, self.C)).float())  # F+3 x F+3
        self.register_buffer("P_hat", torch.tensor(self._build_P_hat(self.F, self.C, self.P)).float())  # n x F+3
        ## for fine-tuning with different image width, you may use below instead of self.register_buffer
        #self.inv_delta_C = torch.tensor(self._build_inv_delta_C(self.F, self.C)).float().cuda()  # F+3 x F+3
        #self.P_hat = torch.tensor(self._build_P_hat(self.F, self.C, self.P)).float().cuda()  # n x F+3

    def _build_C(self, F):
        """ Return coordinates of fiducial points in I_r; C """
        ctrl_pts_x = np.linspace(-1.0, 1.0, int(F / 2))
        ctrl_pts_y_top = -1 * np.ones(int(F / 2))
        ctrl_pts_y_bottom = np.ones(int(F / 2))
        ctrl_pts_top = np.stack([ctrl_pts_x, ctrl_pts_y_top], axis=1)
        ctrl_pts_bottom = np.stack([ctrl_pts_x, ctrl_pts_y_bottom], axis=1)
        C = np.concatenate([ctrl_pts_top, ctrl_pts_bottom], axis=0)
        return C  # F x 2

    def _build_inv_delta_C(self, F, C):
        """ Return inv_delta_C which is needed to calculate T """
        hat_C = np.zeros((F, F), dtype=float)  # F x F
        for i in range(0, F):
            for j in range(i, F):
                r = np.linalg.norm(C[i] - C[j])
                hat_C[i, j] = r
                hat_C[j, i] = r
        np.fill_diagonal(hat_C, 1)
        hat_C = (hat_C ** 2) * np.log(hat_C)
        # print(C.shape, hat_C.shape)
        delta_C = np.concatenate(  # F+3 x F+3
            [
                np.concatenate([np.ones((F, 1)), C, hat_C], axis=1),  # F x F+3
                np.concatenate([np.zeros((2, 3)), np.transpose(C)], axis=1),  # 2 x F+3
                np.concatenate([np.zeros((1, 3)), np.ones((1, F))], axis=1)  # 1 x F+3
            ],
            axis=0
        )
        inv_delta_C = np.linalg.inv(delta_C)
        return inv_delta_C  # F+3 x F+3

    def _build_P(self, I_r_width, I_r_height):
        I_r_grid_x = (np.arange(-I_r_width, I_r_width, 2) + 1.0) / I_r_width  # self.I_r_width
        I_r_grid_y = (np.arange(-I_r_height, I_r_height, 2) + 1.0) / I_r_height  # self.I_r_height
        P = np.stack(  # self.I_r_width x self.I_r_height x 2
            np.meshgrid(I_r_grid_x, I_r_grid_y),
            axis=2
        )
        return P.reshape([-1, 2])  # n (= self.I_r_width x self.I_r_height) x 2

    def _build_P_hat(self, F, C, P):
        n = P.shape[0]  # n (= self.I_r_width x self.I_r_height)
        P_tile = np.tile(np.expand_dims(P, axis=1), (1, F, 1))  # n x 2 -> n x 1 x 2 -> n x F x 2
        C_tile = np.expand_dims(C, axis=0)  # 1 x F x 2
        P_diff = P_tile - C_tile  # n x F x 2
        rbf_norm = np.linalg.norm(P_diff, ord=2, axis=2, keepdims=False)  # n x F
        rbf = np.multiply(np.square(rbf_norm), np.log(rbf_norm + self.eps))  # n x F
        P_hat = np.concatenate([np.ones((n, 1)), P, rbf], axis=1)
        return P_hat  # n x F+3

    def build_P_prime(self, batch_C_prime):
        device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
        """ Generate Grid from batch_C_prime [batch_size x F x 2] """
        batch_size = batch_C_prime.size(0)
        batch_inv_delta_C = self.inv_delta_C.repeat(batch_size, 1, 1)
        batch_P_hat = self.P_hat.repeat(batch_size, 1, 1)
        batch_C_prime_with_zeros = torch.cat((batch_C_prime, torch.zeros(
            batch_size, 3, 2).float().to(device)), dim=1)  # batch_size x F+3 x 2
        batch_T = torch.bmm(batch_inv_delta_C, batch_C_prime_with_zeros)  # batch_size x F+3 x 2
        batch_P_prime = torch.bmm(batch_P_hat, batch_T)  # batch_size x n x 2
        return batch_P_prime  # batch_size x n x 2

Sampler

STN中的反向插值算法，pytorch的实现函数为F.grid_sample

class TPS_SpatialTransformerNetwork(nn.Module):
    """ Rectification Network of RARE, namely TPS based STN """

    def __init__(self, F, I_size, I_r_size, scale=1, I_channel_num=1):
        """ Based on RARE TPS
        input:
            batch_I: Batch Input Image [batch_size x I_channel_num x I_height x I_width]
            I_size : (height, width) of the input image I
            I_r_size : (height, width) of the rectified image I_r
            I_channel_num : the number of channels of the input image I
        output:
            batch_I_r: rectified image [batch_size x I_channel_num x I_r_height x I_r_width]
        """
        super(TPS_SpatialTransformerNetwork, self).__init__()
        self.F = F
        self.I_size = I_size
        self.I_r_size = I_r_size  # = (I_r_height, I_r_width)
        self.I_channel_num = I_channel_num
        self.LocalizationNetwork = LocalizationNetwork(self.F, self.I_channel_num, scale)
        self.GridGenerator = GridGenerator(self.F, self.I_r_size)

    def forward(self, batch_I):
        batch_C_prime = self.LocalizationNetwork(batch_I)  # batch_size x K x 2
        build_P_prime = self.GridGenerator.build_P_prime(batch_C_prime)  # batch_size x n (= I_r_width x I_r_height) x 2
        build_P_prime_reshape = build_P_prime.reshape([build_P_prime.size(0), self.I_r_size[0], self.I_r_size[1], 2])

        if torch.__version__ > "1.2.0":
            batch_I_r = F.grid_sample(batch_I, build_P_prime_reshape, padding_mode='border', align_corners=True)
        else:
            batch_I_r = F.grid_sample(batch_I, build_P_prime_reshape, padding_mode='border')

        return batch_I_r

效果示例

image.png

个人经验

TPS网络可以使用比识别网络低一个数量级的学习率
若识别网络的backbone较小（shuffflenet、mobilenet等），需要使用ImageNet上预训练的模型才能较好收敛。如果使用初始化权重并且TPS和识别网络使用相同的学习率则无法收敛（目前不知道详细原因），使用resnet则没有这种问题。
若使用轻量级网络，可以对TPS网络的层通道进行scale（0.25、0.5），同时减少F的个数
在CCPD上进行车牌识别，加入TPS模块，准确率可以上升10-15个点。