Efficient algorithm for directed text detection based on rotation decoupled bounding box

Network architecture

The proposed rotational decoupled oriented text detection algorithm is designed based on the YOLOv5 (Zhu et al., 2021) architecture, as shown in Fig. 1, the proposed architecture consists of four parts: input layer, feature extraction backbone network, feature fusion module and detection output module. Amongst them, the input layer and feature extraction backbone network remain unchanged from YOLOv5, whereas the feature fusion module and detection output module have been improved.

D-PANet feature fusion network

Target detection in the feature fusion process heavily relies on the location information in low-level features and the semantic information in high-level features. Low-level features have fewer convolution times, resulting in relatively less semantic information, but the localisation information is accurate. By contrast, high-level features are rich in semantic information due to repeated convolutions, but the target localisation is more ambiguous. The D-PANet feature fusion network is used to effectively enhance the low-level feature semantic information and the high-level feature location informationin this article.

The proposed D-PANet model network is based on the idea of DenseNet (Huang et al., 2017) dense connectivity, which adds a parallel up-sampling and down-sampling channel to the multiscale features by using cross-scale connectivity, converting the feature fusion from the original sequential stacking method to a parallel two-branch structure. The structure can effectively enable each scale to acquire more features, enhance image feature fusion, and realise the improvement of semantic information and location information for each scale feature. The specific network of D-PANet is shown in Fig. 2.

Taking the acquisition of F₅ as an example, we retain the 1 * 1 convolutional layer that reduces the number of C₅ feature channels in the original YOLOv5 feature fusion module whilst retaining the down-sampling operation with a stride size of 2 for feature F₄. Then, the second branch is used to down-sample operations with a stride size of 4 for feature F₃. Finally, BottleneckCSP is used to eliminate the overlapping effect caused by the down-sampling process. The acquisition process is shown in Fig. 3, where each scale output F₃, F₄ and F₅ can be expressed as follows: (1)${\displaystyle \left\{\begin{array}{c}{F}_3=X_3\hfill \\ F_4=f\left(X_4,Z_3\right)\hfill \\ F_5=f\left(X_5,Z_4,Z_3^{\prime }\right)\hfill \end{array}\right.}$

where $\left(X_3,X_4,X_5,Z_3,Z_3^{\prime },Z_4\right)$ denotes the output features of each scale feature layer and f denotes the concat + conv operation.

Design and implementation of the rotational decoupling bounding box detection module

Figure 4 shows the output diagram of the rotational decoupled bounding box detection module. The image is passed through the feature extraction backbone network and D-PANet feature fusion module to obtain the final detection feature layer. The feature channel output dimension is 3(5+1), where 3 is the different preset aspect ratio anchor box, 5 indicates the detection output bounding box information, and 1 indicates the text confidence degree. In the training process of this detection module model, firstly, the rotationally decoupled bounding box representation is defined, and then the positive sample matching strategy between the rotationally decoupled bounding box and the horizontal anchor box desings. Secondly the bounding box in the regression stage of the bounding box prediction is encoded. Finally, the loss function is added to complete the model iteration by the BP algorithm.

Rotational decoupling bounding box characterization method

Currently, the two main rotational bounding box characterisation methods for rotational characterization arethe eight-parameter method (x₁, y₁, x₂, y₂, x₃, y₃, x₄, y₄) (Zhu et al., 2015) and the five-parameter method (x, y, w, h, θ) (Ren et al., 2017). Amongst them, the five-parameter method is divided into two types according to the range of angular definition variation, namely the OpenCV definition method θ ∈ [ − π/2, 0) and the long-edge definition method θ ∈ [ − π/2, π/2), and the specific rotational characterisation is shown in Fig. 5.

All of the characterization methods are capable of detecting directed text, but the eight-parameter characterisation method requires significantly more model parameters than the five-parameter characterization method. Both the OpenCV definition method and the long-edge definition method have issues with angular periodicity, thereby a effecting the stability of model training (Yang & Yan, 2020). A new rotationally decoupled bounding box five-parameter characterisation technique, shown in Fig. 6, is suggested in this study.

The rotational decoupling bounding boxes come in two varieties: w ≥ h($\left|\theta \right|\le \pi /4$ with X-axis) and w < h(|θ| < π/4 with Y-axis). The two types of bounding boxes are symmetric and independent of each other. During the model training, the rotational decoupling bounding box is decoupled to the horizontal bounding box HBB_X + θ when w ≥ h and to the horizontal bounding box HBB_Y + θ when w < h. Under this definition θ ∈ [ − π/4, π/4], the decoupled bounding box representation is compatible with the original YOLOv5.

Positive sample matching strategy

In this study, the head module’s positive sample acquisition is based on the algorithmic idea that ‘high-quality anchors are easier to regress to obtain an accurate prediction bounding box’ in YOLOv4, using the positive sample matching strategy based on the maximum IOU threshold, as shown in Fig. 7. Initially, the anchor is matched with the ground truth bounding box to obtain the IOU, and when the IOU is larger than the set threshold (the threshold value is set to 0.7 in this study to ensure the precision of the detection results and avoid the difficulty of convergence of the model training due to the small number of positive samples), it is marked as a positive sample and mapped to the feature layer, and vice versa for the background.

Regression to obtain precise predicted bounding box is simpler when using a positive sample sampling strategy based on IOU thresholding, but determining how to increase the IOU of matching the ground truth bounding box with the anchor box becomes crucial. The OBB bounding box and HBB anchor box matching cannot achieve a better IOU when the conventional five-parameter bounding box characterisation method is used. The parameter calculation of the model is evidently enhanced by introducing the OBB anchor box, However, the problem of introducing the OBB anchor box to improve the IOU can be effectively avoided by decoupling the OBB bounding box into the combination of HBB_X/Y + θ through the rotation decoupling bounding box characterisation method. As shown in Fig. 8, any rotated bounding box is decoupled into a unique horizontal bounding box HBB_X/Y matched with the horizontal anchor box, and the decoupled result θ is used for loss calculation, allowings the model to calculate the angle between the bounding box and the predicted bounding box, effectively avoiding the problem of angle periodicity in the OpenCV definition method and the long-edge definition method.Moreover, the two positive sample matching strategies of introducing a rotating anchor box and a rotating decoupled bounding box are compared. The result shows that both rotating decoupled bounding boxes can match with an HBB anchor box to obtain a higher IOU and improve the accuracy rate of the prediction bounding box.

Bounding box coding

After the positive samples are obtained by the matching strategy, they need to be predicted by the bounding box regression. However, in the implementation of the algorithm, if the obtained positive samples are directly subjected to the bounding box regression, then oscillations in the losses occur due to the absence of any constraints on the parameters of the bounding box, which is not conducive to model training. The parameters of the bounding box are encoded to effectively carry out model training. The encoding principle is shown in Fig. 9, where the model initially restricts the regression prediction bounding box to a certain offset (t_x, t_y, t_w, t_h) of the corresponding anchor box by indexing the grid coordinates (c_x, c_y) and presetting the anchor box width and height (p_w, p_h). Then, the output range is controlled to [ − 1, 1] by using the tanh function according to the angle change interval θ ∈ [ − π/4, π/4] in the bounding box, strictly controlling the model’s angle prediction interval t_θ, and finally obtaining the prediction angle $\left(\hat{x},\hat{y},\hat{w},\hat{h},\hat{\theta }\right)$ by using the angle decoding formula. The encoding of the offset and the decoding of the predicted bounding box are calculated as follows: (2)${\displaystyle \left\{\begin{array}{c}{t}_x=x-c_x\hfill \\ t_y=y-c_y\hfill \\ t_w=log\left(w/p_w\right)\hfill \\ t_h=log\left(h/p_h\right)\hfill \\ t_{\theta }=4\theta /\pi \hfill \end{array}\right.}$ (3)${\displaystyle \left\{\begin{array}{c}\hat{x}=2\sigma \left(t_x\right)-0.5+c_x\hfill \\ \hat{y}=2\sigma \left(t_y\right)-0.5+c_y\hfill \\ \hat{w}=p_w\times 2\sigma {\left(t_w\right)}^2\hfill \\ \hat{h}=p_h\times 2\sigma {\left(t_h\right)}^2\hfill \\ \hat{\theta }=\pi /4\cdot tanh\left(t_{\theta }\right)\hfill \end{array}\right.}$

where is denoted as a Sigmoid function to scale the predicted bounding box offset to between 0 and 1.

Loss function

Since the model introduces the OBB bounding box, an angle regression channel needs to be added to this model’s bounding box prediction regression channel, and the SmoothL1 loss function is used for angle learning. At the same time, as a single-category text prediction model, it discards the category prediction channel and completes text prediction by OBB prediction regression with a confidence level only to improve the model detection efficiency. The above analysis indicates thatthe loss in this study consists of three types: bounding box loss, angle loss and confidence loss; the specific losses are calculated after modification, as follows: (4)${\displaystyle Loos=los{s}_{\text{bbox}}+los{s}_{theta}+los{s}_{conf}}$ (5)${\displaystyle los{s}_{\text{bbox}}={\lambda }_{coord}\sum _{i=0}^{S^2}\sum _{j=0}^B{I}_{ij}^{obj}\left[{\left({\hat{x}}_i^j-x_{\mathrm{i}}^j\right)}^2+{\left({\hat{y}}_i^j-y_i^j\right)}^2\right]+{\lambda }_{coord}\sum _{i=0}^{S^2}\sum _{j=0}^B{I}_{ij}^{obj}{\left(2-{\hat{w}}_i^j\times {\hat{h}}_i^j\right)}^2\times \left[{\left({\hat{w}}_i^j-{\mathrm{w}}_i^j\right)}^2+{\left({\hat{h}}_i^j-h_i^j\right)}^2\right]}$ (6)${\displaystyle los{s}_{theta}={\lambda }_{coord}\sum _{i=0}^{S^2}\sum _{j=0}^B{I}_{ij}^{obj}smoot{h}_{L1}\left({\hat{\theta }}_i^j-{\theta }_i^j\right)}$ (7)${\displaystyle los{s}_{conf}=\sum _{i=0}^{S^2}\sum _{j=0}^B-I_{ij}^{obj}\left[c_i^{j}ln{\hat{\mathrm{c}}}_i^j+\left(\mathrm{1}-c_i^j\right)ln\left(\mathrm{1}-{\hat{\mathrm{c}}}_i^j\right)\right]+{\lambda }_{noobj}\sum _{i=0}^{S^2}\sum _{j=0}^B-I_{ij}^{noobj}\left(c_i^j-{\hat{\mathrm{c}}}_i^j\right)}$

where $I_{ij}^{obj}$ denotes whether the i th anchor box in the j th grid is responsible for the predicted text bounding width position, and the value of which is 1, and 0 otherwise. $\left(x_i^j,y_i^j,w_i^j,h_i^j,{\theta }_i^j,c_i^j\right)$ and $\left({\hat{x}}_i^j,{\hat{y}}_i^j,{\hat{w}}_i^j,{\hat{h}}_i^j,{\hat{\theta }}_i^j,{\hat{c}}_i^j\right)$ denote the real text marker bounding box and the predicted bounding box parameters and the confidence level, respectively; λ_coord and λ_noobj denote the penalty coefficients.

Rotationally decoupled bounding box detection module implementation

Based on the baseline network design of rotationally decoupled bounding box and horizontal anchor box positive sample matching strategy, predicted bounding box regression coding and loss calculation,the rotationally decoupled bounding box detection module can effectively realize autonomous learning and inference,and its specific learning and inference process are shown in Tables 1 and 2, respectively.

Analysis of experimental results

In this section, we first present the experimental data and experimental setup of the proposed method, then analyse the model performance of the proposed method by conducting comparison tests on a public dataset, and finally analyse the effectiveness of different improved methods by ablation experiments on a self-built dataset.

Experimental data and experimental setup

For this experiment, the model was evaluated on ICDAR 2015, MSRA-TD 500 and a self-built dataset to test the model’s performance. In addition, to shorten the experimental data training and its validation time, training and testing were performed on a 64-bit Windows system with the hardware configuration and running environment configuration shown in Table 3. During model training, the model used stochastic gradient descent as the optimiser, the momentum factor was set to 0.9, the weight decay was set to 0.005, the initial weights of the network model were generated by random initialisation, the initial learning rate was 0.001, 16 images were input in each iteration, and the learning rate was multiplied by 0.1 at 100 and 200 times to reduce the learning rate, and the regularisation factor was set to 0.0005 to suppress overfitting. The coefficient of regularisationwas set to 0.0005 to suppress overfitting.A total of 300 iterations were performed, the model was saved every one iteration, and the best weight amongst 300 iterations was finally selected as the detection network model.

Model comparison experiments

To verify the effectiveness of the proposed directed text detection model based on rotation decoupled bounding boxes, it compared and analysed by validating it on the MSRA-TD500 and ICPR2015 datasets with recent years text detection algorithms. The recent years algorithms mainly include CTPN (Tian et al., 2016), RRPN (Ma et al., 2018), R2CNN (Jiang et al., 2017), EAST (Zhou et al., 2017), DBNet (Liao et al., 2020), SegLink (Shi, Bai & Belongie, 2017), TextSnake (Long et al., 2018), PSENet (Wang et al., 2019a) and PANNet (Wang et al., 2019b), ATRR (Wang et al., 2019c), TextFuseNet (Ye et al., 2020). The model’s performance is referenced by four metrics: precision, recall, F₁-Score, and detection speed, which are calculated as follows: (8)${\displaystyle Precision=\frac{TP}{TP+FP}}$ (9)${\displaystyle Rec\mathrm{all}=\frac{TP}{TP+FN}}$ (10)${\displaystyle F1-Score=\frac{2\times Precision\times Recall}{Precision+Recall}}$

where TP is the positive sample whose model prediction is accurate, FP is the negative sample whose model prediction is positive, and FN is the positive sample whose model prediction is inaccurate.

Table 4 shows the performance results of different algorithm models on the MSRA-TD500 dataset, where a segmentation-based DBNet algorithm achieves an accuracy of 91.5 in the dataset detection compared with the comparison algorithm, which achieves the best performance in terms of detection accuracy. However, the proposed rotational decoupled bounding box-based directed text detection algorithm model is second only to the DBNet algorithm in terms of accuracy and achieves better performance results in terms of recall, F₁-Score, and detection speed compared with the other algorithms. Table 5 shows the performance results of different algorithmic models on the ICDAR2015 dataset, in which the frame rate measured by the proposed model is 39.7 fps, and its detection speed achieves the best amongst other algorithms. In addition, the detection precision, recall and F₁-Score of the proposed model are similar to the SOTA methods, effectively verifying that the proposed model is competitive. Figure 10 displays some of the detection outcomes of the proposed arithmetic model used in this article on the MSRA-TD500 and ICDAR2015 datasets. It is clear from the figure that the model is capable of performing directed text detection.

Ablation experiments

The improved YOLOv5 multiscale special fusion sign pyramid D-PANet and multi-angle rotation decoupled bounding box detection modules on self-built datasets are tested and compared under the same deep learning environment and parameter configuration to analyse the effectiveness of different improved methods and to use the proposed methods in an industrial real-world environment.

(1) Effectiveness of D-PANet feature fusion module.

Table 6 demonstrates the effect of whether the model employs the D-PANet feature fusion module on the performance of the detection network. The experimental results show that the use of the D-PANet feature fusion module leads to a 1.2%, 0.3% and 0.5% improvement in the model detection accuracy, recall, and F-value, respectively, compared with the original model performance, and the detection speed only decreases by 0.9 fps, effectively demonstratingthat the D-PANet dual-branch feature information fusion structure can further improve the performance of the shallow and deep feature information fusion and highlight the target information, thereby improving text detection precision with only a small increase in parameter calculation.

(2) Effectiveness of rotating decoupled bounding box detection modules

Four experimental control groups are designed in this experiment to investigate the effect of different bounding box detection modules on the performance of the detection network. The first group of experiments uses the original YOLOv5 detection module; the second group of experiments uses the OpenCV bounding box definition method detection module; the third group of experiments uses the long-edge definition method detection module; the fourth group uses the decoupled bounding box detection module. The performance of the detection models is verified by experiments, as shown in Table 7.

The comparison betweenthe first group and the fourth groupindicates that the detection speed of the decoupled bounding box detection module is reduced by 3.6 fps, but the detection speed is improved by 6.7 and 6.5 fps compared with the second group and the third group, respectively. This finding shows that the decoupled bounding box detection module can effectively avoid the introduction of rotating anchor boxes to reduce the model parameter calculation and improve the model detection efficiency. Meanwhile, the model detection precision, recall and F₁-score, are 95.6%, 90.1%, and 82.8%, respectively, using the decoupled bounding box detection module. The detection performance is improved compared with the second and third groups, effectively showingthat the decoupled bounding box detection module can solve the problem of angular periodicity in the regression bounding box and improve the model detection performance. The results of the corresponding detection model are shown in Fig. 11.