Analysis of the mixed teaching of college physical education based on the health big data and blockchain technology

Human pose estimation method based on Transformer

Recently, Transformer (Zhao et al., 2022) andits variants (Xianjun, 2022) have been used by researchers for human pose estimation. For example, TransPose  (Yang et al., 2021) uses the attention layer of Transformer to implicitly reveal the dependence relationship between key points, which provides an explanation for the model to infer the global spatial relationship among layers. Inspired by ViT (Xianjun, 2022) (Vision Transformer) model, TokenPose (Li et al., 2021) explicitly models key points as markers to learn the constraint relationship between visual information and key points from images. Both methods require a large number of Transformer encoders, but do not consider low-resolution global semantic features. HRFormer (Yuan et al., 2021) (High Resolution Transformer) uses multi-resolution architecture design and local window self-attention to achieve high-resolution feature representation, which has the characteristics of low memory and low computational cost. This method needs to up-sample the low-resolution features, which can lose the spatial semantic information. The proposed method will further employ the low-resolution global semantic features and make interactive fusion through cross-attention to ensure that the spatial semantic information is not lost.

Cross-attention

Cross-attention is first used to connect the encoder and decoder in Transformer (Zhao et al., 2022). The cross-attention network (Hou et al., 2019) (CAN) uses cross attention to model the association semantics of class features and query features in the small-sample classification to highlight the target object, which is conducive to subsequent matching. CrossViT  (Chen, Fan & Panda, 2021) uses two different branches to process image tags of different sizes in image classification, and then uses cross attention to fuse class tags. Cross attention has also been used in multi-modal tasks (Lu, Zhou & Ye, 2019; Rodrigues et al., 2022). In this article, we apply the cross-attention for the interaction between different multi-resolution markers to improve the identifiability of low-resolution markers.

Multi-scale representation tokens in Transformer based on cross attention

The high-resolution Transformer encoder module is shown in Fig. 2. The input of Transformer encoder (Li et al., 2021) is composed of concatenated highest resolution visual markers and high-resolution keypoint markers. In order to obtain visual markers, the feature map output by the deep convolutional network needs to be transformed into markers. The high-resolution feature map X^l ∈ ℝ^C×H×W is divided into feature blocks of the same size p_h × p_w. Each feature block is then flattened into a 1-dimensional vector, and the embedding vector $x_v^l$ of d-dimension is obtained by linear transformation. In order to obtain fine-grained features, the low-resolution feature map X^s ∈ ℝ^{C′×H′×W′} is also divided into smaller blocks of size $p_h^{\prime }\times p_w^{\prime }$, which are finally transformed into embedding vectors $x_v^s$. The process of feature map transformation into tags is as follows:

(1)${\displaystyle X^l\in {\mathbb{R}}^{C\times H\times W}\stackrel{P}{\to }{x}_v^l\in {\mathbb{R}}^{\frac{H\times W}{p_h\times p_w}\times d}}$ (2)${\displaystyle X^s\in {\mathbb{R}}^{C^{\prime }\times H^{\prime }\times W^{\prime }}\stackrel{P}{\to }{x}_v^s\in {\mathbb{R}}^{\frac{H^{\prime }\times W^{\prime }}{p_h^{\prime }\times p_w^{\prime }}\times d}}$

where H > H′ and W > W′.C and C′ respectively represent the channels of the corresponding feature maps. P includes segmentation, flattening and linear projection operations. In this article, the keypoint markers that are spliced with high-resolution visual tokens that are called high-resolution keypoint markers and the keypoint markers are not randomly initialized, but the predictive heat map of each keypoint is obtained by deep convolutional network and regressor. Then it is projected into the keypoint marker space.

Intermediate estimates of keypoint markers

In this article, deep convolutional networks are used to predict the heat map of key points, which is then projected into the marker space. The detailed calculation steps are as follows: Firstly, the feature map H × W with the number C of channels is transformed into N tensor heat maps by 1 ×1 convolution operation. Then, it is transformed into a 1-dimensional vector and mapped to a fixed dimension. CH × WN

The process of predicting key point labeling z ∈ ℝ^N×C can be formalized as the following formula:

(10)${\displaystyle y=C\left(x\right)}$ (11)${\displaystyle z=MLP\left(F\left(y\right)\right)}$

where x ∈ ℝ^C×H×W represents the feature map output by convolutional neural network, y ∈ ℝ^N×H×W represents the heat map after 1 × 1 convolutional transformation, C represents the convolution operation of 1 × 1, MLP represents the multi-layer perceptron function and F represents the dimensionality reduction operation.

Dataset and implement details

The experimental platform of this article is Ubantu20.04.LTS, programming environment pytorch1.8.1 and Cuda11.1, which are equipped with two Quadro RTX8000 and RTX 3090. All experiments are performed on MS COCO datasets which contains more than 200K images and more than 250K human instances with 17 types of key points. The model is trained only on the COCO train2017 dataset without additional training data and is tested simultaneously on the val2017 dataset and the test-dev2017 dataset, which contained 57K, 150K and 5K samples, respectively.

Object keypoint similarity (OKS) is the evaluation index adopted by keypoint detection task on COCO2017 dataset and the formula is as follows: (12)${\displaystyle OKS=\frac{\sum _{i}epx\left(-d_i^2/2s^2{k}_i^2\right)\sigma \left(v_i>0\right)}{\sum _i\sigma \left(v_i>0\right)}}$

where $d_i^2$ represents the Euclidean distance between the estimated value and the true value of the key point, s denotes the target scale, k_i denotes a constant to control the decay of each class of keypoints and a visibility marker v_i represents the true value. Average accuracy (AP) records the average of all values for a given threshold $T\in \left[0.5:0.05:0.95\right]$, for example the calculated AP⁷⁵ represents the performance of T = 0.75. When the OKS is greater than this threshold, the position of the key point is detected correctly, otherwise it will be considered to have missed detection and false detection. AP^M and AP^L represent the accuracy of model detection of key points in the medium (32² < pixel <96²) and large (pixel >96²) target areas, respectively.

Intermediate estimation

On the basis of extracting high-resolution features from the backbone network HRNet-W48-3stage of the model, CTHPose initially estimates the distribution of key points in the label space. As shown in Table 1, although the model parameters have been increased, the scores of AP and AR have been increased to some extent. As shown in Fig. 3, with the increase of a certain training period, the convergence speed of the two models in the early training is also very different, but the Loss gap is gradually narrowed in the end because the expression ability of the model is limited. This indicates that the key point labeling estimated by the convolutional network provides support for the subsequent Transformer architecture model learning and alleviates the problem of slow Transformer convergence speed.

Results and comparison

Table 2 records the results of this and other state of the art methods. (i) HRNet-w32 (Sun et al., 2019) provides both pre-trained and untrained versions, while the proposed CTHPose is pre-trained in the convolutional network part and untrained in the Transformer part. Compared with the pre-trained HRNet-w32, the average accuracy (AP) and average recall (AR) of CTHPose are increased by 1.4% and 1.1%, respectively, with a slight increase in the computation (GFLOPs) and a slight decrease in the model parameters. Compared with HRNet-w48, CTHPose also achieves better results. Because CTHPose reduces the up-sampling fusion process of low-resolution features, it can interact low-resolution features with high-resolution features in the label space, which improves the low-resolution identifiability. Moreover, the subsequent cross-fusion can eliminate the inaccuracy of up-sampling fusion. (ii) Compared with HRFormer-B (Yuan et al., 2021), CTHPose has a slightly lower score in AP⁵⁰, but it has a slight improvement in overall performance. HRFormer is a Transformer with window self-attention as the main module for extracting deep features of high-resolution network, while CTHPose adopts the cross-attention mechanism to complete the interactive fusion of deep features, which greatly reduces the parameters used and the computational burden. (iii) Compared with TokenPose-L/D6 (Li et al., 2021), CTHPose has the same number of Transformer encoders and achieves better experimental results under the condition of only a small amount of computation (↑6.6%). Compared with TokenPose-L/D24, although some indicators are relatively lower, CTHPose achieves a comparable performance on the entire, while the computational load is reduced by 11.8%. TokenPose-L/D24 uses a 24-layer Transformer encoder, which brings a huge computational burden to the model, while CTHPose uses only six layers. Because the time complexity of the cross-attention mechanism is linear, the inference cost of CTHPose is greatly reduced.

Table 3 records the performance of CTHPose and other methods on the test set. Compared with HRNet, the average accuracy (AP) and average recall (AR) of CTHPose are 1% and 0.7% higher than those of HRNET with fewer parameters and less computational effort, although the score of AP⁵⁰ is slightly different. CTHPose performs better than other methods with Transformer architecture, such as TransPose-H-A6 (Yang et al., 2021) and TokenPose-L/D24 (Li et al., 2021). This is because the proposed CTHPose makes full use of low-resolution global semantic information. Thanks to the linear time complexity of cross-attention, CTHPose can achieve results comparable to TokenPose-L/D24 with less computation. Since small images will be enlarged and fixed to a specific size before input into the model, the image will become extremely blurred, which brings challenges to the model key point location. Compared with TokenPose-L/D24, CTHPose also achieves the same result in AP^M index. It shows that the performance of CTHPose-s2 is comparable to that of TokenPose-L/D24, the most advanced method for small target areas. According to the CTHPose, we can realize online supervision of college sports teaching, improve the big data platform for college students’ health and improve their physical quality.