Design of sports training information analysis system based on a multi-target visual model under sensor-scale spatial transformation

Target detection

Target detection is a foundational pursuit within computer vision, garnering substantial attention in research endeavors. Before the advent of deep neural networks, target detection algorithms primarily relied on extracting manual image features for subsequent feature matching. Esteemed manual features such as SIFT and SURF yielded commendable results. Many initiatives sought to tailor diverse manual features for target detection, aligning with the specific characteristics of the target; examples include V-J detection (Viola & Jones, 2001), HOG detection (Dalal & Triggs, 2005) algorithms, and the like. In recent years, propelled by the evolution of deep learning, target detection algorithms founded on convolutional neural networks have ascended to prominence. Concurrently, the introduction of large-scale universal target detection datasets has furnished indispensable data support for convolutional neural network-based target detection algorithms. Many efforts have delved into diverse convolutional neural network structures to realize effective target detection. The YOLO series (Redmon & Farhadi, 2017) perceives detection as a regression problem, seamlessly predicting target categories and locations end-to-end through convolutional neural networks. The Faster-RCNN series (Girshick, 2015) enhances detection accuracy by bifurcating target detection into two phases: initially extracting foreground frames potentially containing the target, followed by detailed classification and regression. Lin et al. (2017) optimize the detection performance of densely distributed small targets by introducing focal loss. Xiao & Jae Lee (2018) elevate video detection performance by incorporating a sequence of spatiotemporal memory stores. Zhu et al. (0000) harness optical flow to capture motion information in videos, seamlessly integrating it with feature maps in CNN to achieve end-to-end video detection. Zhang & Wang (2016) delve into stability in video detection, mitigating detection frame jitter by incorporating a tracking strategy. Yin & Liu (2017), grounded in the concept of multi-cascading, decompose target detection into multiple classification and regression problems, achieving precise face detection. Liu et al. (2019) discard the conventional window detection approach, opting to directly predict the body center and dimensions, thereby realizing pedestrian detection.

Posture estimation for sports movement

As evident from the current state of target detection research, the continuous evolution of deep learning technology has led to the maturation of single-target monitoring research. However, a notable gap exists in sports and training as the research landscape predominantly addresses single-target tracking scenarios. Hence, there is a pressing need to convene seminars addressing the intricacies of multi-target tracking. The primary mission of Multiple Object Tracking (MOT) revolves around identifying the positions of all targets of interest within an image sequence in each frame while preserving their identity information. These targets include people, vehicles, animals, and other moving objects. Notably, pedestrian tracking has been the most explored domain, finding applications in intelligent surveillance and autonomous driving. Yet, the challenges in sports scenarios, marked by dynamic and unpredictable player movements, render multi-target tracking more complex than pedestrian tracking. Yu et al. (2003) introduced a ball detection method grounded in trajectory analysis. This method initiates by detecting multiple candidate targets during the detection phase, followed by trajectory analysis to ascertain the ball’s movement trajectory. Liang et al. (2005) proposed a comprehensive ball-tracking algorithm that amalgamates detection and tracking. The algorithm tracks the ball via Kalman filtering by detecting candidate objects and computing the optimal path using the Viterbi algorithm. It determines whether the tracker has lost the ball by evaluating the tracked region and repeating the detection and tracking process. Acknowledging the challenges of adapting single-camera-based tracking to prolonged ball occlusion, scholars have proffered various multi-camera tracking schemes (Yoon, Song & Jeon, 2018). Ren et al. (2009) proposed a method based on multiple stationary cameras for football games, leveraging the characteristics of the ball’s size, appearance, and moving speed. The approach utilizes the Kalman filtering method to derive the ball trajectory, subsequently fusing trajectory segments from different viewpoints into the 3D trajectory of the ball. To address occlusion challenges within a multi-camera framework, Wang et al. (2014) presented a ball-tracking method that considers the dynamic relationship between players and the ball. Ivankovic et al. (2012) employed the AdaBoost algorithm for player detection in football and basketball scenarios, utilizing hand-designed directional gradient histogram HOG features and Haar features. While effective for pedestrian detection, these methods exhibit limitations in player detection tasks due to the large posture variations and significant occlusions in sports videos. It is attributed to the interference and reduced effectiveness of traditional manual features in facing such challenges. In recent years, the ascendancy of deep learning has steered the focus toward player detection methods based on convolutional neural networks. For instance, Lu et al. (2018) proposed a player detection method addressing issues such as substantial changes in the player body and scale induced by camera movement, applicable across diverse and complex sports scenes, including basketball, football, and ice hockey.

The preceding research underscores that integrating deep learning and diverse target detection methodologies facilitates a nuanced exploration of sports competition or training information within sports training and video data analysis. This approach enables in-depth performance analysis of individual players, imparting significant value to enhancing athletes’ skill levels. Notably, using deep convolution, multimetric vision, and other advanced methods in target detection proves highly effective for addressing the challenges associated with multi-target detection tasks. The amalgamation of multidimensional data becomes imperative given the importance of positional status, along with players’ nuanced body shapes and clothing information during sports training. In this context, realizing multi-source data fusion is a pivotal strategy for achieving enhanced target detection capabilities.

KSP-based multi-target tracking

The k-shortest-path algorithm (Maidana et al., 2024), a venerable concept in graph theory, is fundamentally employed to identify the first K paths in a graph representing the shortest routes between specified start and end points. The applicability of the KSP algorithm to addressing the multi-objective tracking problem stems from its capacity to model the situation as a linear programming challenge. That facilitates the construction of a network flow graph based on temporal and spatial parameters, subtly transforming the problem into a graph optimization endeavor to identify the shortest paths within the spatiotemporal graph.

The KSP tracking algorithm holds a distinctive advantage in its simplicity of inputs, encompassing solely the positional coordinate points of tracked targets and their corresponding features. Effectively transforming the multi-target tracking problem into an integer programming challenge, the algorithm operates under the assumption that the current video stream under consideration for tracking comprises T frames. Each frame, in turn, detects K candidate targets, represented as nodes in the algorithm. The inter-frame relationships between candidate targets are established as edges based on distance constraints, culminating in the depiction of a one-dimensional node model graph, as illustrated in Fig. 1.

Each node variable ‘n’ signifies the number of target objects presently situated at that node, and the weight of each edge indicates the flow of ‘f’ targets through that particular connection. Building upon this foundation and taking into account the two-dimensional nature of the camera data, we can posit that the probability distribution for the player’s position on the pitch plane is as follows: (1)${\displaystyle {\mathrm{\rho }}_{\mathrm{i}}^{\mathrm{t}}=\mathrm{P}\left({\mathrm{X}}_{\mathrm{i}}^{\mathrm{t}}=1\mid {\mathrm{I}}^{\mathrm{t}}\right)}$

where ${\mathrm{\rho }}_{\mathrm{i}}^{\mathrm{t}}$ denotes the position of the course plane, i is the probability that a player is present at frame t, The likelihood that a player is present at frame 1, ${\mathrm{X}}_{\mathrm{i}}^{\mathrm{t}}$ is a random variable, and ${\mathrm{X}}_{\mathrm{i}}^{\mathrm{t}}=1$ is a random variable, then it means that the player exists at t player exists at the time; otherwise it does not exist, and I^t is the probability that a player exists at the time of frame t frame corresponds to the court plan.

In the realm of multi-target tracking, the overarching objective resides in the identification of numerous tracking trajectories. Facilitating these trajectories’ interconnection becomes paramount, optimizing the likelihood of capturing players’ movements. Building upon this premise and leveraging the extant probabilities, the challenge of multi-objective tracking transforms into a linear programming paradigm. (2)${\displaystyle max\sum _{\mathrm{i}=1}^{\mathrm{T}}\sum _{\mathrm{i}=1}^{\mathrm{G}}log\left(\frac{{\mathrm{\rho }}_{\mathrm{i}}^{\mathrm{t}}}{1-{\mathrm{\rho }}_{\mathrm{i}}^{\mathrm{t}}}\right)\cdot \sum _{\mathrm{j}\in \mathbb{N}\left(\mathrm{i}\right)}{\mathrm{f}}_{\mathrm{i},\mathrm{j}}^{\mathrm{I}}}$

Its constraint is: (3)${\displaystyle \forall \mathrm{t},\mathrm{i},{\mathrm{f}}_{\mathrm{i},\mathrm{j}\in \mathbb{N}\left(\mathrm{i}\right)}^{\mathrm{t}}\ge 0}$ (4)${\displaystyle \forall \mathrm{t},\mathrm{i},\sum _{\mathrm{j}\in \mathbb{N}\left(\mathrm{i}\right)}{\mathrm{f}}_{\mathrm{i},\mathrm{j}}^{\mathrm{t}}\le 1}$ (5)${\displaystyle \forall \mathrm{t},\mathrm{i},\sum _{\mathrm{j}\in \mathbb{N}\left(\mathrm{i}\right)}{\mathrm{f}}_{\mathrm{i},\mathrm{j}}^{\mathrm{I}}-\sum _{\mathrm{k}\in \mathbb{N}\left(\mathrm{j}\right)}{\mathrm{f}}_{\mathrm{j},\mathrm{k}}^{\mathrm{t}-1}\le 0}$

where T denotes the number of frames in a batch, G denotes the number of nodes in the network flow graph, and ℕ(i) represents node I, the neighborhood of a node. J is one of the neighbors of i, one of the neighboring nodes of the node, the ${\mathrm{f}}_{\mathrm{i},\mathrm{j}}^{\mathrm{I}}$ is one of the neighboring nodes of t at the moment of flow from node i to the adjacent node j at the time of flow from the node to the neighboring node, and ${\mathrm{\rho }}_{\mathrm{i}}^{\mathrm{t}}$ denotes the position of the pitch plane i at the node in the first t The probability that a player is present at the frame. In addition, the three constraints indicate that the number of network flows is all non-negative, and the number of network flows between two positions is 0 or 1. In this solution process, since the constraint matrix has the property of unimodularity, it can be solved by the above conditions using the relaxation linear programming method and converge to the integer solution quickly.

The decision to employ the KSP methodology and AlphaPose network is underscored by their exceptional accuracy, robustness, scalability, and state-of-the-art performance in human pose estimation tasks. Their combination offers a reliable solution capable of handling diverse environmental conditions, large datasets, and real-time processing requirements. Furthermore, their open-source availability fosters accessibility and collaboration within the research community, driving further innovation. This selection ensures the success of specific projects and contributes to advancing human-centric technologies. Overall, the choice of the KSP methodology and AlphaPose network reflects a commitment to excellence, innovation, and impact in human pose estimation and beyond.

Alphabet-based multiplayer attitude estimation

Human pose estimation methodologies can be dichotomously categorized based on the sequencing of their procedural steps: top-down and bottom-up recognition methods. The top-down approach involves initially pinpointing the precise localization of each individual in the image, followed by the subsequent recognition of crucial points for individual persons. This method’s critical key point recognition accuracy is often contingent upon the precision of human body localization. Conversely, the bottom-up methodology first detects all vital points in the image, then groups and aggregates these key points to delineate each individual. Although this method exhibits a faster recognition speed, its accuracy tends to be lower than the aforementioned top-down approach. The AlphaPose human pose estimation model employed in this study (Halleck et al., 2023) adheres to the fundamental framework of the Regional Multi-Person Pose Estimation (RMPE) proposed by the research team. Notably, this model iteration excels in the real-time estimation of multi-person poses while ensuring commendable recognition outcomes. The comprehensive flow of the entire framework is elucidated in Fig. 2.

AlphaPose harnesses convolutional neural networks (CNNs) to primarily extract image features, employing multiple layers of convolution, activation functions (e.g., ReLU), and pooling layers. This process maps the image into the feature space, facilitating the discernment of human vital points. Each key point’s location is determined by generating a corresponding heat map, with localization further refined using local maxima. Initially, the input image undergoes scrutiny by the human detector to delineate the target region of the human body. The YOLOv5 module, renowned for its current popularity and accuracy, is employed within the human detector module. Subsequently, the body region frame is directed SSTN into the SSTN, including the STN, SPPE, and SDTN. Finally, all candidate poses are subjected to the P-NMS (Parametric Pose Non-Maximum-Suppression) module. A traversal of predicted candidate poses is necessary within the P-NMS module (Sabo et al., 2023). Poses with confidence scores surpassing a predefined threshold are selected, and the pose with the highest confidence is designated as the reference pose. Subsequently, the elimination criteria are applied, involving the calculation of distances between the remaining poses and the reference pose. Alphapose provides significant advantages in target tracking through attitude estimation, mainly reflected in its high accuracy and strong robustness. By accurately identifying human key points, Alphapose can accurately track multiple targets in complex environments, maintaining stability even in occlusion or rapid motion. In addition, its sensitive ability to capture dynamic postures provides rich information for understanding target behavior, which is particularly important in application scenarios such as sports training and monitoring. The integrated Alphapose tracking system can enhance the coherence of target recognition and tracking through attitude information, improving overall tracking performance. Therefore, Alphapose enhances the accuracy of multi-target tracking and provides a powerful tool for in-depth analysis of target behavior.

When distances fall below the threshold, the corresponding poses are eliminated. This process iterates until all poses conform to the threshold criteria (Zwölfer et al., 2023). The elimination criterion is defined by Eq. (6): (6)${\displaystyle \mathrm{f}\left({\mathrm{P}}_{\mathrm{i}},{\mathrm{P}}_{\mathrm{j}}\mid \mathrm{\lambda },\mathrm{\eta }\right)=1\left(\mathrm{d}\left({\mathrm{P}}_{\mathrm{i}},{\mathrm{P}}_{\mathrm{j}}\mid \mathrm{\lambda },\mathrm{\lambda }\right)\le \mathrm{\eta }\right)}$

where d() calculates the spatial distance of the two poses and the weighted distance of the pose distances. If the distance is less than or equal to the threshold η, then it represents the reference gesture P_i and the pose P_j are too similar, and the gestures need to be eliminated. The formula for calculating the distance which is shown in Eq. (7): (7)${\displaystyle \mathrm{d}\left({\mathrm{P}}_{\mathrm{i}},{\mathrm{P}}_{\mathrm{j}}\mid \mathrm{\lambda }\right)={\mathrm{K}}_{\text{Sim}}\left({\mathrm{P}}_{\mathrm{i}},{\mathrm{P}}_{\mathrm{j}}\mid {\mathrm{\sigma }}_1\right)+\mathrm{\lambda }{\mathrm{H}}_{\text{Sim}}\left({\mathrm{P}}_{\mathrm{i}},{\mathrm{P}}_{\mathrm{j}}\mid {\mathrm{\sigma }}_2\right)}$

where λ={ σ1, σ2, λ}, K_Sim represents the stance distance, H_sim represents the spatial distance. Assuming that Bi is the predicted bounding box of Pi, we can compute the pose distance by using Eq. (8): (8)${\displaystyle {\mathrm{K}}_{\mathrm{im}}\left({\mathrm{P}}_{\mathrm{i}},{\mathrm{P}}_{\mathrm{j}}\mid {\mathrm{\sigma }}_1\right)=\left\{\begin{array}{c}\sum _{\mathrm{n}}tanh\frac{{\mathrm{c}}_{\mathrm{i}}^{\mathrm{n}}}{{\mathrm{\sigma }}_1}tanh\frac{{\mathrm{c}}_{\mathrm{j}}^{\mathrm{n}}}{{\mathrm{\sigma }}_1}\text{if}{\mathrm{k}}_{\mathrm{j}}^{\mathrm{n}}\mathrm{ }\text{is within}\mathrm{ B}\left({\mathrm{k}}_{\mathrm{i}}^{\mathrm{n}}\right)\hfill \\ 0\text{otherwise}\hfill \end{array}\right.}$

The spatial distance is used to calculate the spatial similarity of the distance of the corresponding feature information between two poses, which can be related by Eq. (9). (9)${\displaystyle {\mathrm{H}}_{\mathrm{Sim}}\left({\mathrm{P}}_{\mathrm{i}},{\mathrm{P}}_{\mathrm{j}}\mid {\mathrm{\sigma }}_2\right)=\sum _{\mathrm{n}}exp\left[-\frac{{\left({\mathrm{k}}_{\mathrm{i}}^{\mathrm{n}}-{\mathrm{k}}_{\mathrm{j}}^{\mathrm{n}}\right)}^2}{{\mathrm{\sigma }}_2}\right].}$

Object tracking framework fusing KSP and position estimation information

Upon concluding the extraction of attitude information and shortest path details in the multi-target tracking process, we operationalize the multi-target tracking objective within the sports training regimen. That is achieved by utilizing both the shortest path information and position constraint data.

The framework begins by utilizing YOLO to delineate target divisions accurately. Subsequently, AlphaPose is employed to compute the joints of all individuals in the diagram. Joint matching is conducted, resulting in a diverse array of human body postures derived from the joints. Athlete posture matching is executed based on the athlete’s detection frame, eliminating extraneous human body postures. This process ultimately yields the posture estimation of athletes within the diagram.

Adopting a bottom-up approach, which returns to the joints of the entire graph before assembling different human postures, mitigates the issue of missed detections stemming from people aggregation. This approach also obviates the need for repetitive calls to the single-person posture detector.

Simultaneously, optimal path selection is achieved through the KSP method based on recognized targets. Features such as jersey and player number are extracted using the ReID network (Zhong et al., 2018), known for its prowess in discerning the same individual across diverse scenes. The ReID network’s probability is computed through the Probability of Model (POM) method, prioritizing consideration of model classification error probability over mere accuracy. Inputs for the KSP method are derived from these features, followed by subsequent shortest path computation and heatmap generation. Finally, the generated shortest path information and bitmap details are fused at the decision level, creating new feature vectors that effectively fulfill the multi-target tracking objective.

The convergence process seamlessly integrates data from the CNN and the KSP method before incorporating alpha-derived data. Initially, the CNN extracts features such as jersey and player number, while the KSP method facilitates optimal path selection based on recognized targets. These features converge as inputs for subsequent shortest-path computation, ensuring paths encapsulate spatial relationships and target-oriented trajectories. Following this convergence, Alphapose-derived data is seamlessly integrated. AlphaPose network computes the joints of all individuals, providing crucial pose information. This data enriches generated paths with context for human movements (Fleuret et al., 2007). By integrating Alphapose-derived pose information with CNN and KSP-established paths, the convergence process achieves a comprehensive understanding of the ethene, accounting for spatial and kinematic aspects of human activity. This transition ensures a holistic approach to multi-target tracking, leveraging each component’s strengths to enhance accuracy and reliability. The smooth integration of CNN, KSP, and Alphapose data streamlines the convergence process, improving understanding and application across various domains, from surveillance to sports analytics.

Integrating pose information with shortest path computation early in the framework development establishes a vital connection between these components, enhancing overall performance. This integration ensures paths are optimal and contextually relevant to human postures, improving tracking accuracy. Considering the spatial constraints imposed by poses, the system generates paths aligning closely with actual movements, reducing false positives and erroneous paths.

Experiment setup

Upon the completion of the model construction, as depicted in Fig. 3, the ensuing phase involves validating the model across diverse datasets. This study focuses on data sourced from three distinct sports for comprehensive data analysis. The datasets originate from videos captured during sports events and institutions associated with sports training. Specific details about these datasets are elucidated in Table 1.

Upon finalizing the selection and construction of pertinent data, our objective shifts toward assessing and comparing the model’s efficacy. For this purpose, we opt for commonly employed evaluation metrics in multi-target tracking. In this article, our evaluation hinges on multiple object tracking accuracy (MOTA) and global multiple object tracking accuracy (GMOTA), renowned metrics for gauging the accuracy and effectiveness of algorithms in detecting and tracking multiple objects within video sequences. The computation of MOTA is delineated by Eq. (10): (10)${\displaystyle \text{MOTA}=1-\frac{\sum _{\mathrm{t}}\left({\mathrm{c}}_1\cdot \mathrm{f}{\mathrm{n}}_{\mathrm{t}}+{\mathrm{c}}_2\cdot \mathrm{f}{\mathrm{p}}_{\mathrm{t}}+{\mathrm{c}}_3\cdot \mathrm{ids}{\mathrm{w}}_{\mathrm{t}}\right)}{\sum _{\mathrm{t}}{\mathrm{g}}_{\mathrm{t}}}}$

fpt, and where fn_t, fp_t, idsw_t are the number of t the number of missed detections, the number of false detections, and the number of identity exchanges in each trajectory at a time, respectively, and g_t denotes the true value, and c₁, c₂, c₃ is a constant. To better evaluate the impact of the number of identity switches on tracking performance, Shitrit et al. (2013) introduced the Global Identity Switches (gidsw) metric instead of Identity Switches (idsw) to measure the number of identity switches (De Vleeschouwer et al., 2008). (11)${\displaystyle \text{GMOTA}=1-\frac{\sum _{\mathrm{t}}\left({\mathrm{c}}_1\cdot \mathrm{f}{\mathrm{n}}_{\mathrm{t}}+{\mathrm{c}}_2\cdot \mathrm{f}{\mathrm{p}}_{\mathrm{t}}+{\mathrm{c}}_3\cdot \text{gids}{\mathrm{w}}_{\mathrm{t}}\right)}{\sum _{\mathrm{t}}{\mathrm{g}}_{\mathrm{t}}}.}$

To effectively illustrate the efficacy of KSP shortest path information and Alphapose information based on KSP in multi-target visual model training, we conducted target analysis using data solely under each modality. Building upon this, we selected two prominent methods, T-MCNF (Ibrahim et al., 2016) and MOT-CE (Shitrit et al., 2013), for comparative assessment. The T-MCNF algorithm conceptualizes the multi-target tracking problem as a multi-network streaming issue, leveraging the KSP-based methodology. This approach models the situation as a trajectory-based, multi-network flow problem, and To achieve this, the KSP algorithm (Ghedia, Vithalani & Kothari, 2017) is executed twice. The first iteration generates multiple trajectory segments, while the second iteration segregates these segments into distinct groups based on their appearance characteristics. Subsequently, these trajectory segments are treated as nodes to construct multiple network flow graphs, and the KSP algorithm is once again applied to derive the final trajectories. On the other hand, MOT-CE is primarily designed to track rigid body targets in three-dimensional space. This method relies on geometric position information rather than appearance characteristics for tracking.

Experiment result and analysis

After finalizing the dataset selection and specifying the comparison methods and metrics, we proceeded with the experimental phase. The experimental settings for this article are outlined in Table 2:

On this basis, we carried out the calculation of the indicators under the three datasets, and the results in APIDIS are shown in Fig. 4.

In Fig. 4, it is evident that the proposed method attains a MOTA of 0.81 and a GMOTA of 0.64 under its evaluation indices. While not surpassing the 0.9 tracking threshold, these results are deemed acceptable, particularly in addressing the challenges posed by complex environments and multiplayer scenarios. Notably, the overall performance surpasses hybrid multi-target tracking methods at this stage. Figure 5 and 6 illustrate the test results under the volleyball and soccer datasets, respectively.

In Figs. 5 and 6, it is apparent that the model, leveraging KSP shortest path optimization and Pose information convergence as proposed, attains optimal results under both the volleyball and soccer datasets. Specifically, the MOTA scores are 0.83 and 0.85 for the two datasets. These results outperformed the method without feature fusion and the current-stage T-MCNF and MOT-CE methods.

We computed the average metrics across the five methods using three datasets to compare model performance comprehensively. The summarized results are presented in Fig. 7.

The average results depicted in Fig. 7 demonstrate that the approach employed in this article consistently outperforms the other methods across all three datasets; this substantiates that incorporating the feature fusion method can significantly enhance the model’s performance.

The ablation experiment

Upon concluding the evaluation of model metrics across diverse datasets, we proceeded with further ablation experiments to assess the model’s performance. Considering the potential impact of color variations in application scenarios, this article scrutinizes the results under different color information settings. The outcomes of these experiments are illustrated in Fig. 8.

Figure 8 shows that the model’s MOTA and GMOTA metrics improve when different proportions of color information are provided. Leveraging color information augmentation proves beneficial, mainly due to the distinctive color variations in the attire worn by various teams during the game. This augmentation significantly enhances the trajectory-tracking performance of the model.

Furthermore, this article compares the tracking performance between the single and convergence models under varying training iterations. The results of this comparison are presented in Fig. 9.

Figure 9 shows that the MOTA metrics notably improve after feature fusion across all three datasets. The performance is notably superior to the KSP method alone, and the approach is based solely on attitude information. Additionally, when considering the overall data distribution of the model, the fusion of information enhances robustness, leading to more excellent stability in overall performance. The MOTA results showcase consistent improvement across different numbers of training iterations.