Calibration

The overview of IMU calibration is illustrated in Fig. 3. Referring to DIP (Huang et al., 2018), the calibration aims to transform the IMU readings to a common body-centric coordinate frame. Denoting the acceleration data relative to the sensor coordinates as G^s and the orientation data relative to the global inertial coordinates as G^I. To convert the inertial measurement value to the SMPL global coordinate frame G^M, it is necessary to calibrate the coordinate sensor frame to the global inertial coordinate frame, represented as a rotation R^SI: G^S → G^I. From the global inertial coordinate frame to the 3D human body model (SMPL) global coordinate frame, it is defined as a rotation R^IM:G^I → G^M. Supposing G^I and G^M are unchanging during the calibration period, the process is expressed as: (1)${\displaystyle G^M=R^{MI}{G}^I,}$ where R^MI = inv(R^IM) represents the transformation of the global inertial coordinate to the SMPL global coordinate.

The bone offset under the SMPL global coordinate is necessary as the sensor can be placed in any direction. At the beginning of each sequence, each subject stands in a known T-pose with bone orientation $O_0^{BI}$ and acceleration $A_0^{BI}$. The orientation offset $R_{offset}^{BM}$ and acceleration offset $A_{offset}^{BM}$ are assumed to be unchanged during the calibration period, the per-sensor bone offsets are computed by: (2)${\displaystyle R_{offset}^{BM}=inv\left(O_0^{IB}\right)R^{IM},}$ (3)${\displaystyle A_{offset}^{BM}=A_0^{BI}{R}^{IM}.}$

Based on bone offset, the orientation and acceleration data of each sensor is first transformed to the global inertial coordinates, then converted to the SMPL global coordinates. This process is expressed as: (4)${\displaystyle O_t^{BM}=O_t^{BI}{R}^{IM}{R}_{offset}^{BM},}$ (5)${\displaystyle A_t^{BM}=\left(A_t^{BI}{R}^{IM}\right)-A_{offset}^{BM},}$ where O ∈ ℝ^3×3 represents the orientation and A ∈ ℝ³ indicates the acceleration, $O_t^{BM}$ and $A_t^{BM}$ represent the direction and acceleration of t-th frame in the SMPL global coordinates, respectively. $O_t^{BI}$ and $A_t^{BI}$ represent the direction and acceleration of t-th frame in the global inertial coordinates, respectively. Then the normalized $O_t^{BM}$ and $A_t^{BM}$ are used as input for the improved IMU feature extractor.

Data normalization

After converting the IMU readings to the SMPL coordinate frame, denote the normalized direction and acceleration as $\overline{O}$ and $\overline{A}$, respectively. The leaf joint inertial measurements are aligned with the root joint: (6)${\displaystyle A_{leaf}={\overline{O}}_{leaf}^{-1}\left({\overline{A}}_{leaf}-{\overline{A}}_{root}\right),}$ (7)${\displaystyle O_{leaf}={\overline{O}}_{root}^{-1}{\overline{O}}_{leaf}.}$

The standard root is defined as: (8)${\displaystyle A_{root}={\overline{O}}_{root}^{-1}{\overline{A}}_{root},}$ (9)${\displaystyle O_{root}={\overline{O}}_{root}.}$

Dual-stream feature extractors

To balance the efficiency, scalability, and long-term modeling ability to extract representative information from the temporal inertial data, the bidirectional recurrent neural network (Schuster & Paliwal, 1997) with long short-term memory (LSTM) (Hochreiter & Schmidhuber, 1997) unit is employed as the IMU feature extractor. A detailed implementation of the IMU feature extractor is illustrated in Fig. 5. The input data is a temporal sequence containing the normalized positions of the leaf joint relative to the root. The output of the IMU feature extractor is a set of the corresponding frame-level feature vector. Specifically, a linear embedding layer is first used to encode the normalized position data, then a two-layers LSTM is employed to aggregate the temporal cues in the context. The intermedia feature of five leaf joint positions relative to the root joint is obtained through a fully connected layer. Inspired by the position encoding scheme proposed by Zhao, Wang & Tian (2022) in image-based keypoint detection, we add an extra temporal encoder containing a two-layer bidirectional gated recurrent unit (GRU) to refine the temporal feature with position embeddings and generate T ×2048 features as output.

For the image-based input branch, we refer to the previous single image-based method (Kanazawa et al., 2018; Kolotouros et al., 2019) and employ the ResNet-50 (He et al., 2016a; He et al., 2016b) based backbone with pre-trained weights by the ImageNet as an image encoder to extract the visual feature f_img ∈ ℝ²⁰⁴⁸. Then the extracted visual feature is processed by two branches. As the cubes in blue color shown in Fig. 6, the first one is used as the input to fuse with the IMU features in the residual model-attention network, while the second one is connected to the final regression module to obtain the pose parameter θ and body parameter β.

Residual model-attention network

To closely couple distinct dual-modal features and eliminate ambiguity, we design a residual model-attention network to refine the image feature guided by the IMU feature. As shown in Fig. 6, the fusion network combines a model-attention network and a residual connection network. The input of model attention is a distinct dual-modal feature, and the output is a fused feature aggregated from visual image and IMUs. Referring to the residual connection introduced by He et al. (2016a) and He et al. (2016b), the fused feature $f\left(t\right)$ is generated by concatenating the model-attention feature $f_{attention}\left(t\right)$ with the original IMU feature $f_{imu}\left(t\right)$ and image feature $f_{img}\left(t\right)$ in turn. The detailed process is expressed as: (10)${\displaystyle f_{attention}^{\prime }\left(t\right)=Max\left(f_{attention}\left(t\right),f_{imu}\left(t\right)\right),}$ (11)${\displaystyle f\left(t\right)=LayerNorm\left(f_{attention}^{\prime }\left(t\right)+f_{img}\left(t\right)\right),}$ where $Max\left(\cdot \right)$ indicates the max pooling operation, $LayerNorm\left(\cdot \right)$ indicates the layer normalization inspired by previous work (Yu et al., 2019).

The final regression module takes in the fused feature $f\left(t\right)$ and generates a set containing pose parameter θ and body parameter β. Following the iterative regression module introduced in Kolotouros et al. (2019), the regression network is initialized with an average posture, then the output pose θ is regressed with the input fused feature $f\left(t\right)$ through an iterative process.

Loss function definition

As SMPL is a differentiable digital model that can generate 3D keypoints by a linear regression $\hat{X}\left(\theta \right)=WM\left(\theta ,\beta \right)$, where $\hat{X}\left(\theta \right)\in {\mathbb{R}}^{J\times 3}$ and J represents the number of joints. The body mesh $\hat{V}\left(\theta \right)=M\left(\theta ,\beta \right)$ can be also generated by linear blend skining, where $\hat{V}\in {\mathbb{R}}^{6890\times 3}$, indicating that the mesh consists of 6,890 points. Therefore, the constraints of the endpoint regression module include the mesh, 3D keypoints, pose parameters θ, and shape parameters β. The loss function for training the end regression network is defined as: (12)${\displaystyle L_c=\tau L_{vertices}+\phi L_{keypo\mathrm{int}s}+w{L}_{SMPL},}$ where L_vertices denotes the body mesh loss, L_keypoints denotes the 3D keypoints loss, and L_SMPL denotes the pose parameter loss. τ, φ and w denote the corresponding weights of three loss items. These losses are denoted as follows: (13)${\displaystyle L_{vertices}=\left|\right|V-\hat{V}|{|}_1,}$ (14)${\displaystyle L_{keypo\mathrm{int}s}=\left|\right|X-\hat{X}|{|}_2,}$ (15)${\displaystyle L_{SMPL}=\left|\right|\beta -\hat{\beta }|{|}_2+||\theta -\hat{\theta }|{|}_2,}$ where θ ∈ ℝ^24×3×3, and V, X, β and θ represent ground truths of $\hat{V},\hat{X},\hat{\beta }$ and $\hat{\theta }$, respectively.

To better utilize the relative joint position information calculated by IMU, we add an intermedia constraint for the IMU feature extractor during the end-to-end training. Denote the leaf joints containing five relative-root joints in a sequence as $J_{leaf}=\left[J_{r\text{\_}ankle},\dots ,J_{l\text{\_}wrist}\right]$. The loss item for the IMU feature extractor is defined as: (16)${\displaystyle L_{imu}=\sum _t^T\left|\right|J_{leaf}-J_{leaf}^{GT}\left|\right|.}$

The overall objective is defined as: (17)${\displaystyle L=L_c+\lambda L_{imu},}$ where λ is the corresponding weight to control the relative importance.

Data preparation

The Total Capture dataset is currently the only dataset that provides images, IMU data, and ground-truth annotations for 3D keypoints. Its data contain images captured by eight cameras and 13 IMUs (containing rotation information and acceleration). This dataset contains four actions performed by five subjects, with each action repeated three times. The experiment used images taken by the eighth camera and values from six IMUs as input to the proposed network. Experiments on this dataset follow the protocol proposed by previous work (Zhang et al., 2020), the movements roaming 1, 2, and 3; walking 1 and 3; acting 1 and 2; and freestyle 1 and 2 of the first three subjects is used as the training set. The actions of the five subjects (walking 2, acting 3, and freestyle 3) are used for evaluation. Following the IMU sensor setting of DIP (Huang et al., 2018), the selected six IMUs’ rotation and acceleration are provided by the authors of DIP.

As the Human3.6M dataset does not provide IMU data, we follow the evaluation setup of DIP (Huang et al., 2018) to validate the general applicability of this fusion strategy. The synthetic IMU training data, i.e., the accelerations and orientations of six keypoints indicated by the orange color in Fig. 2, is obtained by creating virtual sensors on the SMPL mesh surface. The global rotation is generated by local rotation through forward kinematics (FK), where the global rotation is the orientation of the IMU. The acceleration of the virtual sensor is computed through finite differences. More precisely, assuming the position of a virtual IMU is p_t for time t, and the time interval between the two frames is d_t, the fitted acceleration can be calculated by (18)${\displaystyle a_t=\frac{p_{t-1}+p_{t+1}-2\times p_t}{d_t^2},}$ (19)${\displaystyle R_{IMU}=FK\left(\theta ,\beta \right).}$

The Human3.6M dataset contains seven subjects with 15 sets of actions, with each set repeated twice. The SMPL parameters were taken from the same source as the previous HybrIK (Li et al., 2021) and L2LMeshNet (Moon & Lee, 2020). Following HMR and SPIN (Kanazawa et al., 2018; Kolotouros et al., 2019), this article used the first five subjects for training and the last two subjects for evaluation.

Evaluation metrics

For a fair comparison, we employ the most common metrics to report the experimental results. The evaluation metrics are defined as follows:

1. SIP error measures the global rotation error of the upper arm and thigh in degrees.

2. The angle error indicates the global rotation error of each body joint in degrees.

3. MPJPE represents the average distance error of the node in mm, aligned with the root joint.

4. PA-MPJPE indicates all nodes’ average Euclidean distance error after further rigid alignment in mm.

5. PVE means mesh error; the unit is cm, representing the average error of the vertex in the mesh model. It is a better representation of deformation error.

Analysis of feature fusion

This article aims to investigate the efficacy of fusing IMU and image in improving the accuracy of 3D pose estimation. To this end, the Total Capture dataset containing ground-truth annotations is employed as the benchmark. This ablation study was conducted by giving four different inputs, designated as “IMU only”, “Image only”, “IMU+Image (TP)”, and “IMU+Image” employing four evaluation metrics including reconstructed joint, body mesh, and angle errors. The detailed experimental settings with four different inputs are as follows:

1. “IMU only” follows Transpose as the baseline, which uses the LSTM network to obtain leaf keypoints and all relative-root keypoints, connect IMU measurements, and generate IMU features. The regressor is then used to generate pose parameters.

2. “Image only” follows HMR as the baseline, which uses the single-view image feature as input to the regression network to generate pose parameters.

3. For the “IMU+Image (TP)”, involved IMU feature and image feature follow the above “IMU only” and “Image only” baselines. “TP” represents the IMU feature extractor following TransPose (Yi, Zhou & Xu, 2021). The fusion scheme defined in Eqs. (10) and (11) is employed to generate pose parameters.

4. In contrast to “IMU+Image (TP)”, “IMU+Image” used the LSTM network to obtain the leaf keypoints of the relative root as the intermediate result, connecting the IMU measurement value and generating a combined IMU feature. The fusion scheme follows ”IMU+Image(TP)”.

The detailed results are listed in Table 1. Overall, the experimental results demonstrate that the fusion of IMU and image features, referred to as “IMU+Image (TP)” and “IMU+Image”, significantly improves pose estimation accuracy. The superiority on all metrics indicates that the proposed network can effectively couple the different features and reduce the 3D pose and shape error. In addition, the results of “IMU+Image” are further better than those of the “IMU+Image (TP)”, which that indicates the improved IMU feature extractor is more robust to the sensor disturbance on aggregating the temporal information. “IMU+Image (TP)” depends on a leaf joint’s target position, and the position information is reverse derived for n parent joints on the bone chain where it is located to determine the position of the entire bone chain. However, this derivation process is sensitive to the incorrect feature, resulting in an inaccurate pose position.

Influence of the IMUs on individual joints

Specifically, to explore the effects of IMUs on improving the accuracy of each joint position, the method was evaluated by reconstructing 14 key joints defined by Leeds Sports Pose (LSP, Johnson & Everingham, 2010) with taking in IMU data as input or not. Following the previous work (Kanazawa et al., 2018; Kolotouros et al., 2019), the 14 LSP joints were regressed from the body mesh by a pre-trained regressor. The main result is shown in Table 2. The auxiliary use of IMUs results in a significant reduction in position errors of six key joints. In particular, the error of wrist and ankle joints reduce by 29.5% and 34.3%, respectively. The end joints, such as the ankle and wrist, are more sensitive to visual occlusion due to the flexible movement of the human body and the limitations of the camera view. This experimental result indicates that the number of IMUs could be further reduced to correct the estimated error of image-based motion capture for specific applications like virtual action sports games.

Analysis of IMU drift with noise

As the IMU is sensitive to the magnetic field and noisy during the measurement, resulting in prediction jitter and drift. To address this issue, this section mainly discusses the effectiveness of fusing IMU and image features in overcoming measurement noise. Inspired by TransPose (Yi, Zhou & Xu, 2021), this experiment was conducted on the Human3.6M dataset to simulate the IMU noise quantitatively, as the ideal data could be generated by inferring the positions and rotations of a virtual IMU on the corresponding vertices of the SMPL mesh. Note that we add Gaussian noise to the virtual IMU data during the test phase. Three different standard deviations i.e., 0.12, 0.2, 0.3 were randomly added to the 30% of the raw IMU data to simulate the noisy measurements in real-world settings. The experimental results are presented in Table 3. The ablation study compares two types of input data, which are denoted as “IMU only” with four standard deviations and “IMU+Image” with four standard deviations, respectively.

Compared to noiseless IMU data, the predicted pose error with using noisy IMU data would increase by about 27.7%. However, the pose error predicted by fusing images and noisy IMU data only increased by about 15.1%. By adding the same IMU noise, the joint accuracy of the fused method improves by about 45.4% compared to that of the single-source IMU data. The superiority in improving joint accuracy even with noisy inference demonstrates that the proposed fusion network is qualified for reducing the ambiguity and drift of conventional approaches.

Comparison with sparse IMUs-based methods

To demonstrate the superiority of our improved IMU feature extractor in encoding the temporal information, the experiment employs the Total Capture dataset as the benchmark and compares the results with the recent methods based on sparse IMUs. The quantitative comparison between state-of-the-art methods and ours on the Total Capture dataset is shown in Table 4. The metrics of SIP, Angle error, MPJPE, and PVE are used to evaluate the predicted human pose and body mesh. For the Angle error, the AAGC-LSTM network proposed by Puchert & Ropinski (2021) performs better. However, the model presented in this article is superior in the other metrics. The results outperform those of the latest method PIP (Yi et al., 2022). Especially, the per-vertex error (PVE) is reduced by 14.4%, indicating the effectiveness of the proposed approach.

Figure 7 presents some visualization results of our method and Transpose (Yi, Zhou & Xu, 2021) on the Total Capture dataset. The SMPL model is projected to the same view for better visual effects of human posture. The cases shown in the top two rows are selected from the outputs of Transpose (Yi, Zhou & Xu, 2021) with relatively high scores. The case shown in the third row is a representative ambiguous frame, and the bottom row shows a case chosen from the reconstruction with a lower score by our proposed model. As shown in the first- and second-line examples in Fig. 7, while each model is correct in rough structure, the model is much better at reconstructing arm and leg position details, such as step span, leg bend, and visual arm posture. The model performed better when reconstructing the pose with both legs bent, as shown in the third example of squatting below; the reconstruction is visually closer to the ground truth than the Transpose (Yi, Zhou & Xu, 2021). In the fourth example, each model fails to produce the correct pose, but our model still achieves much better results in the detail of the leg position.

Comparison with image-based methods

Following previous work (SPIN and HMR), this experiment evaluated the reconstructed 14 LSP joints on the Human3.6M dataset compared with previous methods based on single-view images. Table 5 lists the detailed results using the rigid alignment average position error per joint (PA-MPJPE) and position error per joint (MPJPE). As shown in Table 5, this method could reduce the joint error (MPJPE) by 13.5% compared to the SOTA method.

To further analyze the influence of partial occlusions on pose estimation, the detailed results on various human postures with performing 15 movements are listed in Table 6. Note that the results of several methods using multi-view images are also given for a fair comparison. This method outperforms better for most actions, particularly the refine actions such as “Eating” and “Photo” which are significantly corrected benefits from the auxiliary virtual IMU information. The accuracy for general actions like “Sitting Down” and “Walking Dog” is inferior to that of single-view methods. The main reason is the proposed method does not consider the global translation, resulting in the estimated position error.

Comparisons with fusion-based methods

As mentioned above, the work on fusing sparse IMUs and a single-view camera is hardly addressed in the community. To provide a reference of our approach to other methods, the experiment follows the protocol of train and test partitions introduced by Trumble et al. (2017) on the Total Capture dataset. The comparison is regardless of the same number of IMUs or multiple viewpoints and employs the MPJPE in mm as the evaluation metric. The detailed results of these similar works are listed in Table 7. The employed numbers of camera viewpoints and IMUs are also given for clarity. As shown in Table 7, the presented method is superior to the learning-based approach introduced by Trumble et al. (2017), which uses all eight cameras and fuse IMU data with the probabilistic visual hull (PVH). The proposed method also outperforms that of Malleson et al. (2017), which report the MPJPE in 62 mm using eight cameras and all 13 IMUs. The performance of the proposed method is inferior to that of Von Marcard et al. (2018) by 2.8 mm. The main reason is that the method of Von Marcard et al. (2018) is based on the video to model the time information and additional association of 2D and 3D pose. However, the proposed method does not need the extra 2D pose detection compared to Von Marcard et al. (2018), which would make this approach take less time for online applications.