GaitTriViT and GaitVViT: Transformer-based methods emphasizing spatial or temporal aspects in gait recognition

Spatial feature extraction

In gait recognition research, the introduction of deep convolutional neural networks was pioneered by Wu et al. (2017), they studied an approach to gait-based human identification via similarity learning by deep CNNs, aiming to recognize the most discriminative changes of gait patterns which suggest the change of human identity with a pretty small group of labeled multi-view human walking videos.

Subsequently, GaitSet, proposed by Chao et al. (2018) adopted the strategy of dividing feature maps into strips from prior person re-identification research, enhancing the description of the human body. It has been adopted by many following researchers ever since; GaitPart introduced by Fan et al. (2020) pushes forward the part-based concept further, presenting a part-dependent approach, they argued that the part-based schemas applied in gait recognition should be part-dependent rather than part-independent, because despite there are significant differences among human body parts in terms of appearance and moving patterns in the gait cycle, it is highly possible that different parts of human body share the common attributes, e.g., color and texture. Thus, the parameters are designed part-dependent in FConv (focal convolution) layers to generate the fine-grained spatio-temporal representations; GaitGL developed by Lin, Zhang & Yu (2020), Lin et al. (2022) elaborated the disadvantage of extraction from either global appearances or local regions of humans only. They argued the representations based on global information often neglect the details of the gait frame, while local region-based descriptors cannot capture the relations among neighboring regions, thus reducing their discriminativeness. Thus, they effectively combined global visual features and local region details, demonstrating the necessity to address both aspects simultaneously; SMPLGait by Zheng et al. (2022) aims to explore dense 3D representations for gait recognition in the wild. Leveraged the human body mesh to acquire three-dimensional geometric information, they proposed a novel framework to explore the 3D skinned multi-person linear (SMPL) model of the human body for gait recognition; MetaGait designed by Dou et al. (2023) argued that there are still conflicts between the limited binary silhouette and numerous covariates with diverse scales. Their model can learn an omni-sample adaptive representation by injected meta-knowledge in a calibration network of the attention mechanism, which could guide the model to perceive sample-specific properties; also (Fan et al., 2022), in their code repository OpenGait, drew insights from previous state-of-the-art methods and introduced GaitBase, which achieved excellent results. These studies often stack deeper convolutional layers or complex architecture to capture fine-grained, more robust, and discriminative features, to meet the various challenges of gait recognition tasks.

Temporal feature aggregation

The temporal modeling has consistently remained a significant focus in gait recognition tasks due to the inherent periodicity of walking patterns in the time dimension, i.e., gaits are repeating loops. Presently, there are three popular directions in existing research: 3D convolutional neural network (3DCNN)-based, Set-based, and LSTM-based approaches.

Among 3DCNN-based methods, Wolf, Babaee & Rigoll (2016) and Tran et al. (2015) directly employ 3DCNN to extract spatio-temporal features from sequential data. They indicated that 3D Convolutional Networks are more suitable compared to 2D and a homogeneous architecture with small [3, 3, 3] convolution kernels in all layers is among the best performing architectures for 3D Convolutional Networks. However, this approach often encounters training difficulties and yields suboptimal performance; set-based methods view frames within a cycle as an unordered set since humans can easily identify a subject from a shuffled gait sequence. Furthermore, due to the short duration of each gait cycle, long-range dependencies and duplicate gait cycles are considered redundant. Take GaitSet, for example Chao et al. (2018). In contrast to prior gait recognition methods which utilize the frames of either a gait template or a gait sequence, they argued that the temporal information is hard to preserve in the template, while the sequence keeps extra unnecessary sequential constraints and thus has low flexibility. So, they present a novel perspective regarding gait as a set consisting of independent frames. Their method is immune to permutations of frames and can naturally integrate frames from different videos under different scenarios. These set-based methods typically study spatial features frame by frame and then perform temporal aggregation at the set level. On the other hand, LSTM-based methods like GaitNet by Zhang et al. (2019) argue that for each video frame, the current feature only contains the walking pose of the person in a specific instance, which can share similarity with another specific instance of a very different person. Therefore, modeling its change is critical. That is where temporal modeling architectures like the recurrent neural network or long short-term memory (LSTM) work best. They use a three-layer LSTM network to extract ordered sequence features. These LSTM-based methods are capable of capturing features between consecutive frames, often yielding slightly better performance. However, they lack efficiency and robustness to noise; therefore, many researchers still prefer set-based approaches.

Attempts with transformer

Multiple works have tried to tackle the gait recognition task by introducing the Vision Transformers (ViTs) (Dosovitskiy et al., 2020). Since the ViTs are more compact in contrast to a multi-layer CNNs when they need to achieve similar performance, and ViTs are full of potential in the computer vision field. For example, Gait-ViT by Mogan et al. (2022) emphasized the lack of attention mechanism in Convolutional Neural Networks despite their good performance in image recognition tasks. The attention mechanism encodes information in the image patches, which facilitates the model to learn the substantial features in the specific regions. Thus, this work employs the Vision Transformer (ViT) integrated attention mechanism naturally. However, they used the gait energy image (GEI) to model the time dimension by averaging the images over the gait cycle; Pinić, Suanj & Lenac (2022) proposed a self-supervised learning (SSL) approach to pre-train the feature extractor, which is a Vision transformer architecture using the DINO model to automatically learn useful gait features (Caron et al., 2021; Cui & Kang, 2022) proposed GaitTransformer. They used a multiple-temporal-scale transformer (MTST), which consists of multiple transformer encoders with multi-scale position embedding, to model various long-term temporal information of the sequence. Furthermore, e.g., Yang et al. (2023) and Zhu et al. (2023) also explore the vision transformer in gait recognition. However, the transformer-based methods have not outperformed CNN-based methods on the popular testing benchmarks and other challenging in-the-wild gait datasets (Fan et al., 2023).

Introduction

Having studied the previous works, two transformer-based Gait Recognition methods are proposed: GaitTriViT and GaitVViT. GaitTriViT integrates the strengths of ViT and incorporates excellent ideas from previous works and recent advancements in related fields. It places emphasis on both global and local regions, considering both temporally and spatially. Furthermore, several modifications are also made in this work, which differs from the common gait recognition frameworks. GaitVViT enhances the temporal modeling ability of a common framework leveraging a Video Vision Transformer (VViT), which works as a novel temporal pooling (TP) module.

Common framework

Recent studies indicate a common framework in various gait recognition tasks (Fan et al., 2023), as shown in Fig. 1. This framework abstracts complex structures into multiple modules, omitting internal details. The backbone maps the input gait sequence to features, typically used to extract frame-level spatial information. The TP module then aggregates feature maps along the time dimension, with operations e.g. max pooling, RNNs (Fan et al., 2020; Iwama et al., 2013; Tran et al., 2021).

Subsequently, the horizontal pooling (HP) module divides the feature map into several different parts in the horizontal direction, in line with the part-dependent concept introduced by Fan et al. (2020) and processes them independently. The Head may include several fully connected layers to obtain predicted labels, and it may also have a batch normalization neck (BNNeck) to map the features to different spaces before calculating the loss (Luo et al., 2020). Finally, both triplet loss and cross-entropy loss are used to optimize the model simultaneously (De Boer et al., 2005; Hermans, Beyer & Leibe, 2017; Hoffer & Ailon, 2015; Rubinstein & Kroese, 2004).

GaitTriViT

Pipeline

The common framework of gait recognition often apply the temporal pooling (TP) module and horizontal pooling (HP) module serially in model, which leads to the TP module may inevitably lose some low-rank features affecting the model performance, our method treats the original serial TP and HP modules as two separate and parallel branches, providing more potential to preserve refined patterns in different dimensions, as shown in Fig. 2. The overall structure can be divided into several modules, including the backbone, local part spatial branch, global temporal branch, BNNeck head, and optimizer (Luo et al., 2020). Aligned silhouettes are fed into the Transformer-based backbone first, after the feature extraction, the feature maps go separately into two parallel branches. The local part spatial branch works to extract the spatial features focused on different local parts, and the global temporal branch works to model the high-level spatio-temporal features. Those local and global features are delivered to classification heads with BNNeck to generate the discriminative final feature representations and predicted labels (Luo et al., 2020), which will be used to calculate the losses for model optimization. Equations (1), (2), (3), (4) below briefly describe the workflow of model GaitTriViT in Fig. 2.

(1) $$\begin{array}{c}{F}_{bone}^i=Vi{T}_{bone}\left(\sum _{j=0}^n\left[em{b}_{pos}^j;em{b}_{case}^j;Em{b}^j\left(F^i\right)\right]\right)\end{array}$$

(2) $$\begin{array}{c}{F}_{local}^{it}=Hea{d}_{local}\left(Vi{T}_{local}\left(Se{g}^t\left(F_{bone}^i\right)\right)\right)\end{array}$$

(3) $$\begin{array}{c}{F}_{global}^i=Hea{d}_{global}\left(Attention\left(Vi{T}_{gobal}\left(F_{bone}^i\right)\right)\right)\end{array}$$

(4) $$\begin{array}{c}Outpu{t}_{GaitTriViT}^{F^i}=L\left(F_{global}^i,\sum _{t=1}^{parts}{F}_{local}^{it}\right)\end{array}$$where $Em{b}^j$ is the j-th embedding after patch cutting and flatten, along with position and case embeddings, $Vi{T}_{bone}$ is the Backbone ViT block, $Se{g}^t$ is the t-th part of Patch Shuffle and Part Segment, $Vi{T}_{local}$ represents the ViT block in local branch, $Hea{d}_{local}$ is Classification Head for local branch, similarly, $Vi{T}_{global}$ and $Hea{d}_{global}$ are counter parts in global branch, here $Attention$ is the Temporal Global Attention layer, finally features pass through function $L$ to get final loss.

Backbone

In this work, a Vision Transformer block is used to build the backbone to extract frame-level spatial features (Dosovitskiy et al., 2020). The original gait silhouette is in the form of a frame sequence $V_i=\left\{F_0,F_1,\dots ,F_t\right\}$, where each frame, after data rearrangement and pre-processing, is in the form of $F_j\in {\mathbb{R}}^{H\times W\times C}$, with H, W, and C representing the height, width, and channels of the frame image, respectively. Each frame is divided into multiple patches of the same size as the original article does, i.e., $F_j=\left\{P_0,P_1,\dots ,P_n\right\}$ (Dosovitskiy et al., 2020). However, drawing inspiration from works by He et al. (2021) and Wang et al. (2022), this work also adopts their patch embedding strategy of allowing patches to overlap with each other. This approach helps the model to focus on local information while strengthening the connections between adjacent patches and reducing feature loss at the patch edges, which meets the need of this task to focus on body parts while not ignoring the constraints among each body part.

When cutting images into patches, different from the traditional method, the overlapping strategy is adopted for a more robust performance as Fig. 3 shows.

(5) $$\begin{array}{c}N={\displaystyle \frac{H+d-s}{s}\times {\displaystyle \frac{W+d-s}{s}}}\end{array}.$$

In Eq. (5), N is the number of divided patches, d is the patch size, and s is the stride length. After the patches are generated, we need to flatten them into tensors of 1-D dimensions using linear projection $\ell$. Moreover, a learnable class token $P_{cls}^j$ is inserted at the head position to represent the overall features of this frame.

(6) $$\begin{array}{c}{F}_j=\left[P_{cls}^j;\ell \left(P_0^j\right);\ell \left(P_1^j\right);\dots ;\ell \left(P_N^j\right)\right]\end{array}.$$

(7) $$\begin{array}{c}{E}_j=F_j+{\lambda }_1{E}_{pos}+{\lambda }_2{E}_{angle}+{\lambda }_3{E}_{case.}\end{array}$$

Following the Vision Transformer original article (Dosovitskiy et al., 2020), A learnable position embedding $E_{pos}\in {\mathbb{R}}^{N+1\times D}$ is added to represent the spatial position of each patch. Furthermore, due to the challenges posed by cross-view and different walking statuses in appearance-based gait recognition tasks, we manually incorporate information that represents different subject appearances and various camera angles into the patch embedding. Many studies have demonstrated the effectiveness of this operation, e.g., research by He et al. (2021) and Alsehaim & Breckon (2022), indicating that these lightweight learnable embeddings perform well tackling cross-view and cross-status tasks. For example, the current frame sequence is selected from a video where a subject is captured by a camera at the front while carrying a bag, which means the camera angle is 0 and the walking status is bag carrying. Similar to position embedding, we introduce case embedding $E_{case}\in {\mathbb{R}}^{\left\{c\times D\right\}}$ and angle embedding $E_{angle}\in {\mathbb{R}}^{a\times D}$, c is the total number of existing walking situations, a stands for the total number of different camera angles. Then, we add these four altogether in proportions denoted by ${\lambda }_1$, ${\lambda }_2$, and ${\lambda }_3$, where we generate the final patch embedding $E_j$.

Local part spatial branch

For each frame sequence representing a unique subject ID with a unique camera angle and walking status, only several frames are selected in one batch, which is regarded as a frame bundle. For each frame bundle B that has undergone processing by the backbone, it now exists as follows:

(8) $$\begin{array}{c}B=\left[F_0\left\{P_{cls}^0,P_0^0,\dots ,P_N^0\right\};\dots ;F_T\left\{P_{cls}^T,P_0^T,\dots ,P_N^T\right\}\right]\end{array}.$$

The bundle is sent to two branches, one of which is the local part spatial branch that will be discussed in this section. It corresponds to the HP module in the gait recognition common framework and is used to extract spatial features at the frameset level within the bundle. Chao et al. (2018) and their proposed method GaitSet suggest that gait recognition does not require long-term dependencies and that treating gait frames as a set can improve model robustness. As our method has two separate branches to process the same feature representations in parallel, each branch can go further on its own specialized work and has no worries about affecting another branch. In the local part branch, the work is all about spatial local features. Therefore, we merge patches belonging to different frames but at the same position within the bundle as group $G_n$, where $n\in \left\{1,2,\dots ,N\right\}$, these groups form a new bundle $\hat{B}$ as follows:

(9) $$\begin{array}{c}\hat{B}=\left[G_{cls}\left\{P_{cls}^{F_0},\dots ,P_{cls}^{F_T}\right\};G_0\left\{P_0^{F_0},\dots ,P_0^{F_T}\right\};\dots ;G_N\left\{P_1^{F_0},\dots ,P_N^{F_T}\right\}\right].\end{array}$$

Then, we shift and shuffle these patch groups (excluding class token group $G_{cls}$ as it will always appear in the head position) using TCSS proposed by Alsehaim & Breckon (2022) to make the model more robust to appearance noise (see left part of Fig. 4), which has been indicated by Zhang et al. (2018) and Huang et al. (2021). Briefly, the first few patch groups (in the order of position, in this work the number is 2) are cut off and shifted to the end of patch groups, then, these patch groups are shuffled, the patches in latter half are inserted into the gaps between each patch in former half, as shown in the middle part of Fig. 4.

In the local part branch, as the left part of the figure shows, each feature map needs to undergo shift operation by a given amount, followed by a shuffle operation shown in the middle of the figure, then each feature map will be divided into four strips from top to bottom for separate treatment.

Subsequently, the frame bundle that has undergone shuffling is sent to part-dependent feature extraction (Fan et al., 2020). We divide image patches within each frame into multiple horizontal strips independently based on morphological characteristics (from top to bottom) as shown in the right part of Fig. 4. In this work, the number of strips is set to 4 due to the balance between performance and computing complexity. The features of these strips are then sent to a shared Vision Transformer Block, different from the Vision Transformer in Backbone which extracts at frame level; the block here sees the frame bundle in a unique way, like parts of aggregated frames stack. Then the shared Vision Transformer Block generates the corresponding local part features of strips, named $loca{l}_1$, $loca{l}_2$, $loca{l}_3$, and $loca{l}_4$.

GaitVViT

Pipeline

For the proposed method GaitVViT, the model structure is shown in Fig. 5. The development of this model stems from the demand to enhance the capability of temporal information extraction within a common framework. As current research indicate, gait recognition methods emphasize temporal aspects typically by employing complex attention mechanisms or RNNs (Dou et al., 2023; Zhang et al., 2019). In the current SOTA methods, their TP modules often consist of a single layer of max pooling on the temporal dimension (Fan et al., 2023). These TP modules can be improved. Video ViTs treat images in sequence as frames in video, which differ from the ideas seeing them as a set and can be well-suited for this task. Thus, GaitVViT is introduced.

GaitVViT adopts the local temporal aggregation (LTA) module and global-local convolution (GLConv) module from GaitGL as the backbone (Lin, Zhang & Yu, 2020). So, in contrast to GaitTriViT, GaitVViT utilizes a traditional CNN as the backbone, GaitVViT takes a sequence of gait silhouette frames $Sils\in {\mathbb{R}}^{B\times C\times S\times H\times W}$ as inputs, where B is the batch size, C is the channel size, S is the number of frames, and $H\times W$ are the height and width of the pre-processed gait frames. After the extraction of the CNN-based backbone, the inputs Sils are mapped to a group of features $F\in {\mathbb{R}}^{B\times C^{\mathrm{\prime }}\times S^{\mathrm{\prime }}\times H^{\mathrm{\prime }}\times W^{\mathrm{\prime }}}$ where $C^{\mathrm{\prime }}$ is the channel size after convolution, $S^{\mathrm{\prime }}$ is length after LTA, $H^{\mathrm{\prime }}\times W^{\mathrm{\prime }}$ is the shape of each feature map. Similar to GaitTriViT, strips and part-dependent ideas are adopted, GaitVViT segments the feature maps generated by the backbone into multiple horizontal parts, shown as $F=\left\{P_1,P_2,\dots ,P_n\right\}$, where n equals the number of strips. For each $P_i\in {\mathbb{R}}^{B\times C^{\mathrm{\prime }}\times S^{\mathrm{\prime }}\times \frac{H^{\mathrm{\prime }}}{n}\times W^{\mathrm{\prime }}}$, feature map height becomes ${\displaystyle \frac{H^{\mathrm{\prime }}}{n}}$ by partition. These part features are then processed by a modified Video Vision Transformer, generating the n part features $F_{VViT}=\left\{P_1^{VViT},P_1^{VViT},\dots ,P_n^{VViT}\right\}$. For each $P_i^{VViT}\in {\mathbb{R}}^{B\times C^{\mathrm{\prime }}\times \frac{H^{\mathrm{\prime }}}{n}\times W^{\mathrm{\prime }}}$, the time dimension is reduced by aggregation. Subsequently, Horizontal Pooling pools each $P_i^{VViT}$ to $P_i^{HP}\in {\mathbb{R}}^{B\times C^{\mathrm{\prime }}\times 1}$, then model concatenates these n part of $P_i^{HP}$ together at the last dimension. The final feature is shown as $P_{final}^{HP}\in {\mathbb{R}}^{B\times C^{\mathrm{\prime }}\times n}$. After passing through the classification head, each part will calculate its loss individually. The Eqs. (13), (14), (15) below demonstrate the brief workflow of frame $F^i$ in model GaitVViT in Fig. 5.

(13) $$\begin{array}{c}{F}_{conv}^{it}=Par{t}^t\left(Conv\left(F^i\right)\right)\end{array}$$

(14) $$\begin{array}{c}{F}_{VViT}^i=\sum _{t=1}^{parts}VViT\left(\left[em{b}_{position}^{it};em{b}_{case}^{it};Emb\left(F_{conv}^{it}\right)\right]\right)\end{array}$$

(15) $$\begin{array}{c}Los{s}_{GaitVViT}^{F^i}=L\left(Head\left(Pool\left(F_{VViT}^i\right)\right)\right)\end{array}.$$where $Conv$ is the convolutional backbone, $Par{t}^t$ means the t-th part of horizontal strips partition, $Emb$ is patch embedding, along with position and case embeddings, $VViT$is the Video Vision Transformer block, $Pool$ represents horizontal pooling, $Head$ is the classification head and finally a function $L$ is used to calculate the loss.

Backbone

In GaitVViT, a traditional CNN is implemented as the backbone, the LTA and GLConv layer proposed by Lin et al. (2022) are also adopted. The overview of the backbone structure is shown in Fig. 6. The CNN-based backbone consists of multiple convolutional layers. At first, each input will be extracted by a 3DCNN layer with a kernel size of $\left[3\times 3\times 3\right]$ to obtain shallow features. Next, the LTA operation is employed to aggregate the temporal information and preserve more spatial information for trade-off. After that, Global and local feature extractor layers are implemented which consist of the GLConvA0 layer, Max Pooling layer, GLConvA1 layer, and GLConvB0 layer. The max pooling operation is implemented to down-sample the feature size at the last two dimensions for computing complexity trade-off. After the extractor, the combined feature assembling both global and local information is generated.

The details of the global and local convolutional layer (GLConv) are shown in Fig. 7. It basically consists of two parallel paths: one for local feature extraction and one for global feature extraction, which can take advantage of both global and local information.

The feature map will go through two branches. The upper branch is for local extraction where feature maps need partition before 3D convolution, the branch below is global extraction takes the whole feature map as input. And there are two combination methods: element-wise addition and concatenation.

The global branch implements a basic 3DCNN layer. It extracts the whole gait information and pays attention to the relations among local regions. The local branch is basically a 3D version of the Focal Convolutional layer proposed by Fan et al. (2020). It implements a 3DCNN layer with a shared kernel, the feature map will be split into several parts before the 3DCNN layer. They extract the local features and then combine them, which contain more detailed information than the global gait features. GLConv has two different structures due to different combinations between global and local features, GLConvA uses element-wise addition and GLConvB uses concatenation.

Summary

In this section, two Transformer-based architectures are proposed for gait recognition, GaitTriViT and GaitVViT. For GaitTriViT, the Vision Transformer is used as the frame-level backbone while incorporating case embedding and angle embedding to enhance frame-level feature extraction performance. Taking into account the similarity between gait recognition tasks and person re-identification tasks, this work draws inspiration from several articles on ReID tasks and introduces TCSS by Alsehaim & Breckon (2022), as well as the combination of part-dependent strategy (Fan et al., 2020), dividing frame-level features into different strips before another Vision Transformer block. These components above collectively build the local part spatial branch after the backbone, dedicated to extracting local spatial features. Another branch after the backbone employs another Vision Transformer block to extract global features, where temporal attention is used to jointly learn global temporal features (Fu et al., 2019; Rao et al., 2018; Zhang et al., 2020). The features from both branches are combined to generate the final features, and the predicted labels are generated by classification heads. For the optimizer, multiple loss functions are introduced to optimize the model together.

For GaitVViT, given the gait recognition common framework (Fan et al., 2023), where the wildly implemented TP module often consists of a single max pooling layer, which will waste the sequence information and need more attention for a complete improvement. The Video ViT is a variant of the original Vision Transformer (Dosovitskiy et al., 2020). Video ViT is created from the idea that regarding every frame in video as a patch. In the traditional transformer structure, every patch is a non-overlapping square region of an image, so when we change the scale and arrangement, the transformer can conduct the extraction on a whole sequence and run the self-attention on the time dimension. Adopting the LTA and GLConv layers from GaitGL proposed by Lin et al. (2022), this work connects the extracted feature representations to a Video ViT Encoder. The encoder implemented by LongFormer will conduct temporal extraction and aggregate the inputs to obtain the final features (Beltagy, Peters & Cohan, 2020).

Introduction

In this section, the datasets and implementation details are discussed, including two popular benchmarks, CASIA-B and OUMVLP (Takemura et al., 2018; Yu, Tan & Tan, 2006). Moreover, several details during the training and evaluation phase are explained.

Datasets

CASIA-B is provided by Yu, Tan & Tan (2006) for gait recognition and to promote research. CASIA-B is a large multi-view gait database, which was created in January 2005. It has 124 subjects, and the gait data was captured from 11 views. Three variations, namely view angle, clothing and carrying condition changes, are separately considered. In this article, we use the human silhouettes extracted from video files as benchmarks. The format of the filenames in CASIA-B is ‘xxx-mm-nn-ttt.png’, where ‘xxx’ is subject id, ‘mm’ stands for walking status, including ‘nm’ (normal), ‘cl’ (in a coat) or ‘bg’ (with a bag), ‘nn’ is sequence number for each walking status, normal walking has six sequences, wearing coat and carrying bag have two sequences each; ‘ttt’ is view angle can be ‘000’, ‘018’, …, ‘180’ . Examples of CASIA-B are shown in Fig. 8.

Each subject has a maximum of 110 sequences. We use subjects with ID from 1 to 74 as the training set, and subjects with ID from 75 to 124 as the test set. During the testing phase, we use the first four sequences from ‘nm’ (nm-1, nm-2, nm-3, nm-4) as the gallery set, and the remaining six sequences are divided into three query sets based on their respective situations: ‘nm’ query includes ‘nm-5’ and ‘nm-6’, ‘bg’ query includes ‘bg-1’ and ‘bg-2’, and ‘cl’ query includes ‘cl- 1’ and ‘cl-2’ (Chao et al., 2018; Fan et al., 2020; Lin et al., 2022; Yu, Tan & Tan, 2006).

OUMVLP is part of the OU-ISIR Gait Database, which stands for Multi-View Large Population Dataset, provided by Takemura et al. (2018). OUMVLP is meant to aid research efforts in the general area of developing, testing and evaluating algorithms for cross-view gait recognition. The Institute of Scientific and Industrial Research (ISIR), Osaka University (OU) has copyright in the collection of gait video and associated data and serves as a distributor of the OU-ISIR Gait Database. The data was collected in conjunction with an experience-based long-run exhibition of video-based gait analysis at a science museum. The dataset consists of 10,307 subjects (5,114 males and 5,193 females with various ages, ranging from 2 to 87 years) from 14 view angles, ranging 0°–90°, 180°–270°. Gait images of 1,280 × 980 pixels at 25 fps are captured by seven network cameras (Cam1–7) placed at intervals of 15-degree azimuth angles along a quarter of a circle whose center coincides with the center of the walking course. The illustration is shown in Fig. 9. Each subject has two sequences, 00 for probe and 01 for gallery. We select 5,153 subjects with odd-numbered IDs as the training set, and the remaining 5,154 subjects as the test set (Chao et al., 2018; Fan et al., 2020; Lin et al., 2022; Takemura et al., 2018).

Implementation details

The project is implemented on both Windows 11 and Debian, but mainly on Windows 11. The Python IDE chosen is PyCharm 2021.1.3.0 and Python version is 3.9.0. The framework chosen is PyTorch 1.13.0. The project is trained and evaluated on one NVIDIA GeForce RTX 3080 Ti GPU with CUDA version 11.8.

We follow the common way by Fan et al. (2022) to pre-process the silhouette data from CASIA-B and OUMVLP datasets, then choose the hyperparameters taking hardware limitations and efficiency into consideration. This pre-processing involved removing invalid data not having whole body in frame, arranging frames in structure of ID-condition-angle, aligning and cropping silhouette images to ensure the subject’s body is in the center of the image, and has a proper size, more details can refer to original works by Fan et al. (2022). After pre-processing, each frame image’s size is $64\times 44$. For GaitTriViT specifically, since we don’t have the computational resource to train a well performed image model from scratch, we initialized the Vision Transformer backbone with parameters pre-trained on ImageNet-21K (Deng et al., 2009; Wightman, 2019), the input of ViT requires RGB-like images with three channels and a size of $256\times 128$ . But our silhouettes are single-channel binary images. Therefore, we inserted a fully connected layer in the head of backbone with an input dimension of 1 and an output dimension of 3 to map the silhouette from $Sil\in {\mathbb{R}}^{H\times W\times 1}$ to $\widehat{Sil}\in {\mathbb{R}}^{H\times W\times 3}$ pseudo-RGB images. And in data augmentation phase, we resized the images to the required size.

CASIA-B and OUMVLP datasets differ in camera angles and walking scenarios. CASIA-B has 11 camera angles and a total of 10 walking sequences. Therefore, in Eq. (3), $E_{angle}$ has the shape of $\left[a\times D\right]$, where $a=11$, and $E_{case}$ is in shape of $\left[c\times D\right]$, where $c=10$. D is the embedding dimension set to 768. In contrast, OUMVLP has 14 camera angles and no distinction in walking scenarios, so only $E_{angle}^{\mathrm{\prime }}\in {\mathbb{R}}^{a^{\mathrm{\prime }}\times D}$, where $a^{\mathrm{\prime }}=14$.

In this work, excluding the ViT in backbone of GaitTriViT, most parameters are initialized using the Kaiming initialization (He et al., 2015) following the VID-Trans-ReID method. For GaitTriViT, the number of frames T in a frame bundle is set to 4. The selection strategy of frames in every bundle when training is dividing the whole sequence into T parts and randomly selecting one frame from each part, creating a frame bundle where each frame can be selected again. During testing, T frames are sequentially selected from the whole sequence. The batch size is set to 52, the optimizer is stochastic gradient descent (SGD), and the scheduler is using cosine learning rate decay with warming up (Loshchilov & Hutter, 2017). For GaitVViT, during the training phase, the number of frames T in each batch is set to 30, the selection strategy is randomly choosing 30 frames in order among the sequence. During the test phase, the model uses all frames within one sequence to generate the final feature. The batch size is set to 36, the optimizer is Adam, and the scheduler is multi step learning rate.

OUMVLP has 10,307 subjects, silhouettes are shot from 14 different angles, which don’t have different walking status between sequences.

Summary

In this section, the choices of datasets and implementation details are explained. Two popular datasets are chosen in this work: CASIA-B and OUMVLP. CASIA-B is a classic dataset for gait recognition research specifically. It has 124 subjects; each subject has multiple sequences varied in 11 camera angles and three walking statuses. OUMVLP is a new dataset compared to CASIA-B, it has the most subjects among the gait datasets so far, which consists of 5,114 males and 5,193 females captured from 14 camera angles, each subject has two sequences. Furthermore, several implementation details are explained including pre-processing, different settings on each benchmark and details in hyper-parameters.

State-of-the-art comparison

In gait recognition tasks, many researchers have made a lot of contributions. The GEINet by Shiraga et al. (2016) leverages the gait energy images (GEI) as the representations of gait features, open the research towards gait recognition. The GaitSet by Chao et al. (2018) led the new era of appearance-based Gait Recognition, then plenty of novel models came out e.g., GaitPart by Fan et al. (2020), GaitGL by Lin, Zhang & Yu (2020), GaitBase by Fan et al. (2022), SRN by Hou et al. (2021), GLN by Hou et al. (2020) and DeepGait-3D by Fan et al. (2023). They are all regarded as state-of-the-art models by now, in which GaitSet, GaitPart and GaitGL are considered most iconic and are chosen mostly as milestones.

The evaluation metric of single-view-gallery-evaluation is used in almost all silhouette-based gait recognition methods. First, the test set needs to split all the sequences into two subsets named probe and gallery, where each subject in probe has a sequence consist of M camera views and N for gallery, where M is equal to N in most cases. Each probe-gallery view pair is denoted as $\left(V_i^p,V_j^g\right)$, where $i\in \left[1,2,\dots ,M\right]$, $j\in \left[1,2,\dots ,N\right]$.

Second, for each probe-gallery view pair $\left(V_i^p,V_j^g\right)$, after model inference $\mathcal{J}$, every probe sequence under the view $V_i^p$ is compared to all gallery sequences under different view $V_j^g$ to calculate the Euclidean distance, the label of the closest gallery sequence will be assigned to the probe. The rank-1 accuracy $AC{C}_{ij}$ is then computed for each probe-gallery pair by checking the percentage of the predicted label matching the ground-truth $label$. In the end, an accuracy matrix $ACC$ in the shape of $M\times N$ is obtained.

Third, the identical-view cases, where $V_i^p=V_j^g$, are always excluded. So, in the $ACC$ matrix, the corresponding elements (mostly the diagonal elements) are abandoned while the rest are averaged as the evaluation metric. At last, the accuracy for each probe view $AC{C}_i^p$ is acquired by averaging the elements of $ACC$ in row excluding identical-view cases. The processes are as follows:

(16) $$\begin{array}{c}AC{C}_{ij}=R1\left(euc\left(\mathcal{J}\left(V_i^p\right),\mathcal{J}\left(V_j^g\right)\right),label\right)\end{array}$$

(17) $$\begin{array}{c}AC{C}_i^p={\displaystyle \frac{1}{N-1}\left(\sum _{j=1}^{N}AC{C}_{ij}-AC{C}_{ii}\right)}\end{array}$$where $R1$ stands for rank-1 calculation and $euc$ is short of Euclidean Distance.

In this article, the ‘Single’ and ‘Cross’ marks in the following tables indicate the different evaluation protocols. The ‘Single’ stands for the single-view-gallery evaluation which is the regular evaluation method explained above. For example, the CASIA-B dataset has 3 walking status and 11 camera angles (Yu, Tan & Tan, 2006), so, for probe sequences whose walking status is ‘NM’ and view angle is ‘090’, they needed to compare with 10 galleries with different view angle excluding the gallery having the same view, the average of 10 results become the final result of this specific probe. The ‘Cross’ stands for cross-view-gallery evaluation. Particularly, for each probe view, the sequences of all gallery views are adopted for the comparison with the identical-view cases excluded. The accuracy under cross-view-gallery evaluation is quite higher than single-view-gallery, because subjects in some views may experience significant silhouette changes, bringing difficulty and less discriminativeness for recognition (Hou et al., 2023).

The evaluation results of two proposed methods GaitTriViT and GaitVViT are presented below, as shown in Tables 1 and 2. The data of the state-of-the-art methods are collected from their own articles.

The experiments show that GaitTriViT faces huge difficulties with the two popular benchmarks. The regular single-view-gallery accuracy can only surpass the GEINet, indicating the bad generalization of GaitTriViT. Even the cross-view- gallery performances are dropped when the walking status is bringing a bag, or especially, wearing a coat. The GaitTriViT has bad robustness towards appearance noises.

GaitVViT performs better, on CASIA-B, when the walking status is normal walking, the performance of GaitVViT can slightly surpass GaitGL at probe views of 0°, 18°, 36°, 54° and 126°, making the average accuracy slightly better too. But it doesn’t perform well enough for a proposed transformer-enhanced method when the walking status is bag bring or wearing a coat. Maybe due to the sensitivity of transformer-based structure for appearance information, i.e., the method is less robust to appearance noises.

Comparison shows that GaitTriViT focus more on spatial feature extraction by employing three individual vision transformer in each extraction phase (one for frame-level features in the backbone, one for set-level local features in a local branch, and one for frame-level extraction in the global branch before the attention module), while the specific temporal modeling task is assigned to a spatio-temporal attention module; in contrast, GaitVViT adopted the Video ViT to replace the common TP module and enhance its functionality, which focus more in temporal aspect apparently. Given the situation that current Transformer-based methods have not achieved astonishing outcomes in the field of gait recognition, the current ViT structure may not be a good upstream backbone for inputs like gait silhouette. According to the argument by Fan et al. (2023), many patches on a gait silhouette are all-white (all 1) or all-black (all 0), where neither posture nor appearance information are provided. They call them dumb patches. Since all values from a dumb patch are all 0 or all 1, These all-1 or all-0 dumb patches can make backward gradients significantly ineffective or even computationally invalid for the parameter optimization of downstream ViT layers. For GaitVViT, it meets the basic line of current state-of-the-art methods. A traditional CNN backbone makes sure the performance away from too bad, despite the augmentation on temporal pooling module gains no astonishing improvement. Also, these two methods lack in-wild testing. Gait recognition methods on pre-processed datasets, to some extent, have already saturated their performance, application performance in the real world will be more important in future research (Cosma, Catruna & Radoi, 2023; Cosma & Radoi, 2021).

Ablation study

In this work, multiple technologies are employed in two methods. For GaitTriViT, there are TCSS and angle embedding (also case embedding on the CASIA-B dataset). But it achieves not a promising performance on two popular benchmarks. Several ablation experiments were carried to deep dive into the contribution of each technology and try to improve performance by introducing extra mechanisms, e.g., excluding specific modules, changing the selection method, rearranging the order of strip segmentation, and introducing part embeddings. For GaitVViT, there is also an ablation study excluding certain modules or modifications, to explore the individual contribution of each mechanism and potential of the model.

Some other implementation details may also have an influence on model performance. While this work using Kaiming initialization and ImageNet pretraining, as well as a combination of loss functions (e.g., triplet, cross-entropy, and attention loss), we completely follow the steps of original articles, where more details will be demonstrated. Exploring alternative strategies may potentially contribute to the performance of gait recognition. For example, Palla, Parida & Sahu (2024) introduced a hybrid Harris hawks and Arithmetic optimization algorithm (Heidari et al., 2019; Rao et al., 2023) introduced a hybrid whale and gray wolf optimization algorithm (Mirjalili & Lewis, 2016; Mirjalili, Mirjalili & Lewis, 2014), trying to select the most optimal gait features. They argue the optimization problems have become increasingly complex, traditional mathematical methods (e.g., Newton’s descent and gradient descent method) become difficult solving them effectively. Thus, many scholars focus on meta heuristics (MAs), a class of optimization algorithms inspired by natural phenomena (e.g., biological group behavior, physical phenomena and evolutionary laws). MAs require no gradient information, flexible, easily implemented, and widely used in complex situations (Jia et al., 2023). These MAs can also improve the performance compared to the mathematic loss optimization used now.

If not mentioned, the results below are obtained following the cross-view-gallery evaluation, as it is closer to real-world application scenarios.

Limitations

In this work, two Transformer-based methods GaitTriViT and GaitVViT are proposed to tackle the gait recognition task. On two popular benchmarks: CASIA-B and OUMVLP, GaitTriViT meets huge difficulties, the results only surpass the GEINet method leveraging gait energy images (GEI) for temporal modeling. Among its own results, GaitTriViT also has a lot of limitations. On the smaller CASIA-B dataset, based on different walking status, there are three scenarios: normal walking, bag carrying and wearing coat. Compared to probe status of normal walking, the evaluation in bag carrying status encounters a reasonable drop-down relatively. However, in the case of subjects wearing coats, there was a significant performance drop during evaluation, highlighting a lack of robustness in our method when silhouette appearances have significant changes. For GaitVViT, although the performance has met the acceptable level on both popular benchmarks. It’s still a little far away from cutting-edge methods. It is not enough for a temporal augmented method. And the sensitivity toward appearance noise also needs addressing. Furthermore, Vision Transformer also has its inherent limitations, e.g., computation consuming and demand for large datasets, all need extra works in future.

Future works

Given the lack of robustness in GaitTriViT when silhouette appearances have significant changes. If given the opportunity to work on this further, work will be focused on the robustness to minimize the noises of appearance, the influence of subjects’ walking frequency on gait patterns needs to be addressed too. For example, inserting a module to separate the appearance and gait features. For GaitVViT, the generalization needs no worry, it is encouraging to combine the tricks of GaitTriViT (e.g., angel embedding) and the structure of GaitVViT together, the fusion of two methods may help to pursue better performance. Technologies like data augmentation and reinforcement learning may also contribute to the task.

For the inherent limitations of Vision Transformer itself, we might introduce the idea of latent and running the model in latent space, leveraging powerful encoders, e.g., variational auto-encoder (VAE) and denoising diffusion implicit model (DDIM). They may help the model to capture high-level patterns better and increase robustness toward noise as well as efficiency.

Moreover, real-world applications are challenging, rather than popular in-door silhouette datasets, this work needs to face the in-wild datasets captured from real-world scenes, integrating target detection and image segmentation modules. The target would be to enhance the model’s robustness when facing variations in subject appearances, and ultimately, testing a capable real-world application to evaluate the capabilities under diverse conditions.