A weighted sparse coding model on product Grassmann manifold for video-based human gesture recognition

Product Grassmann manifold

A point on Grassmann manifold $\mathcal{G}\left(p,d\right)$ is a p-dimensional subspace of ℝ^d (Absil, Mahony & Sepulchre, 2009). That means it can be spanned by any orthonormal basis X = [x₁|x₂|⋯|x_p] ∈ ℝ^d×p and it is denoted as span(X). For the sake of convenience, we use the same symbol X to represent span(X). The distance of two points X and Y on Grassmann manifold can be defined as ${\displaystyle d_g\left(\mathbf{X},\mathbf{Y}\right)=\parallel \Pi \left(\mathbf{X}\right)-\Pi \left(\mathbf{Y}\right){\parallel }_F=\parallel \mathbf{X}{\mathbf{X}}^T-\mathbf{Y}{\mathbf{Y}}^T{\parallel }_F}$ where embedding mapping $\Pi :\mathcal{G}\left(p,d\right)\to \mathrm{Sym}\left(d\right)$ is defined as Π(X) = XX^T, and Sym(d) is the symmetric matrices space with order d (refer to Harandi et al., 2015). Product Grassmann manifold (PGM) $\mathcal{PG}$(p₁, …, p_M| d₁, …, d_M) is defined as ${\displaystyle \mathcal{PG}\left(p_1,\dots ,p_M|d_1,\dots ,d_M\right)=\mathcal{G}\left(p_1,d_1\right)\times \cdots \times \mathcal{G}\left(p_M,d_M\right)}$ where the symbol × denotes Cartesian product, $\mathcal{G}\left(p_i,d_i\right)$ (i = 1, …, M) is called factor manifold and p_i(i = 1, ⋯, M) is called dimension of each factor manifold. A point on PGM is denoted as [X] = (X¹, …, X^M). The distance between two points [X] = (X¹, …, X^M) and [Y] = (Y¹, …, Y^M) on PGM is defined as weighted average distance of each factor Grassmann manifold ${\displaystyle d_{\mathcal{PG}}\left(\left[\mathbf{X}\right],\left[\mathbf{Y}\right]\right)=\sqrt{\sum _{m=1}^M{\omega }_m{d}_g^2\left({\mathbf{X}}^m,{\mathbf{Y}}^m\right)}}$ where each weight ω_m(≥0) represents the importance of factor manifold $\mathcal{G}\left(p_m,d_m\right)$ and $\sum _{m=1}^M{\omega }_m=1$.

Data representation on PGM

In the real world, there exists many data with multi-dimensional structure. For example, video can be represented as tensor 𝒜 ∈ ℝ^{J₁×J₂×J₃}, where J₁, J₂ and J₃ represent height, width and length of video respectively; Image set can be represented as tensor 𝒜 ∈ ℝ^{J₁×J₂×J₃}, where J₁, J₂ and J₃ represent height, width and number of image set respectively; Light field can be represented as tensor 𝒜 ∈ ℝ^{J₁×J₂×J₃×J₄} (Wang & Zhang, 2020), where J₁ and J₂ represent angular resolution of light field, J₃ and J₄ represent spatial resolution of light field.

Before introducing data representation on PGM, we give a schematic of matrix unfolding for a third tensor in Fig. 1. The reader can refer to Kolda & Bader (2009) for more theory on tensor operation. For ease of understanding we give a corresponding example of two videos described by tensor in Fig. 2. We find that the corresponding unfolding matrix is discriminative for two videos with different labels, hence the multi-dimensional information of video tensor is worth mining for classification task.

In the following, we discuss the way to represent multi-dimensional data on PGM. The variation for each mode of a tensor 𝒜 ∈ ℝ^{J₁×⋯×J_M} can be captured by HOSVD (followed as Lui, 2012), which factorize tensor $\mathcal{A}$ using the orthogonal matrices in the following equation: ${\displaystyle \mathcal{A}=\mathcal{S}{\times }_1{\mathbf{V}}^{\left(1\right)}{\times }_2\cdots {\times }_M{\mathbf{V}}^{\left(M\right)}}$ where V^(m) ∈ ℝ^J_m×d_m (m =1 , …, M) are orthogonal matrices spanning the row space with the first J_m rows associated with non-zero singular values from the unfolded matrices respectively, 𝒮 ∈ ℝ^{d₁×⋯×d_M} is a core tensor, d_m = ∏_i≠mJ_i, and ×_m(m =1 ,…,M) denotes mode- m multiplication. Each V^(m)T ∈ ℝ^d_m×J_m is a tall orthogonal matrix. We take the first p_m (p_m ≤ J_m) columns of V^(m)T and denote it as U^(m) ∈ ℝ^d_m×p_m. Hence, U^(m) is a point on Grassmann manifold $\mathcal{G}\left(p_m,d_m\right)$. And then (U⁽¹⁾, …, U^(M)) is a point on PGM $\mathcal{G}\left(p_1,d_1\right)\times \cdots \times \mathcal{G}\left(p_M,d_M\right)$.

Remark: The value of parameter p_m(m =1 , …, M) reflects the principal information of data. In brief, the information of data may be redundant if the value of p_m is too large and the information of data may be insufficient if the value of p_m is too small. Hence it is important to select the parameters p_m(m =1 , …, M) and we will discuss this problem in details in our experiments.

Weighted sparse coding model on PGM

Let {[X₁], …, [X_N]} be the training set which includes N samples, where $\left[{\mathbf{X}}_i\right]=\left({\mathbf{X}}_i^1,\dots ,{\mathbf{X}}_i^M\right)\in \mathcal{PG}\left(p_1,\dots ,p_M|d_1,\dots ,d_M\right)$ is a point on product Grassmann manifold. Let $\left[\mathbf{Y}\right]=\left({\mathbf{Y}}^1,\dots ,{\mathbf{Y}}^M\right)\in \mathcal{PG}$(p₁,…,p_M|d₁,…,d_M) be a query sample on product Grassmann manifold. The sparse coding model on PGM is formulated as follows: ${\displaystyle \underset{\alpha }{min}{d}_{\mathcal{PG}}^2\left(\left[\mathbf{Y}\right],{⨄}_{i=1}^N{\alpha }_i\odot \left[{\mathbf{X}}_i\right]\right)+\lambda \parallel \alpha {\parallel }_1}$ where α = (α₁, …, α_N)^T is the sparse representation coefficient, the abstract symbols ${⨄}_{i=1}^N$and ⊙ are used to simulate “linear” combination defined on PGM, i.e., addition and scalar-mulitplication. $d_{\mathcal{PG}}\left(\left[\mathbf{Y}\right],{⨄}_{i=1}^N{\alpha }_i\odot \left[{\mathbf{X}}_i\right]\right)$ measures the distance between reconstruction ${⨄}_{i=i}^N{\alpha }_i\odot \left[{\mathbf{X}}_i\right]$ and the query sample [Y]. To get the sparse coding model on PGM, proper definitions of distance and combination operator should be specified. According to the geometric property of Grassmann manifold, we use the embedded distance and linear combination on the space of symmetric matrices. Hence, we construct the weighted sparse coding model on PGM as follows, (1)${\displaystyle \underset{\alpha }{min}\sum _{m=1}^M{\omega }_m\parallel {\mathbf{Y}}^m{{\mathbf{Y}}^m}^T-\sum _{i=1}^N{\alpha }_i\left({\mathbf{X}}_i^m{{\mathbf{X}}_i^m}^T\right){\parallel }_F^2+\lambda \parallel \alpha {\parallel }_1.}$

Algorithm for the weighted sparse coding on PGM

In this subsection, we show how to solve the optimization Eq. (1). We have ${\displaystyle \underset{\alpha }{min}\sum _{m=1}^M{\omega }_m\parallel {\mathbf{Y}}^m{{\mathbf{Y}}^m}^T-\sum _{i=1}^N{\alpha }_i\left({\mathbf{X}}_i^m{{\mathbf{X}}_i^m}^T\right){\parallel }_F^2=\underset{\alpha }{min}\sum _{m=1}^M{\omega }_m\left\{\right.\mathrm{Tr}\left({{\mathbf{Y}}^m}^T{\mathbf{Y}}^m{{\mathbf{Y}}^m}^T{\mathbf{Y}}^m\right)+\sum _{i,j=1}^N{\alpha }_i{\alpha }_j\mathrm{Tr}\left({{\mathbf{X}}_i^m}^T{\mathbf{X}}_j^m{{\mathbf{X}}_j^m}^T{\mathbf{X}}_i^m\right)-2\sum _{i=1}^N{\alpha }_i\mathrm{Tr}\left({{\mathbf{X}}_i^m}^T{\mathbf{Y}}^m{{\mathbf{Y}}^m}^T{\mathbf{X}}_i^m\right)\left\}\right..}$ For simplicity, we define a matrix K^m(X) and a vector K^m(X, Y) as following, i.e., their elements are

${\left[K^m\left(\mathbf{X}\right)\right]}_{ij}={\omega }_m\mathrm{Tr}\left({{\mathbf{X}}_i^m}^T{\mathbf{X}}_j^m{{\mathbf{X}}_j^m}^T{\mathbf{X}}_i^m\right),i,$j=1 , …, N

${\left[K^m\left(\mathbf{X},\mathbf{Y}\right)\right]}_i={\omega }_m\mathrm{Tr}\left({{\mathbf{X}}_i^m}^T{\mathbf{Y}}^m{{\mathbf{Y}}^m}^T{\mathbf{X}}_i^m\right),i=1,\dots ,N$

Hence the model Eq. (1) becomes ${\displaystyle \underset{\alpha }{min}\sum _{m=1}^M\left\{{\alpha }^T{K}^m\left(\mathbf{X}\right)\alpha -2{\alpha }^T{K}^m\left(\mathbf{X},\mathbf{Y}\right)\right\}+\lambda \parallel \alpha {\parallel }_1=\underset{\alpha }{min}{\alpha }^T\left(\sum _{m=1}^M{K}^m\left(\mathbf{X}\right)\right)\alpha -2{\alpha }^T\left(\sum _{m=1}^M{K}^m\left(\mathbf{X},\mathbf{Y}\right)\right)+\lambda \parallel \alpha {\parallel }_1.}$

The symemetric matrix $\sum _{m=1}^M{K}^m\left(\mathbf{X}\right)$ is positive semidefinite since for all v = (v₁, v₂, …, v_N)^T ∈ ℝ^N: ${\displaystyle {\mathbf{v}}^T\left(\sum _{m=1}^M{K}^m\left(\mathbf{X}\right)\right)\mathbf{v}=\sum _{m=1}^M\sum _{i=1}^N\sum _{j=1}^N{\omega }_m{v}_i{v}_j\mathrm{Tr}\left({{\mathbf{X}}_i^m}^T{\mathbf{X}}_j^m{{\mathbf{X}}_j^m}^T{\mathbf{X}}_i^m\right)=\sum _{m=1}^M{\omega }_m\mathrm{Tr}\left(\sum _{i=1}^N\sum _{j=1}^N{v}_i{v}_j{{\mathbf{X}}_i^m}^T{\mathbf{X}}_j^m{{\mathbf{X}}_j^m}^T{\mathbf{X}}_i^m\right)=\sum _{m=1}^M{\omega }_m\parallel \sum _{i=1}^N{v}_i{\mathbf{X}}_i^m{{\mathbf{X}}_i^m}^T{\parallel }_F^2\ge 0.}$ Therefore, the problem is convex and can be solved by a vectorized sparse coding problem. In detail, let UΣU^T be the SVD of $\sum _{m=1}^M{K}^m\left(\mathbf{X}\right)$, then the problem is equal to (2)${\displaystyle \underset{\alpha }{min}\parallel {\mathbf{Y}}^{\ast }-\mathbf{A}\alpha {\parallel }^2+\lambda \parallel \alpha {\parallel }_1}$ where A = Σ^1/2U^T and ${\mathbf{Y}}^{\ast }={\Sigma }^{-1/2}{\mathbf{U}}^T\left(\sum _{m=1}^M{K}^m\left(\mathbf{X},\mathbf{Y}\right)\right)$. The pseudo-code for performing the proposed weighted sparse coding on PGM is summarized in Algorithm 1, which is simply called WSC-PGM.

 
_______________________ 
Algorithm 1 Weighted sparse coding on product Grassmann manifold (WSC- 
PGM)_______________________________________________________________________________________________ 
Require: 
   Training data includes N samples on PGM: [Xi] = (X1i,X2i,...,XMi), i = 
   1,2,...,N and Xmi ∈G(pm,dm), m = 1,2,...,M; the query sample on PGM: 
   [Y] = (Y1,Y2,...,YM) and Ym ∈G(pm,dm),m=1,2,...,M. 
Ensure: 
   The sparse code α∗ 
  for m = 1 : M do 
     for i = 1 : N do 
        for j = 1 : N do 
         [Km(X)]ij = ωmTr(XmiTXm 
j Xm 
j TXm 
i )       /* compute matrix Km(X) 
        end for 
         [Km(X,Y)]i  = ωmTr(XmiTYmYmTXm 
i )            /* compute vector 
   Km(X,Y) 
     end for 
  end for 
   M 
    ∑ 
 m=1 Km(X) = UΣUT          /* compute SVD of  M 
  ∑ 
m=1 Km(X) 
  A ← Σ1/2UT 
    Y∗ ← Σ−1/2UT ( M 
      ∑ 
  m=1 Km(X,Y)) 
  α∗ = arg min 
α  ∥Y∗− Aα∥2 + λ∥α∥1           /* the solution of model (2) 
  return  α∗_______________________________________________    

Classification rule and algorithm

When model Eq. (1) is minimized, the optimal coefficient α^∗ can be used for classification. Following the idea of the Sparse Representation Classification (SRC) (Wright et al., 2008), the query sample can be classified by it’s codes α^∗ of these labeled training samples [X_i]i =(1 , 2, …, N).

In details, let ${\left({\alpha }_1^{\ast }\delta \left(l_1-k\right),{\alpha }_2^{\ast }\delta \left(l_2-k\right),\dots ,{\alpha }_N^{\ast }\delta \left(l_N-k\right)\right)}^T$ be the class- k sparse codes, where l_i(i =1 , 2, …, N) is the class label of training sample [X_i] and δ(x) is the discrete Dirac function. ${\displaystyle \delta \left(x\right)=\left\{\begin{array}{cc}\hfill 1\hfill & \hfill x=0\hfill \\ \hfill 0\hfill & \hfill else\hfill \end{array}.\right.}$ The residual error of a query sample [Y] =(Y¹, Y², …, Y^M) by using the samples associated to class k is defined as (3)${\displaystyle {\varepsilon }_k\left(\left[\mathbf{Y}\right]\right)=\sum _{m=1}^M{\omega }_m\parallel {\mathbf{Y}}^m{{\mathbf{Y}}^m}^T-\sum _{i=1}^N{\alpha }_i^{\ast }\left({\mathbf{X}}_i^m{{\mathbf{X}}_i^m}^T\right)\delta \left(l_i-k\right){\parallel }_F^2.}$ Then the estimated class of the query Y is determined by (4)${\displaystyle \mathrm{Label}\left(\left[\mathbf{Y}\right]\right)=arg\underset{k}{min}{\varepsilon }_k\left(\left[\mathbf{Y}\right]\right).}$ The procedure of sparse representation classification on product Grassmann manifold is summarized in Algorithm 2.

 
________________________________________________________ 
Algorithm 2 Weighted sparse representation classification on product Grass- 
mann manifold (WSRC-PGM)_______________________________________________________________ 
Require: 
   Training data [Xi] = (X1i,X2i,...,XMi),i=1,2,...,N belonging to c classes; 
   the query [Y] = (Y1,Y2,...,YM) 
Ensure: 
   The class label Label([Y]) of the given test sample [Y] 
  Compute α∗ as Algorithm 1 
  Compute residual ɛk([Y]) by using equation (3) 
  Compute the class label by using equation (4) 
  return  Label([Y])___________________________________________________________________________    

Cambridge hand gesture datasets

The Cambridge hand gesture datasets (Kim & Cipolla, 2008) contains 900 video sequences with 9 classes and it is divided into 5 sets according to different illuminations. The 9 classes are flat-leftward (FL), flat-rightward (FR), flat-contract (FC), spread-leftward (SL), spread-rightward (SR), spread-contract (SC), V-shape-leftward (VL), V-shape-rightward (VR) and V-shape-contract (VC) respectively. We follow the experimental protocol in paper (Kim & Cipolla, 2008), set 5 (normal illumination) is considered for training while the remaining sequences (with different illumination characteristics) are used for testing. In this experiment, the original sequences are converted to grayscale and resized to 20 × 20 × 20. Obviously, experiment results depend on the selection of parameters, so we firstly discuss the parameter setting in the following.

Experiment result on testing sets

In this experiment, the parameter λ is set as 0.1. With the three combinations of parameters (p₁, p₂, p₃, ω₁, ω₂), the samples of Set1-Set4 are represented as points on $\mathcal{PG}\left(8,18,12|400,400,400\right)$, $\mathcal{PG}\left(20,10,12|400,400,400\right)$ and $\mathcal{PG}\left(14,12,12|400,400,400\right)$ respectively. Table 5 summarizes the correct recognition rate for Set1-Set4 and the average correct recognition rate which followed by the standard deviation. As Table 5 shows, WSRC-PGM has superior performance compared with TCCA (Kim & Cipolla, 2008), PM (Lui, 2012), gSC and kgSC (Harandi et al., 2015), DMD+SC(SCCD2) (Singh et al., 2021).

The confusion matrix of our proposed approach on the four testing sets under parameter combination 1 are given in Fig. 6. Naturally, confusion matrices for combination 2 and 3 can be discussed similarly and they are omitted here. From Fig. 6 see, the most misclassified class is SL and most of the misclassified samples with lable SL were misassigned to the SC class. The second most misclassified class is SC and most of the misclassified samples with lable SC were misassigned to the VC class.

Table 3:

The top 5% combinations of parameter on Cambridge hand gesture datasets.

Parameter	$p_1^{\ast }$	$p_2^{\ast }$	$p_3^{\ast }$	${\omega }_1^{\ast }$	${\omega }_2^{\ast }$	${\omega }_3^{\ast }$
combination 1	8	18	12	0.3	0.3	0.4
combination 2	20	10	12	0.2	0.4	0.4
combination 3	14	12	12	0.2	0.4	0.4

DOI: 10.7717/peerjcs.923/table-3

Table 4:

CRR of the five cross validation sets on Cambridge hand gesture datasets.

Cross validation sets	1	2	3	4	5
CRR of combination 1	100%	100%	100%	97.22%	100%
CRR of combination 2	100%	100%	100%	97.22%	100%
CRR of combination 3	100%	100%	100%	97.22%	100%

DOI: 10.7717/peerjcs.923/table-4

Table 5:

Recognition results on the Cambridge hand-gesture dataset.

Method	Set1	Set2	Set3	Set4	Overall
TCCA (Kim & Cipolla, 2008)	81	81	78	86	82 ± 3.5%
PM (Lui, 2012)	93	89	91	94	91.7 ± 2.3%
gSC (Harandi et al., 2015)	93	92	93	94	93.3 ± 0.9%
kgSC (Harandi et al., 2015)	96	92	93	97	94.4 ± 2.0%
HOG3DVV+GGDA (Verma & Choudhary, 2018)	86	93	87	93	89.7
WSRC-PGM (combination 1)	98	92	96	97	95.6 ± 2.8%
WSRC-PGM (combination 2)	99	91	94	96	95.0 ± 3.5%
WSRC-PGM (combination 3)	99	89	94	96	94.3 ± 4.2%

DOI: 10.7717/peerjcs.923/table-5

Ballet datasets

The Ballet dataset contains 44 videos including 8 complex motion patterns from 3 persons (Fathi & Mori, 2008). In detail, the actions are “left-to-right hand opening”, “right-to-left hand opening”, “standing hand opening”, “leg swinging”, “jumping”, “turning” , “hopping” and “standing still” . Main challenge of this dataset is large variations among classes such as speed, clothing and motion paths. The frame images are normalized and centered in a fixed size of 20 × 20. We extract total 2400 sub-videos by randomly sampling 6 frames from original video that exhibited the same action and then images are converted to grayscale. We randomly select 1200 samples as training set and the remainder as testing set.

Similar to the discussion for parameter setting of experiment on Cambridge hand gesture dataset, we jointly determine the parameters (p₁, p₂, p₃, ω₁, ω₂) by 5-fold cross validation on training set, where p₁, p₂ are all in the range of {2:2:20}, p₃ is in the range of {1:1:6} and ω₁, ω₂ are in the range as Table 2. The top 5 % optional parameter combinations of (p₁, p₂, p₃, ω₁, ω₂, ω₃) are listed in Table 6. And the samples on testing set are represented on $\mathcal{PG}\left(10,6,2|120,120,400\right)$, $\mathcal{PG}\left(10,4,4|120,120,400\right)$ and $\mathcal{PG}\left(10,2,6|120,120,400\right)$ respectively in experiments. Table 7 summarizes the average correct recognition rate. The results show that our algorithm has superior performance compared with some state-of-the-art methods. And the confusion matrix of our proposed approach on the testing set under the three parameter combinations are given in Fig. 7.

UMD Keck body-gesture datasets

The UMD Keck Body-Gesture Datasets contains 14 naval body gestures acquired from both static and dynamic backgrounds. The subjects and the camera remain stationary in the static backgrounds, the subjects and the camera are moving in the dynamic backgrounds. 126 videos and 168 videos are collected from the static scene and the dynamic environment respectively. The 14 body gestures are turn left, turn right, attention left, attention right, flap, stop left, stop right, stop both, attention both, start, go back, close distance, speed up and come near respectively.

We follow the experimental setting proposed in paper (Lin, Jiang & Davis, 2009). In the static background, we adopt Leave One Out Cross Validation (LOOCV). For dynamic background, the gestures acquired from the static background are used for training, while the gestures in dynamic background are used for testing.

In our experiment, videos are firstly cropped by tracking the region of interest through a simple correlation filter, and then all videos are resized to 32 × 24 × 45. The videos whose frames are less than 45 are appended with the last frame added some Gaussian noise. Similar to the previous discussion, we jointly determine the parameters (p₁, p₂, p₃, ω₁, ω₂) by 5-fold cross validation on training set, where p₁ is in the range of {2:4:32}, p₂ is in the range of {2:4:24}, p₃ is in the range of {10:4:45} and (ω₁, ω₂) are in the range as Table 2. The top 5 % optional parameter combinations of (p₁, p₂, p₃, ω₁, ω₂, ω₃) are listed in Table 8. And the samples on testing set are represented on $\mathcal{PG}\left(6,22,14|1080,1440,768\right)$. Table 9 shows that WSRC-PGM has higher performance compared with TB (Lui, 2011), Prototype-Tree (Lin, Jiang & Davis, 2009) and PM (Lui, 2012). The confusion matrix of our proposed approach with parameter combination 1 are given in Fig. 8.

Discussion

Through above experiments, we conclude that the proposed method is effective for video-based human gesture recognition. In experiments, the selection of parameters is a key step. We jointly selected optional parameters $\left(p_1^{\ast },p_2^{\ast },p_3^{\ast },{\omega }_1^{\ast },{\omega }_2^{\ast },{\omega }_3^{\ast }\right)$ on grid parameter set, through maximizing the average CRRs of 5-fold cross validation on training set. The parameter selection process is time-consuming because of the high dimension of parameter. This limitation may be solved by alternative iterations of optimization, through setting rational initial values based on prior information of data distribution. The reason is that the dimensions of parameter for each iteration can be reduced.