Recognition of human action for scene understanding using world cup optimization and transfer learning approach

Challenges and motivation

Human action recognition involves identifying various physical actions carried out by people that include different human poses, diverse objects, and a cluttered background. Based on the type of input data, human action recognition can be basically divided into three categories: sensor-data-based, video-action-based, and still-image-based activity recognition. Sensor-based and video-based methods are less challenging since multiple frames are available regarding an action, i.e., more information is available regarding a subject’s motion. Therefore, the majority of research is happening in this field. Still, image-based action recognition, unlike the other two categories, is a more complex and challenging task due to a lack of spatial–temporal information. In this single frame, the action performed has to be identified.

In still images, the information regarding the actions needs to be extracted from the pose of the person, the surrounding multiple objects, or the distracting background within a single frame. The unavailability of a sufficient still image dataset is another major issue in the area of still image-based action recognition. Most state-of-the-art methods involve the use of data augmentation for increasing the amount of data, but this has the limitation of reducing the diversity of the data. Also, to improve the accuracy of the model, many researchers in this area use image annotation, which suffers from the disadvantage that it is time-consuming and costly. Owing to these several challenges, significantly less research has been done in the field of still image-based human action recognition. The aforementioned challenges motivated us to work in the area of human action recognition using still images.

Contribution

There are different methods in which we can perform action recognition, and using pre-trained deep CNN models has improved the performance of classification tasks significantly. In our work, we have used the concept of transfer learning to train some of the most powerful CNN architectures on human action recognition for a still image dataset. We have used the existing pretrained models of AlexNet, SqueezeNet, ResNet101, and DenseNet201 for action recognition on the Stanford 40 dataset. The initial weights of these networks are set by the pre-trained weights obtained from the large ImageNet dataset. These networks are then trained on one of the benchmark human action datasets, which updates the network’s weights and biases on each layer for the classification task. This reduces the constraints in training these deep layer network architectures from scratch. In this work, we haven’t used any data augmentation or annotation techniques, as in many existing works. We have tried to improve the performance of our classification task by using feature selection based on the world cup optimization algorithm. Also, instead of using machine learning classifiers, we have used deep neural networks as the classifier. Our contributions to this work are as follows:

• We propose a transfer learning-based human action recognition model. Human activity images are first preprocessed using the fuzzy color stacking technique.

• Pre-trained dense layers of AlexNet-fc8, SqueezeNet-pool10, ResNet101-pool5, and DenseNet201-avg_pool were used for feature extraction in the experimental simulation.

• For improving the accuracy of the model, we have proposed the World Cup optimization algorithm for selecting the best features. This algorithm does not fall into local minima and provides the best solution globally.

• We have modified the classification layer of DenseNet to classify twenty-four human action classes.

• On a benchmark dataset, we show that our unique human action recognition method using a dense feature selection approach outstrips the current state-of-the-art methods.

The remaining sections of our article are organized as follows: ‘Related Works’ presents some of the related works; ‘Proposed Methodology’ details the proposed methodology, which includes datasets, preprocessing, feature extraction, selection, and classification; ‘Performance Matrix Evaluation’ deals with the evaluation of various performance matrices; ‘Experimental Results and Discussions’ shows the results of our experiment; ‘Comparison of Our Method with Different Existing Methods’ discusses the performance of our four pretrained models; and ‘Conclusion and Future Scope’ includes the concluding remarks.

Proposed methodology

The proposed work for the identification of the objects and to understand the human activities in a scene is shown in Fig. 1. Here we have adopted the transfer learning approach of the convolutional neural network (CNN) (Lecun et al., 1989) to annotate the scene. Instead of using a pretrained deep model as a black box that learns and performs classification, we aim at filtering the features learned by the deep model, selecting the best features from them, and then feeding the selected features to the classifier part. It worked on human-object interaction scenes and classified them into different daily activity classes. In this model, we have carried out our work on a standard data set: the Stanford daily activity dataset (Yao et al., 2011). Figure 2 shows the flow diagram of our proposed human action recognition model. In the first step, input images are pre-processed using the fuzzy color stacking technique to remove noise. The pre-processed images are then fed to the pretrained CNN model. The image features are extracted by the deep CNN model. In our work, the pretrained models used are AlexNet (Krizhevsky, Sutskever & Hinton, 2012), SqueezeNet (Iandola et al., 2016), ResNet101  (He et al., 2016), and DenseNet201 (Huang et al., 2017). After feature extraction, we have carried out a feature selection process where the most relevant features are selected from the generated feature map, and only these predominating features are then fed to the classifier. Feature selection is done using the adaptive World Cup optimizer technique. The classifier is DenseNet’s (Huang et al., 2017) 201-dimensional CNN pretrained model. It classifies the input image into 24 different classes of human activity.

Dataset and preprocessing

We have carried out our proposed work on the Stanford 40-action dataset  (Yao et al., 2011; http://vision.stanford.edu/Datasets/40actions.html). This dataset includes various daily activities of humans in a scene. It is a collection of forty different actions performed by humans. Each category of the class consists of 180–300 action images, for a total of 9,532 scenes that relate to daily human activities. It showcases different human-object interaction scenes. We have collected 5,755 images with a total size of 565 MB. These 5,755 images were sorted and grouped into 24 classes of human actions. The number of images in classes varies from at least 182 to at most 315. Different classes of actions that we have used in our work include: applauding, blowing bubbles, brushing teeth, cleaning the floor, cooking, cutting vegetables, drinking, gardening, holding an umbrella, jumping, phoning, playing guitar, playing violin, reading, riding a bicycle, running, taking photos, texting a message, using a computer, washing dishes, watching TV, waving hands, writing on a board, and writing on a book. The number of image samples of each class used in our dataset and their size are shown in Table 2. As the first step, the different classes of image dataset are pre-processed using the stacking method of the fuzzy color technique to reduce the noise and filter the images. These stacked images are then fed to the feature extractor module.

Feature extraction using pre-trained models

The image features are extracted using the transfer learning technique. Pretrained deep CNN architecture models are used here to extract the different features of the image. As we know, the pretrained models are already trained with the existing large image dataset, so it becomes an easy task for these models to get trained with even a small dataset. With this knowledge transfer ability, these pretrained CNN models learn the image features with great accuracy. Here we have extracted the image features using the pretrained models: AlexNet (Krizhevsky, Sutskever & Hinton, 2012), SqueezeNet  (Iandola et al., 2016), ResNet101 (He et al., 2016), and DenseNet201 (Huang et al., 2017).

AlexNet (Krizhevsky, Sutskever & Hinton, 2012), a large deep convolutional neural network, was developed in 2012 to classify the one million images in an ImageNet (Deng et al., 2009) challenge into thousands of different classes. The architectural principle emphasized in the AlexNet model has remained a foundation for the CNN networks that are currently operational. The architecture shown in Fig. 3 can be viewed as a deeper and much larger network that consists of eight hidden weight layers, five convolutional layers used as feature extractors, and three fully connected layers used as a classifier. The first convolutional layer filters the input image using 96 kernels of size 11 × 11 with a stride of 3. The second convolutional layer filters the output of the first layer with 256 kernals of size 5 × 5. Between the first and second convolutional layers, there is maxpooling with stride 2 and size 3 × 3, and then a normalization is applied. The third, fourth, and fifth convolutional layers are connected to one another without any intervening pooling or normalization layers. The third convolutional layer has 384 kernals of size 3 × 3, the fourth convolutional layer has 384 kernals of size 3 × 3, and the fifth layer has 256 kernals of size 3 × 3. The fully connected layers have 4,096 neurons each, with ReLu in all layers.

SqueezeNet (Iandola et al., 2016) is a smart and small convolutional neural network architecture that has the accuracy level of AlexNet (Krizhevsky, Sutskever & Hinton, 2012) on the ImageNet (Deng et al., 2009) dataset with parameters 50 times fewer and 3 times faster. SqueezeNet is designed with three special high-level strategy designs: 1 × 1 filters replace the 3 × 3 filters and form smaller networks, reducing the number of input channels to 3 × 3 filters, and providing a late downsampling in the network to achieve large activation maps in the convolution layers. The Fire Module, shown in Fig. 4, is the key idea introduced in the squeezenet module for achieving these designs. SqueezeNet carries out delayed downsampling by using max-pooling with stride 2 after the first convolution (conv1), the fourth and eighth fire modules (fire4 and fire8), and the tenth convolution (conv10) units. They squeeze the features with the squeeze layer consisting of 1 × 1 convolutional layers and then expand with a combination of 1 × 1 and 3 × 3 convolutional layers. Thus, the Fire Module shown in Fig. 5 is made up of two sections: one section has a squeeze layer that consists of 1 × 1 convolutional layers, and another section has an expand layer that has a combination of 1 × 1 and 3 × 3 filters. The squeeze layer consists of three convolutional layers of size 1 × 1, and the expand layer is a combination of four convolutional filters of size 1 × 1 and four 3 × 3 each.

ResNet (He et al., 2016) is a residual deep neural network that handles the vanishing gradient problem. Residual networks use the concept of skip connections to resolve the problem of vanishing gradients. ResNet101 consists of 101 layers in its architecture. In ResNet, the convolutional layers are arranged like the traditional layers, but the original input is also added to the convolutional block output. This type of connection shown in Fig. 6 is known as a “skip connection” and can eliminate the problem of vanishing gradients. In this type of skip connection method, a few connections are skipped, and the final value will not be small quantity. The main idea behind ResNet is to make input and output equal, which means the network learns from the difference between the input and output. In a traditional network, we train the algorithm based on the output, but in a residual network, the algorithm is trained on the basis of the function of the input.

DenseNet (Huang et al., 2017) is a dense convolutional neural network developed to improve the accuracy of a model by handling the problem of vanishing gradient. In deep networks, we have a lot of hidden layers between the input layer and output layer, and due to this long path, information vanishes before it reaches its destination. But DenseNet overcomes this vanishing gradient problem and offers high accuracy compared to other CNNs by simply connecting every layer directly to one another. Dense has different variants, among which is DenseNet201, which means there are 201 layers in the network. Input is given to a filter structure as shown in Fig. 7, whose size is 7 × 7 and uses a stride of 2. Then comes a pooling layer of filter size 3 × 3 and stride 2. The special structure found in DenNet is known as “dense blocks”. There are four dense blocks in the structure, and inside each block we have convolution layers of size 1 × 1 filter size and 3 × 3 filter size. The number of convolutional layers varies in each block.

Inside a dense block, each convolutional layer is connected to every other layer, and each layer receives the feature map from the previous layer. This helps each layer add some new features on top of the feature maps obtained in the previous stages. Thus, the new features get concatenated onto the existing feature map. We can concatenate the feature maps only if the size of the feature maps is the same. So inside each dense block, the size of the feature maps from each layer should be the same. For this reason, downsampling is not present in dense blocks. Downsampling is handled by another block in DenseNet, referred to as a “transition layer”. After every dense block, there is a transition layer. Each transition layer consists of a convolutional layer of size 1 × 1 and an average poling layer of size 2 × 2 and stride 2. The transition layer downsamples the feature maps obtained from the dense block. After the last dense block, there is a global average pool layer of size 7 × 7. Finally, there is a fully connected layer with a softmax function, and we get a classified output.

In our proposed work, we have used the pre-trained deep CNN models: AlexNet, SqueezeNet, ResNet101, and DenseNet201, for extracting the features from the preprocessed Stanford40 dataset. The different pretrained models and their layers that are selected for feature extraction are presented in Table 3. The input size that corresponds to the image input layer for AlexNet and SqueezeNet is 227 × 227 × 3, and for other models, the size is 224 × 224 × 3. AlexNet has three fully-connected (FC) layers, and the features are extracted from the fc8 layer, which is the third FC layer of AlexNet. The fc8 layer extracts 4,096 features per image for the 5,755 images in our dataset. SqueezeNet has 10 learnable layers, and the global average pooling layer, pool10, is selected to extract the features. For each of the 5,755 images in the dataset, the pool10 layer extracts 1,000 features. ResNet101 is a widely used model and has 101 deep layers. The fully connected (FC) layer, fc1000, is selected for feature extraction. This layer extracts 2,048 features for each of the 5,755 images. DenseNet201 is a powerful deep model with 201 deep layers. The global average pooling layer, avg_pool, is selected for feature extraction. For each of the 5,755 images in the dataset, 1,920 features are extracted by the avg_pool layer.

Feature selection using world cup optimization

The selection of features plays a vital role in the ability of a model to perform correct classification in the minimum amount of time. The pretrained deep CNN models get trained by the image dataset and generate a large feature map collection. These features include the details of an image that could be of less relevance. The majority of the extracted features may not be relevant for classification and may have an impact on the model’s accuracy. In order to solve this issue, a feature selection process can be used to select the best subclass from the original complete features. A general deep learning model classifies the images based on complete features learned by all the layers of the network, and the fully connected layer classifies these images. In our model, we have proposed a feature selection process before feeding these complete features learned by the network layer to the classification layer. Here, we used an adaptive world cup optimization algorithm (WCO) (Razmjooy, Khalilpour & Ramezani, 2016) to select relevant features from the feature bank. The existing search optimization algorithms suffered from the drawback that they fell into local minima. As inspired by the FIFA World Cup football competition, a new optimization algorithm was developed to optimize the performance of the mathematical functions. WCO is based on global heuristic search rather than the local minima search algorithm.

WCO algorithms work as follows: In this optimization algorithm, initial variable values are termed “team”, which forms clusters forming countries. After the teams have been formed, the continents’ scores are calculated. This is accomplished by calculating the mean and standard deviation for each continent, as shown below.

(1)${\displaystyle \text{Mean}\left(X\right)=\frac{1}{n}\sum _{i=1}^{n}Xi}$ (2)${\displaystyle \text{Standard Deviation}\left(X\right),\sigma =\sqrt{\frac{1}{n-1}\sum _{i=1}^n\left[{\left(Xi-\text{mean}\left(X\right)\right)}^2\right]}}$ (3)${\displaystyle \text{Rank}=\frac{\left(\beta \ast \sigma +\text{mean}\left(X\right)\right)}{2}.}$

Where ‘n’ denotes the total number of members of continent ‘X’, ‘ β’ is the decrease or increase coefficient of ‘ σ’, and lies between [0, 1]. Based on the mean and standard deviation, each objective function is evaluated and ranked as shown below. With the ranks calculated, all the continents are sorted into two groups, one with good records and the other with poor records. The competition is then evaluated by first finding the teams with the lowest value on each continent. Later, the minimum among all the values is evaluated. If this evaluation is satisfied, then the process is terminated; otherwise, based on the top-value country record, the population is newly generated, and the process repeats till the challenge is completed. The World Cup optimizer model works by twice filtering the best solution. This kind of filtering is used for better feature selection but is not provided by other optimization algorithms. Rather than applying the complete features to the classifier model, we have given the most superior and relevant features for classification. This reduces the misleading errors, increases the accuracy of the model, and enhances its efficiency.

In our work, by using an adaptive world optimizer, we select the best features among all the input features obtained from the pre-trained model. The steps involved in feature selection that we have used in our work are summarised as follows:

• The input to the algorithm is the extracted features and its labels.

• By using the fitness function, the weight for each feature attribute is calculated.

• The fitness calculation and position update are carried out by setting the probability of individual learning at 0.85 and the probability of exploration learning at 0.1.

• The features that have a high fitness function (weight) are selected, and others are rejected.

• Finally, the index of the selected attributes is given as the output of the optimization algorithm.

In this work, the output of the pre-trained deep model, which are the features extracted from our training dataset, is considered for feature selection and separated into populations. Each population is provided with random samples of extracted image features. The most superior features are then selected by ordering the random features based on the generated cost function and fitness function. We have the features as the input and the corresponding labels as the target. For each column of features, we have the labels, and each particular feature is classified, with the probability of individual learning at 0.85 and the probability of exploration learning at 0.1, which predicts the label value. This predicted value is then compared with the input label values, and an error is calculated. Using this error value, we calculate the cost function, which is given as follows: (4)${\displaystyle \text{Cost}=0.99\ast \text{error}+0.01\ast \frac{\text{number of selected features}}{\text{total number of features}}.}$

Here the fitness function is minimization of error, i.e., the features having the lowest error value will be chosen and others will be discarded. For each population, we generate the cost function, and the fitness function is updated. Based on the updated fitness function, we update the randomly generated initial population. Each time the cost function is calculated, it is compared with the previous value. The one that has the minimum cost function is updated as the fitness value, and the others are discarded. This is repeated until the fitness function converges and the features with the highest fitness value are selected and others are rejected. This double filtering technique excels at world cup optimisation compared to other optimisation algorithms.

The feature selection process increases the performance of our model by removing irrelevant features. Each of our pre-trained models extracts more than 5,700,000 features from our dataset. The total number of features extracted by each model and their corresponding count of selected features using the world cup optimization algorithm are presented in Table 4. Comparing with the number of features extracted and selected by each feature extraction layer of the pre-trained models, AlexNet initially extracted 23,572,480 features that were reduced to 11,648,120 features upon feature selection. The feature selection reduced the 5,755,000 features extracted by SqueezeNet to 2,917,785 features. Initially, 11,786,240 features were extracted by ResNet101, and 5,881,610 of the best features were selected. DenseNet 201 initially extracted 11,049,600 features, and the 5,588,105 best features were selected by feature selection. These selected best features are then used for the classification into twenty-four classes using a modified deep learning model, DenseNet 201.

Feature classification using densenet

The most relevant features selected by the World Cup Optimization algorithm (Razmjooy, Khalilpour & Ramezani, 2016) are fed to the classifier module. Here, the pretrained deep DenseNet network is used to classify the image. The fully connected layers perform the classification by using the superior features obtained from the feature selection module. For classifying our 24 classes, we modified the fully connected layers and pooling section. Rather than classifying based on all features, the network was designed to classify based on superior features obtained through World Cup optimisation. Here the input to the classifier is the selected features from the optimizer module. The dense network architecture that we have used as a classifier model in our research is shown in Fig. 8. The description of the network model is as follows: The input sequence layer accepts a size of 971. The network consists of convolutional filter layers of size 11 × 11 × 96, batch normalisation, and ReLU. This is followed by a 2 × 2 max pooling layer with a stride of [1 1], batch normalisation, and the ReLU layer. These layers are followed by the layer of a convolutional filter of size 1 × 1 × 64, batch normalisation, and the ReLU layer. The last layers are followed by a 1 × 1 max pooling layer with a stride [1 1], global average pooling, a fully connected layer for 24 classes, and a softmax layer with classification output.

In our work, we have also modelled the feature extraction and classification by the DenseNet201 directly without using the feature selection module. The deep model densenet classifier has a default input size of 224 × 224, whose weights were already adapted to the ImageNet dataset and get updated with our image dataset. DenseNet201 has 201 layers with four dense blocks and three transition blocks. It has an initial 7 × 7 convolutional layer of 64 channels followed by batch normalisation, ReLU, and a 3 × 3 max pooling layer with stride 2. We removed the last three layers of the pre-trained model, i.e., ‘fc1000’, ‘fc1000_softmax’, and ‘ClassificationLayer_fc1000’, and defined our new classification layers. These include a fully connected layer, a dropout layer, and a softmax layer. A fully connected layer is designed for 24 classes and is specified with a weight learn rate factor of 20 and bias learn rate factor of 20. We have used a 0.5 dropout factor for the dropout layer. Based on the modified classifier layers, our model classifies the images into twenty-four different daily activity classes. The fine-tuned DenseNet classifier layers are shown in Fig. 9. The classifier module with the feature selection module performed better than the model without the feature selection module.

Performance analysis—feature extraction, selection and classification

In our proposed work, we have adopted the pretrained deep learning models to train our pre-processed Stanford40 dataset, which contains 24 classes of daily human activities. The pretrained AlexNet, SqueezeNet, ResNet101, and DenseNet201 architecture models extract the features from the image. The world cup optimizer algorithm was used to filter and select the most superior and relevant features from the feature map, which were then fed to the DenseNet classifier layer. Figs. 14, 15, 16, 17, 18 and 19 show the performance of different feature selection CNN models.

The accuracy plot of different CNN models is shown in Fig. 14 with comparable accuracy for DenseNet and ResNet compared with AlexNet and SqueezeNet models.

Figure 15 depicts the sensitivity measure analysis of the four models. It shows the best response for the densenet and resnet models and the lowest for the SqueezeNet model.

Figure 16 depicts specificity, which is a measure of correctly predicting the wrong classes. It shows a comparable good response for all the models.

The precision measure of a classifier model indicates how many actual predictions made are correct out of the total count of predictions. Figure 17 shows the precision values of the different models, with better response for both the DenseNet and ResNet models.

According to F1-score values plotted in Fig. 18, the DenseNet and ResNet models performed better than the AlexNet and SqueezeNet models.

The false-positive rate was also calculated for the four architecture models, and the results are shown in Fig. 19. It indicates that the DenseNet model and the ResNet model have a comparable low FPR, while AlexNet and SqueezeNet have a higher value.

The performance analysis of the proposed model having DenseNet201 and ResNet101 as the feature extractors shows a better response compared to other pretrained feature extractors. Here we have also applied the dataset to train the pretrained DenseNet201 deep model without feature selection, where the image features are learned by the deep layers of the DenseNet and the images are classified into 24 different classes. We have evaluated the performance of the DenseNet classifier with and without using the feature selection process. Figure 20 shows that our proposed model with a feature selection process shows a good response for the system parameters like accuracy, sensitivity, precision, and F1-score compared to the ordinary densenet model without feature selection.