User emotion recognition and indoor space interaction design: a CNN model optimized by multimodal weighted networks

Self-attention mechanism

This section introduces the self-attention mechanism to enable the model to prioritize and retain crucial information within the training data. This mechanism operates by computing Query-Key-Value (QKV) associations, allowing the model to enhance its data comprehension by identifying and focusing on key elements within the input. As shown in Fig. 2, the input $H=[h_1,h_2]$, each row therein represents a corresponding input vector. The three-parameter matrix $W_q,W_k,W_v$ in the figure serves to transform the input H into the corresponding query spaces $Q=[q_1,q_2]$, $K=[k_1,k_2]$ and $V=[v_1,v_2]$, and the transformation process is as follows:

(1) $$[q_1=h_1{W}_q,q_2=h_2{W}_q]\to Q=H{W}_q$$

(2) $$[k_1=h_1{W}_k,k_2=h_2{W}_k]\to K=H{W}_k$$

(3) $$[v_1=h_1{W}_v,v_2=h_2{W}_v]\to V=H{W}_v$$

The computation of the output context, following the application of the attention mechanism, relies on the three feature matrices: Q, K, and V. This resultant output encapsulates the model’s understanding of the input H, constructed upon the foundation of the self-attention mechanism. Each line of this output $contex{t}_i$ signifies the model’s focal points at corresponding positions $h_i$ within the input sequence. The process of computing the context unfolds as follows:

(4) $$context=softmax{\left({\displaystyle \frac{Q{K}^T}{\sqrt{D_k}}}\right)}^V$$where $D_k$ is determined by the dimensionality of the vector Q.

Visual emotion recognition based on convolutional attention module

To address the inefficiency of the feature extraction process caused by the computational invariance of convolution, which results from its invariant parameters. We introduce a new convolutional attention module (AM) for the visual recognition task. According to user emotion recognition performance requirements in indoor spatial interaction, we detect the user’s mouth shape, facial expression, and body movement in three directions.

Given the inherent specificity of the visual recognition task, the expressions of a human face are often closely related to our five senses, particularly the eyes and mouth. The expressions of body movements are also associated with our hands, legs, neck, etc. To capture the specific details of the user’s face and limbs, we analyze the relationship between pixel points in the input image and adaptively compute the weight matrix based on the spatial relationships between these pixel points. The rich contextual relationships embedded between different channels of the input feature map are also considered to guide learning the dynamic attention matrix. By utilizing two sets of 1 * 1 convolutions and updating the parameters of the convolution kernel corresponding to each convolution window, the attention is computed while extracting the global features of the input feature map. Eventually, during the aggregation process, different weights are assigned to each pixel point of the input feature map, i.e., different attention is assigned to different input locations. This structure allows the module to be efficiently utilized in traditional convolutional architectures while being more efficient in feature extraction.

In Fig. 3, the AM module unifies the exploration of contextual information and the acquisition of the self-attention mechanism within a cohesive architecture. This integration maximizes the use of abundant contextual details among adjacent keys, effectively enhancing the learning process of self-attention. Consequently, this amalgamation enhances the module’s capacity to characterize and discern nuanced features within the output. Set the input feature map $In$ as $H\times W\times C$. First, we spatially convolve each key using the $k\ast k$ convolution set, and denote the learned context information as T, which reflects the static context information between different channels. The attention mechanism comprises two distinct components, both involving independent convolution operations. Initially, the acquired static context information T is concatenated with the query to enrich the characterization of information among contexts. Subsequently, employing a convolutional group facilitates the learning process of attention, resulting in the derivation of the attention matrix A1.

(5) $$A1=[T,Q]\otimes C1.$$

Meanwhile, to reinforce the input information to some extent, we use another independent $1\ast 1$ convolutional group for query using the same way of learning attention to get the attention matrix A2:

(6) $$A2=Q\otimes C2.$$

The two sets of attention matrices were subsequently fused to obtain the final attention matrix A3.

(7) $$A3=A1+A2.$$

The attention-based output T1 is obtained by aggregating A3 with V:

(8) $$T1=V\times A3.$$

Ultimately, the static information T undergoes fusion with the contextual representation T1, leveraging the dynamic attention mechanism, culminating in the final output of the module. Subsequently, a visual recognition network model is devised, incorporating the ATB module and its resultant output. This network encompasses four distinct sets of convolutions along with one fully connected layer. Each convolutional set comprises varying numbers of layers—specifically, 2, 2, 3, and 3 convolutional layers, respectively.

Multimodal weighted network coding classification model

After feature extraction from visual emotion recognition based on convolutional attention modules, the multi-modal weight network encoding classification model aims to deeply analyze the originally captured visual signals from various dimensions. This model comprehensively considers multiple features and automatically assesses the importance of each individual feature through the network, leading to more accurate and comprehensive emotional classification results.

Illustrated in Fig. 4, the model initially transforms user mouth shape information within the visual signal into textual information through mouth shape recognition. Subsequently, this process facilitates the derivation of the probability distribution for emotion classification:

(9) $$P_t=(t_1,t_2,t_3,t_4).$$

The user expression information f in the visual signal is used as an input to the overall model, and AM obtains a probability distribution for visual emotion classification:

(10) $$P_s=(s_1,s_2,s_3,s_4).$$

A probability distribution is also obtained based on the user’s body movement information in the visual signal:

(11) $$P_i=(i_1,i_2,i_3,i_4).$$

The prediction generated by each individual model represents the expression of a distinct feature in sentiment categorization. Dynamically assigning varying weights to the predicted outcomes of these different models entails dynamically altering the influence of each single feature within the overall categorization process. This dynamic adjustment aims to further refine the accuracy of sentiment categorization.

In this context, the probability distributions of each component from the individual models are utilized within the weighting network. These components play a role in linearly regressing the respective elements of the final classification probability, as described by Eq. (12).

(12) $${\phi }_n={\alpha }_{n1}{s}_n+{\alpha }_{n2}{t}_n+{\alpha }_{n3}{i}_n+g_n$$where n represents the emotion category. This article is a four-category situation (emotion categories are angry, happy, sad, and neutral), so $n=\{1,2,3,4\}$. $s_n,t_n,i_n$ denote the n-th category probability in $P_s,P_t,P_i$ respectively. ${\alpha }_{n1},{\alpha }_{n2},{\alpha }_{n3}$ and $g_n$ are the learnable weights and bias terms that balance the three. For example, $s_1$ denotes the probability that the emotion category obtained from the visual signal after AEM is angry in this article. ${\phi }_1$ denotes the probability that the category of anger is obtained after automatic weighting regression by the weighting network. Since it is a four-category case, the categorization probability $P=({\phi }_1,{\phi }_2,{\phi }_3,{\phi }_4)$ is obtained. After softmax normalization finally get the probability distribution of emotion classification of speech signal $\tilde{P}=({\tilde{\phi }}_1,{\tilde{\phi }}_2,{\tilde{\phi }}_3,{\tilde{\phi }}_4)$.

Evaluation indicators

The prediction evaluation metrics used in this article are accuracy (ACC), precision (Pre) and recall (R). Let true positive (TP) denote that positive samples are predicted as positive samples, false negative (FN) denote that positive samples are predicted as negative samples, false positive (FP) denotes that negative samples are predicted as positive samples, and true negative (TN) denote that negative samples are predicted as negative samples. Then:

(1) The correct rate metric indicates the proportion of positive and negative samples correctly categorized in the user’s visual emotion categorization recognition. The formula is as follows:

(13) $$\mathrm{A}\mathrm{C}\mathrm{C}={\displaystyle \frac{\mathrm{T}\mathrm{P}+\mathrm{T}\mathrm{N}}{\mathrm{T}\mathrm{P}+\mathrm{T}\mathrm{N}+\mathrm{F}\mathrm{P}+\mathrm{F}\mathrm{N}}.}$$

(2) The accuracy rate metric is for the user visual emotion classification recognition results and indicates the proportion of samples predicted to be positive that are positive. The formula is as follows:

(14) $$\mathrm{P}\mathrm{r}\mathrm{e}={\displaystyle \frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{P}}.}$$

(3) The recall metric is for the visual emotion raw labels and indicates how many positive classes in the visual signal sample were predicted correctly. The formula is calculated as follows:

(15) $$\mathrm{R}={\displaystyle \frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{N}}.}$$

Parameterization

This study involves experimentation conducted on the HMDB (https://zenodo.org/records/5758926, doi: 10.5281/zenodo.11046710) and CK+ (https://zenodo.org/records/11221351, doi: 10.5281/zenodo.11221351) datasets. The initial dataset is stored in. H5 format before commencing the experiments. The experimental setup utilizes a 2020Ti graphics card running the PyTorch framework. The training employs an SGD iterator over 300 rounds, incorporating a 0.9 momentum factor, 0.0005 weight decay, and a learning rate 0.1. During the training process, an adaptive adjustment of the model occurs every ten rounds after the initial 80 rounds, contingent upon the change in loss observed.

In the network architecture constructed, traditional convolution is exclusively employed solely within the initial layer of the first convolution set. All subsequent convolution operations are executed utilizing the AM module. Adopting traditional convolution in the initial layer augments the number of channels in the input feature maps, thereby facilitating model construction and subsequent processing via the AM module. The ultimate fully connected layer is tailored based on the fixed count of specific expression categories set at four.

In terms of data processing, this article loads raw data from HMDB and CK+ datasets. For video dataset HMDB, key frames or video fragments are extracted. For the image dataset CK+, the image file is loaded directly. At the same time, the pixel value of the image is normalized to the range of [0, 1] or [−1, 1] to eliminate the difference in brightness, contrast and other aspects of different images.

Comparative experimental analysis

The final recognition accuracy achieved by our model stands at 77.057% and 78.562% on the test sets of the HMDB and CK+ datasets, respectively. In our comparative analysis against current leading methodologies, presented in Fig. 5, our proposed approach demonstrates substantial advancements in expression recognition tasks relative to traditional and contemporary deep learning methods.

Specifically, on the HMDB dataset, the CNN model achieves a correctness rate of 62.44%. Comparatively, GoogleNet, VGG+SVM, and CNN+SVM outperform the CNN model, achieving correctness rates of 65.20%, 66.31%, and 71.20%, respectively. ResNet exhibits a correctness rate of approximately 72.7%, fairly comparable to our proposed model’s prediction. However, with an expansion in the test dataset, a declining trend is noticeable in the accuracy of ResNet due to the absence of a multimodal weight network, resulting in its incapability to accurately recognize user emotions amid increased variation in user features like limbs, mouths, and facial expressions within the dataset.

Conversely, our model’s performance advantage is less pronounced within the CK+ dataset, primarily comprising facial expression images. The accuracy of ResNet is particularly noteworthy, reaching 76.251%, exhibiting a marginal difference of approximately 2.3% from our model’s performance.

To comprehensively illustrate the expression recognition outcomes in this study, we present a longitudinal comparison across different models concerning the HMDB and CK+ datasets. The dataset undergoes classification, wherein the models’ classification accuracy for each expression category is compared. Figure 6 showcases the recognition accuracy rates of each model for the four visual emotion categories alongside their final average accuracy rates. Given the distinct experimental environments and hyper-parameter configurations, the comparison schemes for visual emotion recognition experiments include the CNN network, VGG+SVM, and ResNet. Notably, VGG+SVM achieves a recognition accuracy of 73%. However, the model proposed in this article attains the highest average accuracy of 74.75% across both datasets.

Regarding specific expression categories, the network models introduced in this article exhibit superior recognition rates for all expressions except for the anger and fear categories, where the recognition performance slightly trails that of the VGG+SVM network. Notably, our approach achieves a 91% recognition accuracy for the joy class of emotions. The substantial improvement observed in this article’s model for the sadness class of emotions is particularly noteworthy, showcasing a 5% enhancement over the best-performing ResNet.

To provide a clearer visual representation of accuracy progression during training, we present a graph that illustrates the changes in recall on the HMDB dataset for both the proposed method in this article and the ResNet network model, which shows a similar performance trajectory. As depicted in Fig. 7, it’s evident that for both models, the recall undergoes a rapid increase during the initial training phase, followed by a gradual convergence and eventual decline. However, a notable distinction arises from incorporating a multimodal weighting network in our proposed model.

In this context, the recall of the model outlined in this article showcases a delayed onset of decline compared to ResNet during training. Overall, the model presented in this article exhibits significantly superior performance compared to ResNet, mainly attributed to integrating the multimodal weighting network.

Ablation experiments

Figure 8 illustrates the confusion matrix corresponding to the network model constructed in this article. Diagonal squares denote the accurate recognition of the current emotion. In contrast, squares off the diagonal represent the probability of recognizing the emotion on the vertical axis as the emotion on the horizontal axis. The shading in the grid varies according to the values within each grid.

Notably, our model achieves a 62% recognition rate for emotions categorized as sadness compared to other expressions. This outcome stems from the significant similarity observed in daily life among fear, sadness, and anger. These expressions often share notable similarities in body and facial features, making it challenging even for neural networks to extract distinct features accurately, especially when data is limited. This similarity complicates accurate feature extraction by the neural network, contributing to the observed recognition rate in the sadness category.

Figure 9 displays the ablation experiments conducted on the glad dataset to assess and evaluate the neural network model’s performance. These experiments aimed to understand the impact of the self-attention mechanism and the multimodal weight network coding classification on the model’s efficacy by progressively removing them.

The results depict a discernible effect on recognition accuracy when removing these components. Model 3, derived from Model 4 by removing the self-attention mechanism, attains a recognition accuracy of 89%. Conversely, Model 2, obtained by exclusively eliminating the multimodal weight network coding classification from Model 4, experiences a substantial drop in recognition accuracy, achieving only 79%.

Furthermore, when both the self-attention mechanism and the multimodal weight network coding classification are removed from the proposed Model 4, resulting in Model 1, the recognition accuracy plunges to a minimum of 72%. These findings underscore the significant contributions of both the self-attention mechanism and the multimodal weight network coding classification to the model’s overall performance in visual emotion recognition.

Discussion

“Comparative experimental analysis” and “Ablation experiments” present a comprehensive analysis of the comparative performance of the proposed schemes outlined in this article. Examining three crucial performance metrics—correctness, precision, and recall—reveals the substantial superiority of the scheme introduced in this article, achieved through incorporating a multimodal weight network. This scheme attains a correctness rate of 77.057% and a precision rate of 74.75% on the dataset. Notably, the recall rate peaks at 94.7% during the training process, showcasing a significantly enhanced performance compared to CNN, VGG+SVM, ResNet, and other models.

The confusion matrix diagram for the network model constructed in this article further supports its accuracy in classifying gladness, achieving an impressive recognition rate of 91%. This holds immense significance, particularly in indoor space interaction design. Accurately recognizing emotions, particularly happiness, in visual emotion recognition among indoor users bears substantial value. This recognition aids designers in better-comprehending user needs and emotions, facilitating the provision of more personalized and comfortable designs. By monitoring users’ visual responses, designers can discern which design elements evoke positive emotions like pleasure, excitement, or comfort and negative emotions such as dissatisfaction, despondency, or fatigue. This profound understanding of user emotions empowers designers to optimize their designs effectively, aligning them more closely with users’ emotional requirements.

Visual, emotional recognition is a pivotal tool for designers to elevate both their designs’ artistic appeal and functionality. Exceptional interior design transcends mere functionality by evoking emotional resonance within users. Through the lens of visual emotion recognition, designers gain valuable insights into which design elements effectively convey desired emotions and atmospheres. This understanding empowers them to integrate these emotions seamlessly into their designs, enhancing both their creations’ artistry and functionality.

Moreover, visual emotion recognition technology enables designers to identify which colors, shapes, materials, and layouts best evoke specific emotional responses by testing and analyzing users’ reactions to design elements. This technology makes design more targeted and precise, helping designers achieve artistic and innovative outcomes. Through visual emotion recognition, this article allows designers to gain a deeper understanding of users’ emotional needs, thereby integrating more emotional elements into their designs. This emotional connection with users can enhance their sense of identity and satisfaction with the design, ultimately strengthening brand image and loyalty. While visual emotion recognition technology has made certain progress, it still faces numerous technical challenges. For instance, accurately identifying and understanding complex emotional responses, as well as addressing emotional differences across various cultures and backgrounds, is crucial. Since emotional differences in different cultural contexts may affect the results of visual emotion recognition, future research will continue to focus on visual emotion recognition in cross-cultural contexts, further exploring the interaction between emotions and cognition, and how this interaction affects users’ experience and satisfaction, ultimately providing design services that better meet user expectations.