UMEDNet: a multimodal approach for emotion detection in the Urdu language

Audio and text modality in emotion detection

Siriwardhana et al. (2020) investigate the feasibility of using modality-specific “bidirectional encoder representations from transformers (BERT)-like” pre-trained self-supervised learning (SSL) architectures to represent voice and text modalities for multimodal speech emotion identification. They demonstrate experimentally that jointly fine-tuning “BERT-like” SSL designs achieves state-of-the-art outcomes on three publicly accessible datasets (IEMOCAP, CMU-MOSEI, and CMU-MOSI). The authors also compare two approaches to merging voice and text modalities, demonstrating that a simple fusion mechanism can outperform more complicated ones when employing SSL models. Priyasad et al. (2020) presents an attention-based multi-modal emotion recognition model based on both acoustic and textual data. A SincNet layer hierarchically structured with parameterized sinc functions and band-pass filters is used to extract the auditory features that are fed into a deep convolutional neural network (DCNN). This approach learns filter banks optimized for emotion recognition. The regarded experiments prove that the proposed system is superior to existing approaches.

The main goal of Hazarika et al.’s (2018a) feature-level fusion method is to improve the integration and interaction of the multiple inputs with the proposed self-attention-based framework. They perform a noise stability test to once again demonstrate performance in noisy conditions. These outcomes endorse the optimistic hypothesis introduced in this method and present it as being a drastic improvement within the area of feature-level fusion and its applicability to more extensive multimodal issues. Padi et al. (2022) incorporated a neural network system for emotion recognition that is based on a late fusion of transfer-learned and fine-tuned models of speech and text data. In the case of speech signals, they fine-tune a residual network (ResNet) based model that is trained initially on a large-scale speaker recognition task via transfer learning. They also apply a spectrogram augmentation technique to recognize emotions from speech. For text, they use a fine-tuned BERT for representing and recognizing emotions. Additionally, the proposed system performs the fusion of the ResNet and BERT-based model scores employing a late fusion approach to enhance the performance.

Video and audio modality in emotion detection

Abu Shaqra, Duwairi & Al-Ayyoub (2023) provides a multimodal system for identifying and labeling emotional expressions in an Arabic-based dataset. They first assessed the usefulness of audio and video data separately and later revealed what happened when they were combined into a single model. Emotion identification algorithms that rely on audio data show considerable performance improvements when speaker gender is known in advance. By contrasting audio-based and visual-based systems, they see that each model excels at recognizing a different set of emotional labels. Chen et al. (2014) conducted an exploratory investigation by implementing state-of-the-art emotion recognition techniques in the analysis of interview videos. They developed a video corpus by inviting actors to read three types of scripts related to call center work composure and employee competencies. This corpus was used to assess the ability of Emotient’s FACET toolkit to identify emotions and evaluate the actors’ performances. Additionally, they utilized the Linear Discriminant Classifier (LDC) emotional speech corpus and several open-source tools to construct speech emotion classifiers for four emotion categories.

Chong, Jin & He (2019) proposes EmoChat, an online chatting system that adopts an innovative feature to tag users’ emotions during a chat conversation without being prompted. EmoChat utilizes both facial expressions and text-based inputs to evaluate users’ emotions comprehensively. To fuse these two complementary modalities, they present an information entropy-based method. To increase the accuracy of recognizing emotions they use the Hidden Markov Model (HMM) which takes into account contextual information. Mamieva et al. (2023) presents a new multimodal emotion recognition system using both facial expression and speech data utilizing attention-based frameworks. This technique compares the most relevant features of the facial and spoken parts extracted using different encoders. This strategy does not have the weaknesses of uni-modal systems because it uses additional information from the other modality and improves the recognition of emotions. These modalities are integrated by an architecture called a fusion network to generate a multimodal feature vector which is followed by a classification layer to identify the affective state resulting from the computation.

Tzirakis et al. (2017) introduces an emotion recognition system based on auditory and visual modalities. They employ a convolutional neural network (CNN) to extract features from the speech. For the visual modality, a deep residual network (DRNN) is used. The experiments on the uni-modal modality also indicate that proposed models yield substantially higher accuracy on the test set as compared to the other models accessing the RECOLA database including those introduced to the AVEC2016 challenge, thus proving the effectiveness of learning features that are more suitable for the task under consideration. Moreover, they have realized that their multimodal model outperforms others, not only in terms of valence but also in terms of arousal.

Video, audio, and text modalities in emotion detection

Using multimodal feature extraction and hierarchical modeling, Hazarika et al. (2018b) proposed the Interactive Conversational Memory Network (ICON), a framework for analyzing videotaped conversations that can detect multiple emotions. Using the memories made this way, they can predict how someone will feel in a segment of the video conversation. Lai, Chen & Wu (2020) presents a model for distinguishing across contexts using different contextual window sizes based on recurrent neural networks (DCWS-RNNs). The model employs four RNNs, each using a unique contextual window size. These window widths can represent various contexts’ implicit weights. In addition, four RNNs are used to autonomously model the different situations into memory. Then, these recollections and the test segments are combined using attention-based multiple hops.

Pan et al. (2020) investigated a hybrid fusion method known as a multimodal attention network (MMAN). They propose Convolutional Long Short-Term Memory with Multi-Modal Attention (cLSTM-MMA), a novel multimodal focus mechanism that encourages attention across three modalities and selectively integrates information. Late fusion involves combining cLSTM-MMA with other uni-modal sub-networks. The results show that using visual and textual clues greatly enhances the accuracy of emotion recognition in spoken language. Cevher, Zepf & Klinger (2019) present an in-car experiment they conducted to test emotion identification from spoken interactions across three different modalities. They employed commercially available audio and facial emotion identification technologies. They compared them to a neural transfer learning strategy for emotion recognition from text, which used resources from other domains. Using transfer learning makes it possible for models built using data from different fields to succeed.

Schmitz, Ahmed & Cao (2022) work with audio, text, and video data types and their combinations. In this case, text only performs worst in bias metrics yet contributes significantly to the models scores, questioning the need for multimodal solutions if one is seeking both fairness and high accuracy at the same time. Audio has slightly better results in terms of accuracy and fairness compared to video whose performance is very poor. The combination of audio and video stands out from the other bimodal models while incorporating video decreases both predictiveness and fairness for trimodal models. Caschera, Grifoni & Ferri (2022) proposes a method which generalizes the concept of multimodal data based on the sequence of features derived from facial expressions, voice, gestures, and natural language processing in linguistic terms. In this article, each of the emotions was modeled as an HMM, and these were trained and tested using multimodal samples containing seven basic emotions. The classification rates of the emotions were studied and it was observed that the experiment yielded good results.

Illendula & Sheth (2019) utilizes the significance of various modalities derived from social media posts for the task of emotion classification with the help of advanced deep learning architectures. They have described a multimodal emotion classification method utilizing emojis, textual, and visual elements. They have shown that the three modalities convey different information to produce emotions.

To summarize and overview existing research are shown in Table 1. In overview, there have been some studies on the effectiveness of multimodal approaches for emotion detection, and the results have been mixed. Some studies have found that multimodal approaches can improve the accuracy of emotion detection, while others have found little or no improvement. Developing evaluation resources constitutes a crucial difficulty within this discipline. Development of natural language processing (NLP) tasks depends on benchmark corpora for conducting both development and evaluation. Emotion detection corpora exist for languages like English and French but there are no such resources available for South Asian languages specifically Urdu. The domain of emotion detection in Urdu lacks any publicly accessible dataset at present. The lack of standard Urdu language resources requires us to develop a corpus first which we will utilize in constructing our multimodal emotion detection framework.

In addition, Urdu does not have pre-trained multimodal embeddings and thus it is not easy to develop emotion detection models. To overcome this limitation, we propose a new multimodal architecture incorporating text, speech, and video embeddings. This article aims to present a robust multimodal emotion detection system made up of machine learning and deep learning techniques. When emotion prediction deals with languages with low resources, it faces serious challenges scarcity of publicly available datasets. In contrast to high-resource languages, such as English, there are no such corpora or well-set models for deep learning to effectively train in low-resource languages. In this case, lack of resources impedes progress in the unimodal and multimodal emotion recognition tasks. Given these challenges, our study contributes to the research on low-resource language for emotion recognition by leveraging a multimodal approach. With our approach, we aim to overcome dataset limitations and develop a model for multimodal emotion detection.

Video features extraction

Video feature extraction is important for understanding and analyzing video data to extract important information from frames. For tasks like emotion detection, identity verification, and behavior analysis, faces appearing in videos are important. We focus on extracting features of the facial regions of individuals appearing in videos. To do that, each video is processed frame by frame, extracting facial features from faces detected in the video. We aim to capture high-dimensional representations that capture the spatial and temporal aspects of the video, so video data can be robustly modeled using the emotion detection system. For face detection, the MTCNN (Multi-task Cascaded Convolutional Networks) (Xiang & Zhu, 2017) is used to locate facial regions within each frame efficiently. After face detection, facial embeddings are generated using the FaceNet model (Schroff, Kalenichenko & Philbin, 2015), which is a pre-trained deep learning model. The FaceNet model extracts rich and discriminative features, encoding important facial attributes and expressions essential for accurate emotion detection and other facial analysis tasks. The purpose of MTCNN is to detect faces and align them correctly for additional processing. The face recognition model, FaceNet converts faces into embedding data for identification and verification tasks. The latest DINOv2 (Oquab et al., 2023) model operates as a self-supervised vision transformer that provides generalized image representation learning and delivering optimal outcomes for object recognition and segmentation tasks but does not target face-related processing. MTCNN and FaceNet specialize in facial analysis but DINOv2 is a comprehensive vision model capable of processing different images beyond face data.

Video features processing

A transformer-based model is adopted in this study to evaluate facial embeddings derived from frames in a video for the goal of emotion detection as shown in Fig. 2. The frame processing model commences by feeding input facial embeddings to a higher-dimensional hidden space. This projection is followed by positional encoding which captures temporal information important to analyse video data. The transformer encoder section, which is the core of the whole model, includes multiple layers and attention. The given architecture is good at capturing fine-grained temporal relationships between frames, which makes it possible to interpret facial expressions. The model incorporates padding and masking mechanisms to effectively manage variable-length sequences, ensuring that only valid frames contribute to the output. This makes it possible for the model to incorporate sequenced information from videos which is important for the accuracy of the detected emotion. By including facial embeddings into the transformer architecture, there are lots of advantages in utilizing temporal dynamics, which is beneficial for models that analyze facial expressions in video. After extracting facial embeddings, we combine these embeddings with other modalities to predict the emotion.

Audio features extraction

To begin with, audio from the video files is obtained using the MoviePy library. It involves extracting video files and splitting audio streams from the file for analysis. For feature extraction, we fine-tune the Wav2Vec 2.0 model (Babu et al., 2021) which is a self-supervised learning model used for automatic speech recognition (ASR). Designed by Facebook AI researchers, Wav2Vec 2.0 introduces an effective way of improving the ASR since it does not rely on large labeled datasets. The model is pre-trained on unlabeled audio data, where it learns to generate latent speech representations by masking parts of the input and predicting these masked segments. This process is similar to masked language modeling in NLP, enabling the model to perceive phonetically and linguistically relevant features. By fine-tuning Wav2Vec 2.0, we extract the embeddings that define key aspects of the signal while including phonetic and linguistic information. These feature embeddings are used for tasks such as emotion recognition, which are important to create an adequate base for speech-based applications.

To extract meaningful audio embeddings for Urdu, we fine-tuned Wav2Vec2 with an Urdu vocabulary for feature extraction. During fine-tuning, we froze the first six transformer layers and kept pre-trained representations while saving on the computational overhead. Hidden dropout and mask time probability were optimized as key hyperparameters to make training more stable and adapt the model. It was also evaluated on performance results, using the word error rate (WER). Wav2Vec2 may introduce biases for high-resource languages which may negatively affect Urdu performance. This study does not directly address bias, but acknowledges this limitation and suggests future work to explore the mitigation strategies for improved multimodal emotion detection.

The Wav2Vec2 model was selected due to its excellence in speech recognition tasks with language-specific functionality and rich representation capabilities. Wav2Vec2 behaves as a self-supervised model that can effectively learn speech features automatically while using unlabeled data with pretraining multilingual datasets. The model accuracy improves because it undergoes fine-tuning specifically for an Urdu dataset implementation. While other models like Hidden-Unit BERT (HuBert) (Hsu et al., 2021) pivot their attention on the phoneme level, Wav2Vec2 demonstrates better performance in automated speech recognition from end to end. The model provides remarkable performance and efficiency.

Audio features processing

In the architecture, all audio features are passed through a feed-forward network (FFN) as shown in the Fig. 3. In this FFN, we apply two fully connected layers to expand the input dimension of the audio features into a hidden space. They use a linear transformation followed by rectified linear unit (ReLu) activation, introducing non-linearity into the model. To address over-fitting of data, we used dropout regularizations. The last outcome of the network is a feature map of the audio that has been transformed and encoded for combination with other modalities.

Text features extraction

Extracting features from audio data is accomplished by using automatic transcription that transcribes speech into text form. This uses the Whisper (Radford et al., 2023) model which has gained recognition for accurate and effective results in text transcribing. We utilize the large variant of Whisper, to achieve the highest possible quality of transcription. After transcribing the audio into text, we can analyze its linguistic features through natural language processing. During this phase, XLM-RoBERTa (Li, He & Xu, 2021) is used for feature extraction from the whisper transcriptions. The XLM-RoBERTa large model utilizes its ability to improve the quality of word embedding across multiple languages. XLM-RoBERTa is capable of capturing contextual relationships in words, therefore it is useful in several linguistic tasks. In addition, we employ this model to extract relevant information from text. XLM-RoBERTa is a powerful language representation model for capturing contextual information given the distribution of words in the text. The connected Urdu text is passed to the XLM-RoBERTa model which produces feature vectors that accurately represent the semantics of the text. Since Urdu is a low-resource language compared to English or other widely used languages, it should not come as a surprise if this bias affects textual embeddings. Referring to the architectural details, we trained on XLM-Roberta-large with 24 transformer layers, 1,024 dimensions per layer, and 16 heads per layer. To be compatible with the Urdu text processing, we did not modify the context window size. To tokenize and process Urdu transcriptions with padding and truncation to a fixed maximum length.

Text features processing

Text features are further processed through another feed-forward network as shown in Fig. 4. This network has the same structure as the one used for audio, having fully connected layers coupled with ReLU activation and dropout regularization. Textual features are learned in the network and transformed into a hidden state. The FFN generates a fixed-shape vector that can be used for projection into a shared space, similar to other modalities, for emotion detection.

Multimodal feature projection and emotion prediction

Our model employs projection functions to map the features of various modalities (video, audio, and text) to shared common spaces represented in Fig. 5 to facilitate more effective multimodal alignment and fusion. For every modality M, we define a projection function $P_M$ that projects features from its original feature space into a common feature space. This necessary projection function enables the interaction between modalities and improves the overall performance of the emotion detection task.

Let $\mathbf{V}\in {\mathbb{R}}^{d_V}$, $\mathbf{A}\in {\mathbb{R}}^{d_A}$, and $\mathbf{T}\in {\mathbb{R}}^{d_T}$ represent the feature vectors from video, audio, and text modalities, respectively, where $d_V$, $d_A$, and $d_T$ denote their respective dimensions. To ensure compatibility and facilitate effective fusion, each modality is projected into a shared latent space of dimension $d$ using the following projection functions:

(1) $${\mathbf{V}}_{proj}=P_V(\mathbf{V})=f({\mathbf{W}}_V\mathbf{V}+{\mathbf{b}}_V),{\mathbf{W}}_V\in {\mathbb{R}}^{d^{\mathrm{\prime }}\times d_V},{\mathbf{b}}_V\in {\mathbb{R}}^{d^{\mathrm{\prime }}}$$ (2) $${\mathbf{A}}_{proj}=P_A(\mathbf{A})=f({\mathbf{W}}_A\mathbf{A}+{\mathbf{b}}_A),{\mathbf{W}}_A\in {\mathbb{R}}^{d^{\mathrm{\prime }}\times d_A},{\mathbf{b}}_A\in {\mathbb{R}}^{d^{\mathrm{\prime }}}$$ (3) $${\mathbf{T}}_{proj}=P_T(\mathbf{T})=f({\mathbf{W}}_T\mathbf{T}+{\mathbf{b}}_T),{\mathbf{W}}_T\in {\mathbb{R}}^{d^{\mathrm{\prime }}\times d_T},{\mathbf{b}}_T\in {\mathbb{R}}^{d^{\mathrm{\prime }}}.$$

Here, ${\mathbf{W}}_V$, ${\mathbf{W}}_A$, and ${\mathbf{W}}_T$ are learnable weight matrices that project the features into a shared space of dimension $d^{\mathrm{\prime }}$. The bias terms ${\mathbf{b}}_V$, ${\mathbf{b}}_A$, and ${\mathbf{b}}_T$ ensure flexibility in projection. The activation function $f$ (e.g., ReLU) introduces non-linearity, enabling the model to capture complex relationships within each modality. By projecting all modalities into a shared space of the same dimension $d^{\mathrm{\prime }}$, we ensure dimension compatibility and facilitate meaningful cross-modal interactions.

Cross-modal projection and dimension compatibility

To further enhance the multi-modal feature fusion, we introduce cross-modal projections, where each modality is mapped into spaces aligned with other modalities. Specifically, video features are projected into both audio and text feature spaces, while audio and text features are projected into a video space:

(4) $${\mathbf{V}}_{proj}^A=P_V^A(\mathbf{V})=f({\mathbf{W}}_V^A\mathbf{V}+{\mathbf{b}}_V^A),{\mathbf{W}}_V^A\in {\mathbb{R}}^{d^{\mathrm{\prime }}\times d_V},{\mathbf{b}}_V^A\in {\mathbb{R}}^{d^{\mathrm{\prime }}}$$ (5) $${\mathbf{V}}_{proj}^T=P_V^T(\mathbf{V})=f({\mathbf{W}}_V^T\mathbf{V}+{\mathbf{b}}_V^T),{\mathbf{W}}_V^T\in {\mathbb{R}}^{d^{\mathrm{\prime }}\times d_V},{\mathbf{b}}_V^T\in {\mathbb{R}}^{d^{\mathrm{\prime }}}$$ (6) $${\mathbf{A}}_{proj}^V=P_A^V(\mathbf{A})=f({\mathbf{W}}_A^V\mathbf{A}+{\mathbf{b}}_A^V),{\mathbf{W}}_A^V\in {\mathbb{R}}^{d^{\mathrm{\prime }}\times d_A},{\mathbf{b}}_A^V\in {\mathbb{R}}^{d^{\mathrm{\prime }}}$$ (7) $${\mathbf{T}}_{proj}^V=P_T^V(\mathbf{T})=f({\mathbf{W}}_T^V\mathbf{T}+{\mathbf{b}}_T^V),{\mathbf{W}}_T^V\in {\mathbb{R}}^{d^{\mathrm{\prime }}\times d_T},{\mathbf{b}}_T^V\in {\mathbb{R}}^{d^{\mathrm{\prime }}}.$$

Each projection matrix ${\mathbf{W}}_X^Y$ ensures that all projected features share the same dimensionality $d^{\mathrm{\prime }}$, enabling direct comparison and fusion. This alignment of feature spaces is critical to ensure compatible and meaningful interactions between modalities.

Feature fusion

Following cross-modal projections, we concatenate all aligned feature representations to construct a unified multimodal embedding:

(8) $${\mathbf{F}}_{combined}=[{\mathbf{V}}_{proj}^A,{\mathbf{V}}_{proj}^T,{\mathbf{A}}_{proj}^V,{\mathbf{T}}_{proj}^V].$$

This concatenation-based fusion approach is applied for the following reasons:

• All features are projected into a shared latent space of dimension $d$, ensuring compatibility and meaningful interactions across different modalities.

• Unlike averaging or attention-based approaches, concatenation preserves distinct modality characteristics, preventing the loss of critical unimodal information.

• By maintaining separate feature streams, the model can learn complex inter-modal dependencies in subsequent layers without enforcing any predefined relationships between modalities.

Since the final fused feature vector ${\mathbf{F}}_{combined}$ resides in ${\mathbb{R}}^{4d}$, it is then processed by a classifier:

(9) $$\hat{y}=\mathrm{C}\mathrm{l}\mathrm{a}\mathrm{s}\mathrm{s}\mathrm{i}\mathrm{f}\mathrm{i}\mathrm{e}\mathrm{r}({\mathbf{F}}_{combined}).$$

The classifier, typically comprising fully connected layers, learns a discriminative multimodal representation that optimally combines the projected features to make accurate predictions.

Corpus collection

In all NLP approaches, benchmark corpora are vital instruments that derive methodologies and tools. This research is specifically dedicated to contributing to the field of multimodal emotion detection for the Urdu language. Regrettably, a standard corpus for the Urdu language is not available for multimodal emotion detection, as is the case for other languages. One of the primary goals of this research is to fill the gap by providing a custom-made benchmark corpus in the Urdu language, specially designed for multimodal emotion-based analysis. The initial challenge faced in creating this corpus was finding relevant data sources. Primary sources were selected from Facebook, DramaOnline, YouTube, and DailyMotion. The processes used to acquire this raw data were both automatic and manual. The diversity of the corpus was ensured by obtaining data from multiple regions. This meticulous data collection method aims to present a corpus that covers a wide range of emotional expressions in Urdu reflecting different situations and domains.

Real-time emotion annotation

As shown in Fig. 6, we have developed a web app to enable annotations. The annotation software we built was meticulously designed to ensure usability and facilitate an intuitive workflow. Annotators were met with a clean interface that provided elegant functionality and helped them with easy navigation and efficient annotation. The interface had a video playback window placed next to the annotation panel. Consequently, an annotator could watch the video and annotate each sentence in real time. From the annotation panel, annotators could see a row of buttons assigned to the respective emotion classes. Each class was labeled correctly so that it could be easily recognized. The annotators progressed through the video by clicking the appropriate button to assign the relevant emotion class to the sentence. Whenever the button is clicked, the app automatically crops the video fragment annotated by the sentence and saves it to a specified folder with the corresponding label.

Data annotation

In this work, we intend to develop an automated system for the Urdu language, specifically focusing on multimodal emotion detection. To achieve this goal, we required an annotated corpus with labeled emotions. For this purpose, we used the five fundamental emotions: happy, sad, anger, love, and neutral. The development of an annotated corpus starts once raw data has been collected. Then, the expert annotators filter it using the five emotions discussed earlier. The annotators comprised of five experts. Annotation rules and guidelines for problematic and often occurring labeling ambiguities were provided to annotators with multiple samples in advance. In addition, we also prepared a training section, and weekly seminars were held so that the participants could have interactive discussions to work out annotation problems, thereby providing us with informed feedback on the whole annotation process as shown in Fig. 7. A set of guidelines were provided to the annotators which they were strictly required to abide by.

• 1.

  Assign one of the five class labels to each example.

• 2.

  If an example is not assigned to any predefined emotion class, mark it as neutral.

• 3.

  If an example is part of several emotions, it allocates a class of emotions closest to that example based on the context.

• 4.

  The annotation of emotions is limited to the nature of tagged sentences only.

After completing the data annotation, we analyzed the annotated data and found that the corpus was unbalanced across the emotional classes. For this purpose, YouTube videos were used to collect supplemental data to accompany the existing corpus to ensure the balance and comprehensibility of our dataset for training and evaluation purposes. After completing the annotation process, we converted the videos into audio and then transcribed the audio into text. This process enables the formation of a corpus adding visual, text, and speech, for widespread emotional expression in the Urdu language across different modalities. The main struggle with working on multimodal data involves properly synchronizing different modalities comprising text, audio, and video. The dataset structure relies on a CSV file which provides synchronized links between videos, speech, and text. Such alignment methods between all modalities lead to better model training efficiency.

Corpus characteristics

After the annotation, we performed a statistical analysis of the corpus to explore the characteristics of emotional expressions. The fundamental statistics of the corpus are presented in the Table 2. The Table 2 provides the total number of raw videos which vary in length and contain multiple sentences per video. The raw corpus consists of over 125 h of video data. Using our annotation tool, annotators assigned emotion labels to each sentence in the video while watching them. The average duration of annotated video segments is 8 s, with the maximum segment length reaching 15 s. Following the annotation process, the finalized corpus comprises approximately 17 h of annotated data, comprising 8,278 annotated video segments. The Table 2 also presents the total number of examples for each emotion class. Figure 8 further illustrates the distribution of emotion categories by the time duration of each class, corresponding to the different classes within the corpus.

Validation of the corpus

The corpus validation process plays a critical role in this study. During the annotation phase, videos were assigned to annotators, and they labeled each sentence with one of five emotional categories. We employed five annotators for this task. Each video was assigned to at least two distinct annotators from the group of five. As discussed in the previous section, the video was uploaded to our annotation tool for annotation. After the initial annotation by two annotators, the remaining three annotators were tasked with validating the labels. The annotation was considered valid if all three agreed on the same emotion label. Otherwise, the label was revised after a discussion with the annotation team. In cases where no consensus is reached after three annotations, the corresponding entry is removed from the corpus as shown in the Fig. 9. Additionally, we measured the inter-annotator agreement using Cohen’s Kappa score, which was found to be 0.80, indicating a strong level of agreement. Finally, we analyzed our final corpus, which contained 8,278 video segments, to ensure gender balance. These videos feature individuals from various age groups, ranging from 13 to 80 years old, representing a wide demographic. All videos adhere to our specific criteria:

• 1.

  Video must contain one speaker.

• 2.

  There must be no background noise or music in the video and the speaker’s voice should be clear.

• 3.

  The environment should be properly lit, which makes the speaker visible.

• 4.

  The speaker must be speaking in the Urdu language.

Implementation of UMEDNet

The UMEDNet architecture consists of several key components. First, the input modalities are projected with learnable matrices into the shared space of size $d^{\mathrm{\prime }}$ before being processed with activation functions. This step ensures dimensional compatibility and aligns feature distributions across different modalities. Each single modality undergoes an additional projection step to match the dimension space of connected modalities. The multimodal alignment improves when video features are projected to audio and text spaces simultaneously, with the same process for audio and video features. A unified multimodal embedding integrates projected features after integrating specific attributes of each modality while making it possible for the model to learn complex cross-modal relationships. The final step includes applying fully connected layers with dropout regularization onto the merged multimodal embedding to complete emotion classification using a softmax layer.

Using the UMEDNet model users can detect emotions in the Urdu language from video, audio, and textual sources. The modality components operate alone before entering a shared latent space enabling proper alignment for effective fusion. The Wav2Vec2 model obtains audio embeddings by applying its Urdu-specific adaptation to the pre-trained multilingual dataset through fine-tuning. Assessing text embeddings requires XLM-R which underwent successful fine-tuning as it releases contextual understanding of Urdu text relations. Video features are obtained using MTCNN for face detection and FaceNet for facial feature extraction, which helps capture visual cues such as facial expressions and movements. A uniform dimension was maintained via common latent space for embeddings achieved through learnable transformation matrices during feature extraction. The fused multimodal embeddings proceed to a classification network for emotion prediction.

Extensive hyperparameter tuning was conducted using a validation set to achieve optimal performance. The study of learning rate between {1e−3, 5e−4, 1e−4, 5e−5} showed that 1e−3 delivered the optimal balance of speed and stability during convergence. A batch size of 32 delivered the optimal results from the examined options 16, 32, and 64. Weight decay values tested {1e−4, 1e−5, 1e−6} the 1e−5 value became the optimal choice because it managed to limit overfitting without compromising generalization capabilities. From different dropout rates {0.1, 0.2, 0.3, 0.5} the value of 0.2 exhibited the most effective regularization effect. The analysis included multiple fusion techniques such as concatenation and averaging but concatenation was chosen because it maintained original modality information and enabled efficient cross-modal relationship learning. The final selected hyperparameter values were chosen because they resulted in the most optimal F1 score on the validation set.

The training process for UMEDNet was conducted as a multi-class classification via cross-entropy loss optimization. A learning rate of 0.001 coupled with weight decay at 1e−5 function within the Adam optimizer during which the optimization happens. The training runs for 100 epochs but includes an early stopping technique to monitor validation loss because of its capacity to stop overfitting. Robustness in the model is enhanced through multiple implemented regularization and generalization approaches. The model integrates Dropout layers with a rate of 0.2 across its entire structure for overfitting prevention. Batch normalization serves to stabilize training while speeding up the convergence process of the model. Standard evaluation metrics such as accuracy alongside precision, recall, and F1 score are applied to a separate test set for model evaluation.

The complete model framework from feature extraction to projection fusion and classification operates under PyTorch with graphics processing unit (GPU). The Hugging Face Transformers library enables fine-tuning for Wav2Vec2 and XLM-Roberta whereas MTCNN and FaceNet implementation originates from the facenet-pytorch library. This integrated framework provides end-to-end multimodal emotion detection for the Urdu language.