Emotion detection from handwriting and drawing samples using an attention-based transformer model

Introduction

Emotion detection (ED) is the process of recognizing and evaluating the emotional states and feelings of individuals using a variety of techniques. Accurately understanding and interpreting human emotions is the ultimate objective of ED, which has a variety of applications in areas including mental health, user experience, education, marketing, and security (Acheampong, Wenyu & Nunoo-Mensah, 2020). Emotions are one’s reactions, and they can differ widely among individuals. Defining universal patterns or guidelines for detection might be challenging since individuals may show the same emotion in various ways. Although emotion science has made considerable strides, there is still much to learn about the subtleties and complexity of human emotions. It is still difficult to create comprehensive representations that adequately depict the entire spectrum of emotions (Zad et al., 2021). The development of intelligent systems to assist physicians at the point of treatment uses machine learning (ML) techniques. They can support conventional clinical examinations for the assessment of Parkinson’s disease (PD) by detecting its early symptoms and signs. In patients with PD, previously taught motor abilities, including handwriting, are frequently impaired. This makes handwriting a potent identifier for the creation of automated diagnostic systems (Impedovo, Pirlo & Vessio, 2018). DL models have shown promising outcomes in ED, notably those built on RNNs and convolutional neural networks (CNNs). These models can recognize temporal connections and learn complicated patterns from the emotional data (Pranav et al., 2020).

The study conducted by Kedar et al. (2015) analyzed handwriting features such as baseline, slant, pen pressure, dimensions, margin, and boundary to estimate an individual’s emotional levels. The study concludes that it will aid in identifying those individuals who are emotionally disturbed or sad and require psychiatric assistance to deal with such unpleasant emotions. Additionally, Gupta et al. (2019) examines electroencephalogram (EEG) signals from the user’s brain to determine their emotional state. The study uses a correlation-finding approach for text improvement to change the words that match the observed emotion. The verification of the sentence’s accuracy was then carried out utilizing a language modeling framework built on long short-term memory (LSTM) networks. In a dataset with 25 subjects, an accuracy of 74.95% was found when classifying five emotional states employing EEG signals. Based on handwriting kinetics and quantified EEG analysis, a computerized non-invasive, and speedy detection technique for mild cognitive impairment (MCI) was proposed in Chai et al. (2023). They employed a classification model built on a dual-feature fusion created for medical decision-making. They used SVM with RBF kernel as the basic classifier and achieved a high classification rate of 96.3% for the aggregated features.

Existing ED research has mostly concentrated on a small number of fundamental emotions, such as happiness, sadness, and anger. The complexity and variety of emotional expressions make it difficult to adequately depict the entire emotional range. The performance of ED systems is improved by identifying significant characteristics across several modalities and creating suitable representations (Zakraoui et al., 2023). The capacity to recognize emotions via routine activities like writing and drawing could contribute to mental health (Rahman & Halim, 2023). Collecting handwriting and drawing samples has been easier with the rise of human-machine interfaces like tablets (Likforman-Sulem et al., 2017). Therefore, a key component of the connection between human and computer communication is extracting and comprehending emotion (Yang & Qin, 2021).

In contrast to previous research, which mostly relied on text or audio data, this study involves a novel approach to analyze samples of handwriting and drawings to identify emotions. The suggested method employs a digital platform to analyze handwriting or drawing samples of an individual and forecast their emotional state using an attention-based transformer model. This method can capture more complex and sensitive feelings, which may not be capable of adequately conveyed through spoken words. This work employs an attention-based transformer model, which has proven to be very successful in natural language processing (NLP) tasks. The proposed model can capture long-range relationships and identify significant aspects in the input data, which may be essential in precisely recognizing emotions. The performance of the suggested model is tested on benchmark datasets that outperform the closely related work. The proposed study advances current information by establishing a baseline for evaluating the performance of ED models using handwriting and drawing examples.

A survey of existing work

In recent years, a substantial body of research has been dedicated to exploring the intersection of machine learning and computer vision in the domain of emotion detection (Kanwal, Asghar & Ali, 2022; Azam, Ahmad & Haq, 2021; Asghar et al., 2022; Li & Li, 2023). This has become particularly crucial given the growing significance of understanding and interpreting human emotions. Beyond emotion detection, machine learning has found applications in various domains, including hot topic detection (Khan et al., 2017, 2023), anomaly detection (Haq & Lee, 2023), and recognizing anomalous patterns (Haq et al., 2023), especially in social media contexts. Furthermore, machine learning has played a pivotal role in text classification (Ullah et al., 2023; Khan et al., 2022), providing sophisticated approaches for organizing and categorizing textual data and rating software applications. The ability to classify information is vital not only for sentiment analysis but also for broader applications such as content categorization (Zhang et al., 2023). In addition, information fusion (Wang et al., 2021; Zhang et al., 2020), where data from diverse sources are integrated to provide a more comprehensive understanding, has gained prominence. In the pursuit of accurate diagnoses for depression, Esposito et al. (2020) have explored innovative methodologies to obtain more reliable measurements than conventional questionnaires. This involves analyzing behavioral data, including voice, language, and visual cues, with sentiment analysis techniques evolving from linguistic characteristics to sophisticated tools for text, audio, and video recordings. The research study (Aarts, Jiang & Chen, 2020) introduces an application detecting four distinct emotions from social media posts, outlining techniques, outcomes, challenges, and proposed solutions for the project’s continuity. The research study (Aarts, Jiang & Chen, 2020) utilizes convolutional neural networks to assess visual attributes in characterizing graphomotor samples of Parkinson’s disease (PD) patients. The study achieves an 83% accuracy in early PD prediction using a dataset of 72 subjects through visual information and SVM classification. According to the study (Aouraghe et al., 2020) emotions like stress, worry, and depression have an impact on health, so it is crucial to recognize their emotional symptoms as early as possible. The study conducted in Moetesum et al. (2019) focused on the possibility of using handwriting’s visual characteristics to anticipate PD. The structure and operations of some brain areas are impacted by neurodegenerative illnesses, which lead to a gradual deterioration in cognitive, functional, and behavioral abilities.

A research study (Ayzeren, Erbilek & Çelebi, 2019) introduces a distinctive handwriting and signature biometric database with emotional status labels. The investigation delves into predicting emotional states (happy, sad, stress) from online biometric samples, achieving noteworthy success, especially in stress prediction from handwriting. The database, encompassing 134 participants, includes demographic information for comprehensive analysis. Examining preserved graphic structures in handwriting, the study (Rahman & Halim, 2022) explores the correlation between handwriting features and personality traits. Using a graph-based approach, eleven features are extracted to predict personality traits based on the Big Five model. Employing a semi-supervised generative adversarial network (SGAN) for enhanced accuracy, the study achieved a remarkable predictive accuracy of 91.3%. Nolazco-Flores et al. (2021) characterizes emotional states related to depression, anxiety, and stress using features from signals captured on a tablet. The EMOTHAW database includes handwriting and drawing tasks categorized into specific emotional states. Selected features improve average accuracy classification (up to 15%) compared to the baseline. The study conducted in Shrivastava, Kumar & Jain (2019) presented a DL framework for the challenge of fine-grained emotion identification utilizing multimodal text data. They unveiled a new corpus that depicts various emotions gleaned from a TV show’s transcript. They employed a sequence-based CNN with word embedding as an ED framework to identify emotions. The study conducted in Bhattacharya, Islam & Shahnawaz (2022) provided a unique approach based on the Agglomerative Hierarchical Clustering algorithm, which can recognize the emotional state of the individual by examining the image of the handwritten text. They identify emotions with a maximum accuracy of 75%. There exists a lack of well-established techniques for emotion recognition using handwriting and drawing samples. The precise characteristics or patterns in handwriting that relate to certain emotions are not well discovered. Various experiments are conducted to detect emotion from handwritten text, however, there still exist certain limitations which are explained in the following sub-section.

Proposed methodology

The proposed study introduces an attention-based transformer model designed to generate a more comprehensive feature map from handwriting and drawing samples. This model aims to accurately identify both handwritten information and emotional content. The model supposes that writing and drawing are impacted by one’s emotional state and are connected to behavior. The proposed method involves gathering an individual’s writing through an electronic device and analyzing it to determine her emotional state. The suggested model is founded on the transformer architecture, which makes use of attention processes to provide a more detailed feature map of the data. The goal is to enhance the model’s capacity to recognize and understand the information and feelings represented in handwriting and drawing. The suggested study offers a thorough assessment of numerous writing and drawing traits and identifies their relationship to emotional states. Using writing and drawing samples, this work identified emotions with a high degree of accuracy. The details of how the algorithm works are explained in Algorithm 1.

Utilizing the Attention-Based Transformer Model for Emotion Detection involves a systematic process. First, the required libraries and frameworks are imported, and the model architecture is defined, considering parameters such as the number of layers (L), dimension of the model (d_model), number of attention heads (h), and dimension of the feed-forward network (dff). Pre-trained weights of the Transformer model are loaded, and the provided handwriting and drawing samples undergo preprocessing, converting them into a suitable input format and storing them for subsequent use. Following this, the preprocessed samples are loaded, converted into tensors, and passed through the Transformer model for inference. The model’s output is processed through a linear layer and a softmax function to generate predicted probabilities for each emotion. The final step involves extracting the emotion with the highest probability for each sample, ultimately returning the predicted emotion. This streamlined process ensures effective and nuanced emotion detection from input samples.

Datasets

EMOTHAW dataset

The EMOTHAW database (Likforman-Sulem et al., 2017) contains samples from 129 individuals (aged between 21–32) whose emotional states, including anxiety, depression, and stress, were measured using the Depression Anxiety Stress Scales (DASS) assessment. Due to the dearth of publicly accessible labeled data in this field, this database itself is a helpful resource. A total of 58 men and 71 women participated in the dataset. The age range has been constrained to decrease the experiment’s inter-subject variation. Seven activities are recorded using a digitizing tablet: drawing pentagons and houses, handwriting words, drawing circles and clocks, and copying a phrase in cursive writing. The writing and drawing activities used to get the measures are well-researched exercises that are also used to assess a person’s handwriting and drawing abilities. Records include pen azimuth, altitude, pressure, time stamp, and positions (both on article and in the air). The generated files have the svc file extension, generated through the Wacom device. Figure 1 shows a system overview of the proposed method.

SemEval dataset

The Semantic Evaluations (SemEval) dataset (Rosenthal, Farra & Nakov, 2019) includes news headlines in Arabic and English that were taken from reputable news sources including the BBC, CNN, Google News, and other top newspapers. There are 1,250 total data points in the dataset. The database contains a wealth of emotional information that may be used to extract emotions, and the data is labeled according to the six emotional categories proposed by Ekman (1999) (happiness, sadness, fear, surprise, anger, and disgust).

Experiments

The initial stage in the experiments is to process the data from each dataset to identify the appropriate features for emotion identification from handwriting and drawing examples. An attention-based transformer model for emotion recognition from handwriting and drawing samples is trained and evaluated through a series of experiments. For this experiment, we use the EMOTHAW and SemEval benchmark datasets. These datasets include several types of handwritten and drawn samples as well as the associated emotion labels. An SVC file containing 1,796 points with seven metrics (x location, y location, time stamp, pen status, azimuth, altitude, and pressure) is part of the EMOTHAW dataset. Both points acquired on article (pen status equal to 1) and points acquired in the air (pen status equal to 0) are included. The SemEval dataset includes 4,359 handwritten text samples of various emotions, such as happiness, surprise, joy, anger, fear, and sadness.

Results and discussion

In the first experiment harnessing the rich EMOTHAW dataset, the proposed study employed an attention-based transformer model to meticulously analyze collected features and unravel the intricacies of emotional states. The model demonstrated its prowess on the test set, surpassing conventional machine learning techniques with a peak accuracy of 92.64%. Notably, the integration of both handwriting and drawing traits, as outlined in Table 1, proved to be a game-changer, giving superior accuracy in diagnosing depression and offering nuanced insights into the identification of anxiety and stress.

Using the EMOTHAW dataset, the proposed study used an attention-based transformer model for the collected features to achieve the recognition results for the three emotional states. On the test set, the model outperformed conventional machine learning techniques with the highest accuracy of $92.64\mathrm{\%}$. Using both handwriting and drawing traits together, as shown in Table 1, we were able to diagnose depression with the best accuracy. Similar findings were obtained for the identification of anxiety, with $79.51\mathrm{\%}$ accuracy using drawing, $77.38\mathrm{\%}$ using writing, and $83.22\mathrm{\%}$ integrating both writing and drawing features. For stress detection, we obtained the highest accuracy of $79.41\mathrm{\%}$ using writing features. To encapsulate, the EMOTHAW dataset, coupled with the proposed advanced model, not only advances the field of emotion detection but also sets a new standard in accuracy, particularly when considering the synergistic effects of both handwriting and drawing traits. These findings underscore the robustness of the proposed approach and its potential impact on applications ranging from mental health diagnostics to personalized well-being assessments.

In the second experiment, incorporating the diverse SemEval dataset enriched with emotional annotations from tweets, blog posts, and news articles, the study examined text-based emotion detection. The categorization of features into three distinct emotional states, angry, happy, and sad, unfolded insightful revelations. Notably, the proposed model showcased state-of-the-art performance on the test dataset, particularly excelling in the identification of sadness with a remarkable F1 score of 87.06%, as clarified in Table 2. Moreover, the proposed study achieved notable success in detecting happy and angry states, reaffirming the versatility of the model across various emotional dimensions.

In the SemEval dataset, a variety of text samples with emotional annotations are included, such as tweets, blog posts, and news articles. In this work, the features are categorized into three different emotional states, angry, happy, and sad. On the test dataset, the model produced state-of-the-art results for sad state identification, with an F1 score of $87.06\mathrm{\%}$, as shown in Table 2. For happy state detection, we obtained the highest F1 score of $79.73\mathrm{\%}$ and for angry state detection we obtained the highest F1 score of $83.12\mathrm{\%}$. To sum up, the SemEval dataset, coupled with the proposed model, opens new horizons in text-based emotion detection. The robust performance across different emotional states, especially the outstanding identification of sadness, highlights the adaptability and efficacy of the proposed work. These outcomes not only contribute to the academic discourse but also pave the way for practical applications in sentiment analysis across diverse textual genres.

Figure 2 showcases the robust performance of the proposed model. This high level of accuracy indicates the model’s proficiency in learning from the training data and effectively generalizing it to new, unseen data during testing. The sustained elevation of both lines above 90% emphasizes the reliability and effectiveness of the proposed model in accurately predicting emotional states based on the provided features. To specifically address color differentiation issues, we have manually introduced circles to represent training accuracy and rectangles for testing accuracy. This visual distinction aims to enhance clarity and inclusivity for a diverse audience, mitigating potential challenges associated with color perception.

Conclusion

The drawing and handwriting signals are instances of time series data, which is shown as a collection of data points acquired over time. In this work, we extract time-domain, frequency-domain, and statistical-domain features from the handwriting and drawing signals. The proposed model has the benefit of being able to capture long-range relationships in the input data, which is especially beneficial for handwriting and drawing samples that contain sequential and spatial information. The model’s attention mechanism also enables it to concentrate on relevant components and structures in the input data, which may enhance its capacity to recognize minor emotional signals. The hyperparameters that are adjusted during the testing of the model include the number of layers, attention heads, the dimensionality of embeddings, learning rate, and batch size. Concerning accuracy and F1 scores, the attention-based transformer model used in this study excelled on two benchmark datasets.

In the future, transfer learning techniques could be used to pre-train the attention-based transformer model on large datasets and fine-tune it for specific emotion detection tasks, which could potentially improve the model’s performance on smaller datasets.

Supplemental Information