A review of social background profiling of speakers from speech accents

Introduction

Speaker profiling is the process of estimating the characteristics of an unknown speaker using a classification model that has no prior instance of speeches from the speaker to be profiled. Identification of a speaker’s social origin considering the speech accent characteristics is termed as social background profiling of speakers. The social background of speakers heavily influences their accents in pronouncing different phonemes and the pattern of using them in their speech. These involuntary characteristics in human speech can be used to identify the speaker’s social background without the need for the speaker to reveal it explicitly (Kalluri, Vijayasenan & Ganapathy, 2020; Singh, Raj & Baker, 2016). Speech accent classification considers various linguistic and prosodic features, such as pronunciation, rhythm, intonation, and stress patterns, to determine the accent of a speaker.

Social background profiling from speech has multiple applications, and much research on speaker background profiling has been carried out by people from different disciplines with different objectives. The most widely-used background profiling application is for adapting automatic speech recognition (ASR) models. ASR models perform relatively poorly for accented speech. However, speakers with different backgrounds commonly have a relatively large variation in their accents, especially when speaking a second language. Hence, accent identification has been incorporated into ASR to reduce the error rate. Multiple models are normally trained for different accents, and models trained particularly for a particular accent in the test speech are used for ASR after the classification of the speaker’s accent. Performance results of accent-specific ASR models can highlight the accents that are difficult to recognise; hence acquiring more training data for such accents can lead to more robust ASR (Najafian & Russell, 2020; Weninger et al., 2019).

Social background profiling is also used for the forensic investigation of criminals by utilising a piece of speech evidence related to a crime (Jessen, 2007). As opposed to speaker verification or identification, which is used as forensic evidence to match the identity of a convict, speaker profiling can assist investigators in tracking down an unknown criminal by locating him or her to belong to a specific geographical location or ethnic group. However, the forensic application of speaker profiling has some particular challenges. The difference between accents to be classified may be marginal, which makes it difficult to associate a criminal with a precise geographical location within a country. Furthermore, speech data available for forensic profiling from crime-related evidence are commonly, of very small duration and low quality. More importantly, the margin of error and reliability of the classification are of critical importance as the output of the classification model has legal implications and can be used as a basis for conviction.

Forensic speaker profiling research has focused on the impact of proximate accents as well as linguistic content on accent classification accuracy. Research has also focused on identifying particular phonemes, which are more useful for the classification of particular accents (Brown & Wormald, 2017). The quality and duration of the available samples for forensic analysis are also critical, as mostly the evidence is from a telephone call, with the telephone channel acting as a bandpass filter; shifting the formant frequencies for vowels which are crucial for the classification task (Kunzel, 2001; Moreno & Stern, 1994). Additionally, the commonly short duration of test samples results in the availability of fewer vowels in the speech, which are also critical for the classification task (Brown, 2018). Social, linguistics, forensic, and speech communities have emphasised the need for collaborative research in forensic speech science to avoid innocent convictions and discrimination towards a particular community during the course of criminal investigations or justice processes (Hughes & Wormald, 2020).

Speaker social background profiling can also be used in automated customer services and call centers for identifying the background of a speaker and serving them accordingly. It can also be used for targeted marketing based on the speaker’s background. Finally, it can be used in sociolinguistic and psycholinguistic research to analyze the differences in accent across different societies and geographic regions.

Although significant research works have proposed speaker social background profiling models, no review work has surveyed the state of the art in this area. Some surveys have targeted speaker profiling in general, but none is dedicated specifically to the social backgrounds of speakers. This work surveys recent research on automatic social background profiling for English language speech. The survey covers various aspects of social profiling, including accent classification, dialect identification, and native language identification. The survey examines the application scenarios targeted by the studies and the corresponding challenges.

Rationale and audience

Automatic social background profiling of speakers has numerous practical applications, such as in the development of multilingual speech recognition systems, the creation of personalized speech technologies, and the analysis of speech data for sociolinguistic research. Moreover, recent advancements in machine learning techniques such as deep learning, have led to significant improvement in the accuracy and robustness of social background classification systems.

Despite the significance and rapid advancements in speaker background or accent profiling research, an exhaustive literature review is missing dedicated particularly to speech accent classification. This literature review aims to fill that gap by comprehensively examining recent research regarding speech accent profiling in the context of accents in the English language. Given the diverse applications of speaker accent classification across various domains, this review article caters to a broad audience, encompassing the speech recognition research community, forensic science, and sociolinguistic communities.

This article presents an overview of the current state of research in speech accent classification, including the datasets, speech features and classification models used, and the performance measures achieved. The strengths and weaknesses of the different methods were highlighted. A comparison between and evaluation of different studies, as well as the research gaps that need to be addressed to advance the field, were also made.

This article reviews the recent research on speaker background profiling from speech accents in the English language. The contributions of the article are as follows.

1. It formulates a detailed taxonomy for speaker accent profiling in terms of speech features and the machine learning models used for classification.

2. It provides a detailed literature review of speaker profiling models proposed in recent studies.

3. It provides a comparative analysis of the literature review to present the strengths and weaknesses of the research.

4. It lists research challenges and future research directions to serve as a resource for researchers looking to advance the research on speaker profiling.

The rest of the article is organised as follows. The survey methodology adopted in this article is given in the next section. The following section presents the taxonomy of speech features and machine learning models used for accent classification. The section next reviews recent literature for accent profiling including the datasets designed, the features used and classification models applied for the speaker background profiling tasks. Then a comparative analysis section provides comparisons between state-of-the-art models for speaker background profiling. The next section presents lays down future research directions in the area and the last section presents the conclusion.

Survey methodology

This literature survey systematically evaluates the current state of research in this speech accent classification. It identifies the key research issues and evaluates the effectiveness of different methods. The survey highlights commonly used datasets and performance metrics used for the evaluation of the proposed strategies. Finally, the survey points out the opportunities for future research in the area.

The survey has focused on peer-reviewed journal articles and conference proceedings published within the recent years. The search filtered English language publications published after 2013. The keyword combination used for search was “automatic English speech speaker profiling” OR “native language” OR dialect OR accent OR “social background”. The search targeted all papers containing each of the terms ‘automatic’, ‘English’ ‘speech’, and ‘speaker’ throughout the manuscript and required them to include any of the following: profiling, native language, dialect, accent, or social background which yielded 17,900 results. The results were then filtered, selecting datasets with at least 100 speakers, 2,000 speech samples, and five categories, except for the Accent and Identity of Scottish English Border (AISEB) dataset, which was kept due to its significance for forensic and sociolinguistic applications. The AISEB dataset records accent variations across Scottish English border towns with similar accents. After filtering, nine datasets were analyzed for their state-of-the-art methodologies and results, resulting in a detailed analysis of 10 landmark studies that utilized the shortlisted datasets.

Speech accent classification models have used a variety of datasets for the English language with speaker background annotations. Most models transform these speech samples to derive suitable features for classification and then feed them to various classifiers for the classification tasks. The following information was extracted from each article:

1. Speech features

2. Research methods and techniques

3. Evaluation metrics and datasets

4. Results and conclusions

5. Limitations and future research opportunities

The extracted information was analyzed and organized to present the key trends in speech accent classification research. The main strengths and limitations of current methods have also been compared.

Taxonomy

This section focuses on categorizing the speaker profiling research based on speech accent features and classification models. Figure 1 summarizes the types of features extracted from speech and classification models used for background profiling.

Short-term features

Short-term (ST) features are extracted by small-sized quasi-stationary windows sliding across the speech duration. There are usually spectral or cepstral components for short-duration chunks of speech. Since speech is a time-varying signal produced by changing shapes of the vocal tract as a system response to the glottal excitation signal, the quasi-stationary time windows are usually only a few milliseconds long as humans cannot change the vocal tract shape in time duration, less than that. Short-term features can be generally divided into spectral and cepstral features. The spectral and cepstral features extracted from the short-time windows are referred to as short-term spectral and cepstral features, respectively.

Spectral features

The most well-known ST spectral representation used is the spectrogram, which is the sequence of spectra in the small windows plotted across time. Figure 2 illustrates an audio speech waveform with a corresponding spectrogram plotted against time.

The spectrogram is usually divided into a discrete bank of filters on the Mel scale, and the energies within each filter are summed to get a sequence of Filter-Bank energies. The human neurons respond to the intensity of sound on a logarithmic scale. Inspired by the logarithmic response to sound amplitude, the magnitudes of Filter-Bank energies are also converted to logarithmic representation to obtain the LOG Filter-Bank features (Humayun, Yassin & Abas, 2023; Shon, Ali & Glass, 2018).

Cepstral features

Since speech is produced by the convolution of vocal tract response with glottal excitation, the spectrogram represents the multiplication of both responses, which are changed to additive by taking the logarithm of the spectra. The additive vocal tract and glottal components in the log of spectra are segregated by inverse frequency transform, with the resulting representation referred to as the Cepstrum. Hence, the lower Cepstrum can extract the envelope of the spectrum. The cepstral coefficients capture the spectral envelope of the speech in quasi-stationary short durations. These coefficients have significantly lower dimensions as compared to the spectrum and have very little correlation between themselves, making them suitable as input for the classification models. The most well-known variant cepstral coefficients used in literature is Mel Frequency Cepstral Coefficients (MFCC). MFCC are coefficients of cosine transform computed from energies within band pass filters applied to mel-scaled spectrogram of speech (Lalitha et al., 2015; Najnin & Banerjee, 2019).

Linear prediction coefficients

Linear predictive coefficients (LPC) represent the coefficients of an all-pole autoregressive model estimating the vocal tract response for a smaller time window (Vestman et al., 2018). The coefficients are computed by minimising the mean square of difference between actual and linearly predicted values for each sample using the coefficients. The coefficients represent the vocal tract shape whilst the difference between prediction and the actual sample values represents the glottal excitation component of speech.

Deep learning models

Convolutional neural networks (CNNs) and recurrent neural networks (RNNs) are usually employed as deep architectures for modelling speech sequences (Chung & Glass, 2018; Kim et al., 2023; Wang et al., 2019). RNNs are designed specifically for time-series data, whereby input to each RNN unit is the input feature for that time step concatenated with the weighted output by the same RNN unit from the previous time steps. The final output of the RNN can be the output by the units at the last time step as well as the sequence of outputs for the entire time series. Attention is a mechanism proposed for the RNNs and is state-of-the-art for classification in most speech processing tasks (Qian et al., 2019; Ubale et al., 2019). To employ attention in RNNs, outputs for all-time steps by a single RNN unit are collapsed by weighted averaging while the weights are learned automatically during training. Consequently, critical segments in the input series are highlighted.

On the other hand, CNNs slide multiple matrices (filters), with different weights and fixed-width (kernel size), across the input series. Each convolutional layer is usually followed by a pooling layer, which locally samples by a fixed ratio from the sequence of filter outputs. Finally, global average pooling averages the output sequence from each filter to a single value. Recently, a variant of global average pooling, i.e., attentive pooling, has been proposed for speech accent classification tasks (Ubale et al., 2019). Attentive pooling is a weighted global average with weights learned by training, with the weights for attentive pooling highlighting important input segments. Figure 3 depicts a convolution operation with two filters sliding across time for a speech spectrogram.

Convolutional layers are often followed by activation functions that transform the result of the convolution into a non-linear representation. Activation functions introduce non-linearity into the network and enable it to learn complex representations. Common activation functions used in CNNs include rectified linear unit (ReLU), sigmoid, and tanh. Max pooling and average pooling are two common types of pooling operations that reduce the spatial size of the feature maps produced by the convolutional layer; Max pooling selects the maximum value from a set of adjacent activations, while average pooling takes the average of the same set of activations. These pooling operations help to make the network more robust to translation and scale variations in the input data, as well as reduce computational complexity.

In recent years, speech processing models have used transformer-based models. Transformers are capable of modelling longer dependencies between speech frames, allowing them to better capture the context of speech signals (Vaswani et al., 2017). It utilizes the self-attention mechanisms and originally uses an encoder–decoder architecture, to differentially weight the significance of different parts of the input data.

In terms of computation complexity, CNNs and RNNs are more computationally complex compared to SVM and ELM, as they involve multiple layers and require more computation for each layer. LDA, being a linear classifier, is computationally least complex, but it may not perform well on complex datasets.

Literature review

Comparative analysis

The most common performance measure for evaluating speaker profiling models is accuracy (Rizwan & Anderson, 2018; Singh, Pillay & Jembere, 2020). Accuracy indicates the ratio of correctly predicted speech samples to the total number of speech samples. However, other performance measures, including precision, recall, and f-score have also been reported and compared for many speaker profiling tasks. Precision for each speaker category in the dataset represents the ratio of true predictions to the number of total predictions for the category. On the other hand, recall indicates the ratio of true predictions to the total number of samples for each category. F-score encompasses both precision and recall and is computed as the harmonic mean of precision and recall (Humayun, Yassin & Abas, 2022). All the category-wise metrics including precision, recall, and f-score can be averaged across all categories to indicate an overall performance measure for the complete dataset. The averaging across categories can be simple or weighted with the number of samples in each class, with weighted averaging capable of highlighting if the classifier is biased towards a particular category. Besides these performance measures, a confusion matrix is usually used to illustrate the number of predictions corresponding to the true samples for the categories. The confusion matrix effectively highlights the common misidentifications between specific categories.

Brown & Wormald (2017) has used the AISEB dataset to test forensic accent profiling over similar accents. The dataset consists of speech samples from proximate towns near the Scottish-English border. Experiments were conducted using the ACCDIST-based SVM classifier, with the classifier achieving an accuracy of 86.7% in identifying the four different accents. To test the classification performance for telephone-quality speech, the speech has been degraded by down-sampling and bandpass filtering to mimic the telephone channel. This drops the classification accuracy to 64.4%.

Weninger et al. (2019) use a bi-directional long short term memory (LSTM) based deep learning model for text-independent accent classification to improve Mandarin speech recognition. Acoustic features, which have been extracted by sliding windows over a complete audio frame, are used as input to the bi-directional LSTM for temporal modelling. The classifier achieves a low accuracy of 34.1% in identifying the 15 different accents of mandarin from mainland China.

Jiao et al. (2016b) merged feed-forward and recurrent neural networks for text-independent, native language identification from the TOEFL dataset for the Native Language Sub Challenge (NLSC). In trying to classify speakers from the 11 different countries, the proposed model achieves an accuracy of 51.92%. Both the DNN and RNN are trained without textual information. The audio samples are segmented into fix-sized 4-second frames for classification, whereby for each 4-second frame, a long-term feature vector is obtained and used as input for the DNN. The long-term feature vector is obtained by computing different statistical functions, including mean and standard deviation, over short-term MFCC features across the frame. On the other hand, the RNN models the temporal series of the short-term features directly. A combination of both RNN and DNN is used as the final classifier. However, the accuracy for the short-term features-based classification model is significantly lower than the I-vector based model on the same dataset, with researchers (Shivakumar, Chakravarthula & Georgiou, 2016) reporting an accuracy of 79.93% using I-vector as well as phonetic features for the same NLSC task.

Najafian & Russell (2020) use the ABI dataset, which consists of fourteen different accents of British English, to improve and adapt the ASR model. The model uses a PPRLM with SVM for phoneme-based and universal background-Gaussian mixture model for text-independent accent classifications and merge their outputs. It has been shown that the model is able to achieve an accuracy of 84.87%. Prior research by De Marco & Cox (2013) on the same ABI dataset had demonstrated an accuracy of 81.05% by using the acoustic I-vectors only.

Ge (2015) combines phoneme-specific and text-independent classification using the Foreign Accented English (FAE) corpus classifying seven accents. The proposed model is composed of a combination of phoneme-based and long-term classifiers. Universal background GMM classifier predicts the accent for both, the phonemes and the complete speech utterances. The phonemes which can be recognised with a higher degree of confidence have been selected for phoneme-based weighted classification, with the weight for each classification corresponding to the occurrences in the dataset for that phoneme. Tested with short speech samples, the model is able to achieve an accuracy of 54%. However, the corpus consists of 23 different accents from around the world, which are grouped to collapse into seven different categories for classification, and this makes the target accents significantly different and the classification task relatively easier.

A research (Rizwan & Anderson, 2018) classifies specific word utterances in speech using ELM with MFCC and its delta features as input, to classify English speech. The research used the TIMIT dataset and achieves 77.88% accuracy for seven native United States accents. Singh, Pillay & Jembere (2020) used Speech Accents Archive (SAA) dataset and found that MFCC as short-term features perform the best for accent classification.

Finally, Ubale et al. (2019) propose to use attentive pooling in CNNs for accent classification. The model uses neural network for the classification with short term spectral features as input and PLDA with I-vector inputs. The fusion of both classifications achieves 83.32% accuracy over a recent TOEFL dataset managed by the Education Testing Service (ETS). More recently, Ubale et al. (2019) applied CNN model directly to a raw audio and merged its output with I-vector classifier. The fusion reported an accuracy of 86.05%. Kethireddy, Kadiri & Gangashetty (2020) have also used CNN with the raw audio as input and reported 81.26% accuracy over a subset of CV dataset comprising eight accents.

Table 4 lists benchmark models for speech accent classification with the reported accuracies as well as the strategies and features used. The table demonstrates that until recently, I-vector representations with shallow models, such as support vector machines (SVMs), have been found to perform best for speech accent classification. However, recent studies have shown that using sequences of spectro-temporal features or even raw audio waveforms with CNNs can result in higher classification accuracy. This shift in the use of features and models highlights the evolving nature of the field and the ongoing search for improved methods in speech accent classification. Figure 4 illustrates the ratio for usage of different speech datasets, input features, and classification models by the surveyed studies.

Future research directions

Analyzing the state-of-the-art models and trends in speaker profiling, the following list of key research directions can be identified for future research in speaker profiling.

1. Deep learning-based unsupervised long-term speaker embeddings for profiling

2. Interpretable machine learning for speaker profiling, especially in forensic applications.

3. Exploring the correlation of phoneme articulation features with demographic accents.

4. Speaker profiling models for low-resource languages.

Most successful models that utilise long-term acoustic features use I-vectors that are computed via statistical measures. Although bottleneck embedding from deep neural networks, termed x-vector, has been used for speaker verification models, deep learning-based long-term modelling has yet to surpass the performance of I-vectors for speaker accent profiling.

Lately, there has been a noticeable increase in research exploring interpretable machine learning and deep learning models for various machine learning tasks. However, the task of speaker profiling research does not have many interpretable models. This is particularly crucial, given the substantial advantages that interpretable models can bring, especially in the context of forensic and sociophonetic applications. Interpretable models can significantly enhance the understanding of the correlation between the articulation features of phonemes and demographic accents. Filling this gap provides a valuable research direction that can help grasp the connections between linguistic nuances and demographic characteristics. Moreover, interpretable results for speaker profiling can be more reliable for forensic communities aimed at criminal conviction which has critical human consequences.

Except for a few studies (Humayun, Shuja & Abas, 2023), most of the accent classification models disregard individual phoneme specific time segments, classifying complete utterances. Few works target analysis from individual phoneme time windows. Examining short-term features related to the articulation of a specific phoneme and exploring the variability of these features within a short duration can also significantly aid in enhancing the comprehension of sociophonetic variations. This exploration has potential for future research, particularly in advancing phonetic understanding.

Limited works have analyzed vowel-specific accent classifications. Vowels stand out as the most prominent distinguishing phonemes across various accents, given their usage frequency in speech and their extended durations. Therefore, delving into the articulation characteristics and, consequently, the features unique to vowel-associated speech is an important focus for future research. The specific classification of vowels can not only refine our understanding the classification results but can also enhance interpretable and reliable speaker profiling.

Machine learning-based research demands extensive datasets and substantial resources, often directing its attention predominantly to high-resource languages, such as English, while neglecting low-resource languages, particularly those indigenous to areas with limited resources. Despite the presence of accent variations in many of these low-resource languages, minimal research has been dedicated to speaker profiling within this context. Consequently, there is a significant research potential in exploring low-resource languages for speaker profiling.

Cross-lingual transfer learning capability can also be explored in the future, particularly for low resource languages. Accent or speaker profiling models can be tailored for low-resource languages by adapting through the fine-tuning of models initially trained on high-resource languages. This approach offers a viable solution to address the scarcity of labeled datasets in low-resource languages by transferring the knowledge gained in speaker profiling from high-resource languages.

Conclusions

Speaker profiling ranges from identifying the physical body parameters of speakers to identifying their social traits, geographic origin, and native language. Estimation of the social background of speakers has multiple applications, including forensic investigations and improving speech recognition models.

This article has reviewed the state-of-the-art in speaker profiling from speech accents. The speech features, generally used as input for speaker accent classification, have been presented. Short-term acoustic features are mostly modelled as long-term vectors for complete speech utterances to classify the accents. Traditional machine learning classifiers using I-vector as long-term acoustic representation were the most successful for speaker profiling tasks. However, deep learning models have recently outperformed conventional classification methods by using recurrent and convolutional neural networks on the sequence of short-term spectral features or even directly on the raw audio waveform.

Most notable among the speech datasets for English is the TOEFL English speech dataset by ETS, containing speech samples from candidates worldwide and has been used for native language identification tasks. The highest accuracy of 86.05% has been reported on the ETS dataset by applying a bank of CNNs directly to the raw audio waveforms of speech.