An interpretable multi-transformer ensemble for text-based movie genre classification

Synopsis-based approaches

In recent years, movie synopses have become valuable for multi-label genre classification, providing a concise overview of a film’s plot. Several studies have investigated approaches to using these synopses for automated classification. For instance, Portolese & Feltrin (2018) and Kim et al. (2024) concentrated on extracting text-based features such as term frequency-inverse document frequency (TF-IDF) and word embeddings, finding that models like logistic regression (LR) and multi-layer perceptron (MLP) with TF-IDF features outperformed other methods. Similarly, Akbar, Utami & Yaqin (2022) and Buslim et al. (2022) conducted comparative studies on ML algorithms, including support vector machines (SVM), LR, and Naive Bayes (NB), consistently identifying SVM as the top performer in their experiments. Expanding on this, Wang (2020) explored both ML and deep learning (DL) methods, concluding that recurrent neural networks (RNNs) with long short-term memory (LSTM) layers achieved the highest accuracy in genre classification when applied to online movie synopses.

Subtitle-based approach

Subtitles are textual versions of dialogue and sound effects in movies. They are often used for accessibility and translation. In genre classification, subtitles can be a rich source of linguistic data, capturing a movie’s script’s tone, language, and context.

Rajput & Grover (2022) used movie subtitles for multi-label genre classification by identifying high-frequency words associated with specific genres to train machine learning models. Hasan et al. (2021) proposed a model using a parameter-optimized hybrid classifier (POHC), which combined SVM and Decision Trees (DT). The authors used a dataset of 1,000 movie subtitle files, and their results indicated that the hybrid POHC approach outperformed traditional machine learning classifiers.

Movie plots

Movie plots are a condensed summary of the storyline. Ertugrul & Karagoz (2018) used a deep learning approach with bidirectional LSTM networks to classify movie genres based on plot summaries. They improved their model’s performance by dividing the plot summaries into sentences and training the network on these segments. Their model outperformed traditional RNN and logistic regression models. Kumar et al. (2023) introduced a method that merged problem transformation techniques, like binary relevance and label powerset, with different text vectorizers, including a count vectorizer and TF-IDF. Matthews & Glitre (2021) applied unsupervised topic modelling to textual movie plots. They utilized the model’s topic proportions to investigate genre characteristics, temporal shifts, and canonicity. Kumar et al. (2021) used the universal sentence encoder (USE) for text encoding, combining it with a sequential neural network model and label powerset for problem transformation using movie plots.

Research gaps

Unlike multimodal and audio-visual models such as TimeSformer (Bertasius, Wang & Torresani, 2021) and DIViTA (Montalvo-Lezama, Montalvo-Lezama & Fuentes-Pineda, 2023), which rely on trailers or video data requiring high computational resources and are sensitive to visual noise, text-based models are lightweight, easier to deploy, and provide clearer semantic cues. Visual methods also tend to lack interpretability due to the opaque nature of visual features. Additionally, resources like trailers, posters, or audio clips are often released late in the production cycle, limiting early-stage genre prediction. In contrast, plot summaries and scripts are typically available earlier, making text-based approaches more practical for early genre classification, which aids in timely decisions for marketing and content recommendation.

Most existing studies using textual modalities focus on traditional ML algorithms (e.g., LR, SVM, DT) and DL models such as RNNs and LSTMs. Although these models have shown some success, recent advancements in Transformer-based models have not been widely explored for movie genre classification. Studies such as Ertugrul & Karagoz (2018) and Wang (2020) have applied older architectures, including LSTM and bidirectional LSTM, which are less effective in capturing long-range dependencies in text than transformers.

Although some studies used hybrid models (e.g., hybrid SVM-Decision Tree model (Hasan et al., 2021)), there is a limited exploration of ensemble techniques involving multiple state-of-the-art models, there is potential to improve classification accuracy by combining different models that capture diverse aspects of the text data.

Several studies used traditional text embedding techniques such as TF-IDF and word embeddings. While Kumar et al. (2021) used USE, few studies explore more recent and powerful pre-trained language models such as BERT, which can offer richer contextualized embeddings. This indicates a gap in adopting modern text encoding techniques for more accurate genre classification.

Moreover, to the best of our knowledge, none of the genre classification methods in the literature have employed explainability and interoperability techniques to clarify the classification process. To further clarify the differences between our work and previous studies, Table 1 presents a comparative summary of related approaches focused on text-based movie genre classification.

Data preprocessing

Several pre-processing steps were applied to clean and standardize the text. Punctuation that did not contribute to the meaning of the plot, such as commas and periods, was removed, while elements that could affect sentiment or structure, like question marks, were kept. Special characters (e.g., @, #), which were irrelevant to movie plots, were also removed.

Stop word reduction was performed by minimizing commonly used words (e.g., “the,” “is”) without completely eliminating them, as some still carry contextual significance. Contractions (e.g., “didn’t”) were expanded to maintain consistency and preserve subtle meanings. Redundant phrases or terms were filtered out to avoid inflating the text without adding value. Spacing inconsistencies were corrected for uniformity. These steps helped create a clean dataset suitable for further processing. Table 2 presents an example of pre-processing applied to a sample movie plot.

These preprocessing steps ensure that the textual data fed into transformer models is semantically meaningful, syntactically consistent, and devoid of irrelevant noise, thereby improving the model’s ability to extract genre-relevant features.

Classifier networks

Transfer learning was used by leveraging three classifier networks pre-trained on generalized tasks across different domains. Recent research indicates that choosing between two to five independent, diverse models strikes the best balance between training time and prediction accuracy (Mohammed & Kora, 2023). Three models are selected to ensure a reasonable training duration and to avoid ties during model voting.

In this study, four models were experimented with, including BERT (Devlin et al., 2018), RoBERTa (Liu et al., 2019), AlBERT (Lan et al., 2019), and DistilBERT (Sanh et al., 2019). These models are based on Transformer architecture (Vaswani et al., 2017). These models utilize a bidirectional architecture, allowing for examining input text from both the left and right sides.

BERT (Devlin et al., 2018) is a DL model for pre-training contextualized word representations. Its bidirectional nature makes it very important to handle the nuanced meanings of words in context. RoBERTa (Liu et al., 2019) is an optimized variant of BERT that improves performance by training on much larger datasets, totalling 160 GB, including CC-News (76 GB), OpenWebText (38 GB), and Stories (31 GB). Additionally, RoBERTa uses a dynamic masking strategy during masked language model (MLM) training, generating new masking patterns for each epoch to enhance generalization. DistilBERT (Sanh et al., 2019) is a smaller, faster version of BERT designed to reduce the model size and inference time while retaining much of BERT’s language understanding. It uses knowledge distillation, transferring knowledge from a larger model (teacher) to a smaller one (student), with a distillation loss function combining cross-entropy loss and Kullback–Leibler (KL) divergence. DistilBERT reduces the number of transformer layers from 12 to 6, retains the same hidden size (768) and attention heads (12), and omits the token-type embeddings and next sentence prediction (NSP) objective. It uses the MLM objective during pre-training, similar to BERT. AlBERT (Lan et al., 2019) is a derivative of the BERT model designed to address the challenges of large-scale pre-training regarding both memory consumption and training time.

The fine-tuning process for the BERT, AlBERT, RoBERTa, and DistilBERT models on the Trailers12K dataset for movie genre classification was carefully designed to adapt these models for the multi-label classification task. After obtaining results from these four models, ALBERT was excluded due to its lower performance compared to the other models. These model selections are also based on their proven effectiveness in NLP tasks and their complementary strengths.

Fine-tuning details

Each model was modified to suit the multi-label classification task. The following changes were made:

Ensemble of fine-tuned deep learning models

We experimented with majority voting (hard voting) and probability averaging (soft voting) to create an ensemble for genre classification. In majority voting, predictions from all three models were combined through a hard voting mechanism, where each model’s output contributed equally to the final decision. In probability averaging, each model outputs a probability distribution over the possible genres for each movie, representing the model’s confidence in assigning each genre label, which is then used to combine the predictions. Soft voting aggregates these probability distributions by averaging them across the models. Let P_i,j denote the probability predicted by the i-th model for the j-th genre. The ensemble probability for genre j, denoted as P_j, is computed as the mean of the predicted probabilities from all n models, as shown in the following equation: (1)${\displaystyle P_j=\frac{1}{n}\sum _{i=1}^n{P}_{i,j}.}$

In our case, n = 3, representing the BERT, RoBERTa, and DistilBERT models. The final genre labels are determined by applying a threshold τ to these averaged probabilities. A genre is assigned if its corresponding P_j exceeds τ: (2)${\displaystyle {\hat{y}}_j=\left\{\begin{array}{cc}1\hfill & \text{if}{P}_j\ge \tau \hfill \\ 0\hfill & \text{if}{P}_j<\tau \hfill \end{array}.\right.}$

The experiments indicated that soft voting yielded better performance as it enabled all the models in the ensemble to contribute equally to the final prediction. This method allows each model’s confidence to influence the final prediction, rather than relying solely on discrete labels, making it more suitable for fine decisions in multi-label settings.

Interpretability for ensemble architectures

In the proposed multi-label genre classification model, LIME was used to improve the interpretability and transparency of our ensemble’s predictions. Since the ensemble model functions like a black box, it was difficult to understand how specific input features influenced its predictions. LIME helps generate interpretable explanations, showing why certain genres were predicted for a given movie.

This is particularly important in multi-label classification, where it’s essential to understand how features like plot keywords and linguistic patterns relate to predicted genres. LIME works by perturbing input data and observing how small changes affect predictions, highlighting which features (e.g., key plot elements or themes) contributed to each genre label. Since genres often share overlapping features, LIME clarifies the unique contribution of each feature to multiple labels, offering a detailed breakdown of the model’s decision-making.

The use of LIME improved the trustworthiness and accountability of the model. This is crucial in applications like movie recommendations, where stakeholders must trust the model’s predictions. LIME enables validation of these predictions, ensuring they align with human intuition or reveal areas needing improvement.

Experimental setup

All the experiments were done on Kaggle Notebooks. it is a cloud platform that gives you a ready-to-use setup for machine learning work. The hardware had an NVIDIA Tesla P100 GPU. It came with 16 GB of VRAM and 3,584 CUDA cores. This helped with fast and smooth deep learning tasks using parallel processing. The system also had a single-core Intel Xeon CPU. It ran at 2.2 GHz and had two threads and a 56 MB cache. The machine had 15.26 GB of RAM. It also provided 155 GB of storage to run code and manage data. Kaggle’s system already includes many popular machine learning tools. These include Pandas, Tensorflow, Scikit-learn, Numpy, and Pytorch. Due to these libraries, we could effortlessly train, and test our models. This structure made it simple to run our training several times. It also aided us in obtaining the uniform results each time, which is essential for consistent experiments

Datasets used

Two datasets are utilized in this study: Trailers12k (available at https://richardtml.github.io/trailers12k/) (Montalvo-Lezama, Montalvo-Lezama & Fuentes-Pineda, 2023) and LMTD-9 (available at https://github.com/jwehrmann/lmtd) (Wehrmann & Barros, 2017).

The Trailers12k dataset has twelve thousand movies. It comprises several types of content, like posters (images), trailers (videos), and IMDb material such as plot summaries. Each movie is multi-tagged with up to 10 genres from a set list. The distribution count of movies in each genre is not well-adjusted. Drama is a highly popular genre. It occurs in more than fifty-nine hundred movies among training, validation, and test sets. Some genres, like Fantasy and Sci-Fi, have fewer than 1,600 movies each. The dataset is split into three parts: 70% for training, 10% for validation, and 20% for testing. These splits are stratified, which means the genre distribution stays the same across all parts. Because of these features, the Trailers12k dataset is a good choice for testing multi-label classification models. It is especially useful when dealing with unbalanced genre data.

The LMTD-9 dataset contains approximately 4,007 movies, each labeled with one to three genres drawn from a set of nine. Unlike Trailers12k, it focuses solely on textual metadata, primarily plot summaries. The dataset also exhibits genre imbalance—for example, Drama and Comedy are the most frequent genres, with over 1,900 and 1,500 instances respectively, whereas Sci-Fi appears in only around 310 samples (229 train, 56 test, 25 validation). The dataset is split into training, validation, and test subsets, with at least 10% representation for each genre in every split.

In this work, model training was conducted using the three predefined training splits from Trailers12k, while LMTD-9 served as an additional benchmark to evaluate model generalizability and performance on a smaller, text-only dataset with a different genre distribution.

Evaluation metrics

For evaluation, we employed four commonly used metrics in multi-label classification: micro average precision (µAP), macro average precision (mAP), weighted average precision (wAP), and sample average precision (sAP). µAP computes precision globally by aggregating true positives, false positives, and false negatives across all classes, treating every prediction equally. This makes it particularly effective in reflecting overall model performance regardless of class imbalance. In contrast, mAP calculates the precision for each label independently and then averages these values, giving equal importance to all labels, including those that are underrepresented. wAP addresses class imbalance by weighting each label’s precision according to its frequency in the dataset, thus reflecting performance in more realistic, skewed distributions. sAP evaluates the model’s precision at the instance level by first computing the average precision across all predicted labels for each sample and then averaging these scores across all instances. This metric provides insight into how well the model captures the correct combination of genres for individual movies. Together, these metrics offer a comprehensive evaluation, covering global, per-label, and per-instance performance while accounting for imbalanced genre distributions (Zhang & Zhou, 2014).

Results on Trailers12K dataset

Table 3 compares various ML models (KNN, LR, NB, Random Forest (RF), SVM, and DT) and DL algorithms for multi-label genre classification using the Trailer12K dataset. As mentioned earlier, the Trailers12K dataset is divided into three data splits. The results are reported by averaging the metric values across these splits and the corresponding standard deviation for each average value along with training and testing times for all methods.

The performance of Transformer-based models is particularly noteworthy. BERT, RoBERTa, and DistilBERT performed better than traditional ML approaches, with µAP scores around 79–80%. This suggests that these models effectively capture the nuanced features and complex relationships between different genres. The soft ensemble of DL models achieves the best overall performance (µAP: 80.10, mAP: 78.45), indicating that combining predictions from multiple DL models can effectively capture different aspects of genre classification. This approach benefits from the diverse strengths of individual models in recognizing various genre-specific features.

Among traditional ML methods, SVM performs well (µAP: 74.00 ± 0.04), significantly outperforming other classical algorithms. This suggests that SVMs are adept at handling the high-dimensional feature space. The performance gap between DL and traditional ML methods is substantial, with even the best traditional ML method, SVM, falling short of the worst-performing DL model AlBERT by about four percent in µAP. This underscores the effectiveness of DL in capturing complex, hierarchical features necessary for multi-label genre classification.

Compared with previous state-of-the-art methods, the proposed approach shows significant improvements. Our ensemble outperforms DIViTA Swin-3D (Montalvo-Lezama, Montalvo-Lezama & Fuentes-Pineda, 2023) by 4.53% in µAP and 7.97% in mAP. The advantage is even more pronounced against other video-based methods: 11.12% µAP over TimeSformer (Bertasius, Wang & Torresani, 2021), 7.89% over fastVideo (Cascante-Bonilla et al., 2019), and 5.31% over CTT-MMC-A (Wehrmann & Barros, 2017). These consistent gains across all metrics demonstrate the superiority of our text-based approach.

The relatively small standard deviations in most results indicate consistency across different data splits, suggesting robust performance. However, the higher variability in some ensemble methods, notably the hard ensemble of DL models, points to potential instability. The discrepancy between µAP and mAP scores across all models, with µAP consistently higher, suggests that the models perform better on more common genres and may struggle with rarer ones. This is a common challenge in multi-label classification and indicates an area for potential improvement in future research.

The computational performance reveals our DL Soft Ensemble achieves superior accuracy (80.10 µAP) with practical efficiency (661s training, 23ms testing), being 43× faster to train than video methods (DIViTA Swin-3D: 28800s) while outperforming them by 4.53 µAP.

Results on LMTD9 test dataset

Table 4 compares various ML and DL algorithms’ performance for multi-label genre classification using the LMTD9 test dataset. For these tests corresponding testing time is also reported in the table to reflect the efficiency, the training was performed on Trailers12K dataset.

In the context of traditional ML methods, SVM again demonstrates superior performance (µAP: 74.33, mAP: 71.13), consistent with the results observed on the Trailer12K dataset. LR is the second-best traditional ML method. The DL models, particularly the transformer-based architectures, significantly improve performance over traditional ML methods. BERT achieves the highest individual model performance (µAP: 80.16, mAP: 79.10), closely followed by RoBERTa and DistilBERT. This substantial improvement over traditional ML methods (about six percent in µAP compared to SVM) highlights the power of these language models in capturing nuanced features for genre classification.

The soft ensemble of DL models yields the best overall results (µAP: 80.37, mAP: 79.39), marginally outperforming individual transformer models. This suggests that while ensemble methods still provide benefits, the gain is less pronounced than the Trailer12K dataset, possibly indicating that individual models are already performing near-optimally on this dataset.

Comparing these results with previous state-of-the-art methods reported in the literature, we see significant improvements. For instance, the best-performing soft ensemble outperforms the VGG16+SVM, XGBoost (Mervitz et al., 2020) approach by about six percent in mAP. It also shows substantial improvements over other reported methods like CTT-MMC-TN (Wehrmann & Barros, 2017), C3DLSTM+VRFN (Bi, Jarnikov & Lukkien, 2021), and LOVA (Cai et al., 2023) across various metrics. The performance gap between DL and traditional ML methods is even more pronounced on this dataset compared to Trailer12k. This suggests that the LMTD9 dataset has more complex or nuanced features that DL models are better equipped to capture. Interestingly, the hard ensemble of DL models performs poorly compared to individual models and the soft ensemble. This indicates that simple majority voting (hard ensemble) is ineffective for this dataset.

Statistical significance analysis

We employed the Wilcoxon signed-rank test (Taheri & Hesamian, 2013) to compare the proposed ensemble model’s performance against RoBERTa and DIViTA Swin-3D. This non-parametric test is ideal for paired samples and does not assume normal distribution, making it suitable for small or non-Gaussian data.

The test was applied across three experimental splits using four metrics: μ AP, mAP, wAP, and sAP. A p-value below 0.05 indicates significant performance differences. As shown in Table 5, all p-values were below this threshold, confirming that the ensemble model significantly outperforms both baselines on the Trailers12K and LMTD9 datasets.

Classwise results of ensemble

The results presented in Table 6 show the classwise results of the ensemble on the Trailer 12K and LMTD9 datasets. The ensemble demonstrates effective results, with F1-scores ranging from moderate to good across most genres. This suggests the ensemble approach is robust for this task. Notably, there’s a consistent performance between the two datasets for most genres, indicating good generalization capabilities of the model.

Certain genres stand out as strong performers. Comedy and Drama consistently perform across both datasets, suggesting these genres possess distinctive features that the model can readily identify. Horror performs exceptionally well in Trailer 12K, though slightly less so in LMTD9. Conversely, genres like Adventure and Romance present challenges, with lower F1-scores, especially in recall. This indicates difficulty in identifying all instances of these genres. Fantasy, present only in Trailer 12 K, shows a particularly low recall, suggesting the model struggles to identify many fantasy films.

Figure 2 illustrates the receiver operating characteristic (ROC) and precision-recall (PR) curves for the ensemble model on the Trailers12K dataset. The ROC curve, with an AUC of 0.92, demonstrates the model’s ability to differentiate between classes across various thresholds. The steep rise near the top-left corner indicates excellent performance, particularly at strict thresholds. Meanwhile, the PR curve, featuring an average precision score of 0.76, reveals that the model effectively balances precision and recall. It starts with high precision and gradually decreases as recall increases, which is advantageous when minimizing false positives is crucial.

The performance drop for less frequent genres is further reflected in lower recall and F1-scores, indicating that class imbalance adversely affects the generalizability of the proposed ensemble method. However, by integrating diverse transformer-based models, the ensemble approach captures complementary patterns across genres, helping to mitigate this effect to some extent.

Interpretability and error analysis using LIME

This section presents the results of local and global explainability analyses, providing insights into the model’s genre predictions.

Genre local explainability

Local explainability helps us understand how a model makes predictions for individual samples. Visualizing the importance of different words (tokens) in a movie plot synopsis reveals the extent to which each word contributes to predicting a specific genre. This insight enables us to better understand the model’s decision-making process.

Figure 3 shows the local explanation for a plot correctly classified as “romance”. The bar chart on the left-hand side presents the words that most strongly influenced the model’s prediction, with the colour intensity indicating the strength of the influence. Words like “love,” “relationship,” and “falling” are highlighted as significant contributors to the “romance” classification. The right-hand side of Fig. 3 displays the corresponding text with these influential words highlighted within the context of the synopsis. Figure 4 follow a similar format, providing local explanations for synopses classified as “drama” respectively. In Fig. 4, words like “young,” “super,” and “stardom” are highlighted as key contributors to the “drama” prediction.

Genre global explainability

Global explainability was conducted to better understand the model’s behaviour. This involves identifying the most important words (tokens) for each genre (class) in the dataset.This is achieved by aggregating the word importance values derived from LIME’s local explanations across multiple samples within each genre and calculating the average importance score for each word.

Figures 5A–5B presents the top 10 most relevant words and their corresponding probabilities for each genre. For instance, in Fig. 5A, words like “passion,” “kiss,” and “affection” exhibit high probabilities for the “romance” genre, while in Fig. 5B, words like “conflict,” “emotion,” and “struggle” have high probabilities for the “drama” genre. This analysis reveals distinctive patterns and word-class associations, therefore proving necessary for handling a thorough performance assessment and error analysis. By identifying the most dominant words for each genre, we can more effectively understand the model’s classification results and potential areas for improvement.

Co-occurence matrix

A co-occurrence heatmap is utilized to evaluate the model’s performance in the multi-label, multi-class setting. Figures 6A and 6B show the co-occurrence heatmaps for the LMTD9 and Trailer12k datasets for the proposed ensemble method. The heatmaps display the model’s predictions, with true labels on the vertical axis and predicted labels on the horizontal axis. Matching cells indicate correct predictions, while off-diagonal cells highlight errors.

From the heat maps, varying performance between different genres can be observed.For instance, in Fig. 6A, the model exhibits lower accuracy for genres like “sci-fi” and “thriller,”. Similarly, in Fig. 6B, genres like “adventure” and “fantasy” show relatively lower accuracy compared to others. This performance variation can be attributed to several factors. Trailer12k and LMTD9 datasets have an imbalanced distribution of samples across different genres. Additionally, the semantic and thematic similarities between specific genres, such as “sci-fi” and “action,” or “fantasy” and “adventure,” can make it challenging for the model to differentiate between them accurately.