Contradiction in text review and apps rating: prediction using textual features and transfer learning

Machine learning models

RF creates numerous decision trees, which lowers the variance, but the results are difficult to understand. Regression and classification are two uses of RF. Results are combined into a single final result using a group of decision trees. By using a subset of characteristics randomly and training them on various data samples, variance is reduced by RF in a couple of different ways (Breiman, 2001).

AdaBoost is the very first algorithm that outshines other related algorithms in boosting classification tasks concerning accuracy. AdaBoost is an adaptive model as it takes into account prior misclassifications made to adjust for weak learners (Schapire & Singer, 1999). AdaBoost attains a high level of accuracy when taking into account a fully learned model. Computing a feature significance score is also utilized to identify key characteristics.

For classification and regression issues, GBM is applied. It utilizes several weak learners like decision trees to create a prediction model (Natekin & Knoll, 2013). Boosting is a technique in which weaker models are turned into stronger ones. Boosting is characterized by a tree model in which every tree is adjusted to the original dataset in an updated/modified state. Loss functions use gradients to evaluate the performance of coefficient models to give GB its strong performance. Depending on the amount, feature, or quality that has to be optimized, the loss function’s exact specification will change.

Due to its efficiency and quick execution, the boosting method XGBoost is quite popular among data scientists. Its implementation, which consists of stochastic gradient, regularized gradient boosting, gradient boosting, and boosting is what gives it its high performance. Both classification and regression issues are addressed by it. As XGBoost helped multiple Kaggle tournament winners succeed, it received higher attention (Chen & Guestrin, 2016). As in other bagging and boosting classifiers, feature significance scores are also computed.

To increase prediction accuracy and reduce overfitting, the ET classifier incorporates a meta-estimator that fits randomized decision trees (RT) on different sub-samples of a dataset (Geurts, Ernst & Wehenkel, 2006). The ET classifier uses the complete sample at each step in contrast to RF and chooses decision limits at random rather than choosing the best one. The phrase “very randomized tree” is another name for it.

Transfer learning models used for predicting numeric rating

This research work also analyzes the performance of state-of-the-art transfer learning models.

Dataset used for experiments

Using the web scraper BeautifulSoup, the Google applications dataset is extracted from the Google Android Play store. The data is collected for the last five years. The following standards are used

1. The app must be at least five years old.

2. There must be at least 4,000 reviews on the app.

The dataset comprises several features including ‘app id’, ‘app category’, ‘app rating’, ‘app name’, and ‘app review’. Table 1 provides names and descriptions of these features. The total count of extracted data is 502,658. This research focused on examining the inconsistency between reviews and rating estimates centered upon the attributes collected from the reviews. Other variables and associated meta-data are also available on the Play Store. However, we ignored other features and meta-data that are unnecessary.

There are 14 different types of mobile applications in the scraped data. To cover the wide range of applications, sampling across many categories is done primarily for that reason. The numerous forms of textual evaluations in each area such as sports, news, entertainment, health, and business use a wide range of idioms and terms. So, rather than focusing on a small number of categories, the aim is to precisely estimate the classification accuracy by testing the classifiers performance and applying them to a variety of reviews. A few sample reviews from sports, communication, weather, action, and health & fitness categories are displayed in Table 2.

All of the categories in the dataset are listed in Fig. 5 along with their relative sizes. Each app has almost 4,000 reviews in the scraped database. Individual user data on app reviews and ratings are examined.

The flow chart in Fig. 6 shows how the data is collected employing the BeautifulSoup (BS) web scraper. By seeing trends, patterns, and connections, it is found that they could go unnoticed in data based on text, data visualization is crucial for comprehending the dataset. Data visualization makes it simple to spot these linkages in various datasets. To present the numerical ratings given to mobile apps, we displayed the dataset.

Figure 7 shows that five stars are the most common app user rating. Yet, a significant worry is the potential for slanted or even false ratings provided by anonymous people. So, we took into account how frequently each category received numerical ratings.

Figure 8 demonstrates that casual mobile apps consistently receive better scores, ratings, and/or rankings and are expected to exhibit a 5-star rating from the users of the app. The lowest numerical ratings, on the other hand, are typically given to creators of video players and editors.

Reviews pre-processing

The dataset related to the Android Play Store is either unstructured or semi-structured and hence holds a lot of unnecessary information that does not significantly contribute to the prediction. Text pre-processing is a prerequisite to avoid this limitation since extended training is needed for large datasets while ‘stop words’ lower predicted accuracy. Pre-processing entails many processes such as stemming, lowercase conversion, punctuation, and removing terms that do not have a greater significance concerning the text. According to research, text preparation significantly increases prediction accuracy (Feldman & Sanger, 2007). So, before beginning the training process using the suggested technique, the data must be normalized. The use of the smartphone app, Seven 7 min workout challenge, is reviewed as an illustration for exercise challenge as a text pre-processing to have a better understanding of it. Table 3 displays a series of pre-processing steps and their results. The ensemble learning classifiers may be used on the processed dataset when pre-processing is finished.

Feature engineering

To train supervised machine learning algorithms, textual materials need to undergo vectorization. This process involves converting text into numerical representations while preserving all the relevant information. Various methods can be employed for text transformation, such as the Bag-Of-Words (BOW) approach, in which a unique number is assigned to each word. However, BoWs effectiveness is compromised due to limitations on character length in reviews. Additionally, the accuracy of BOW-based techniques is hindered by inadequate occurrences of certain words (Sriram et al., 2010). To address these challenges and accomplish the transformation, we utilize the concept of term frequency (TF) (Scikit Learn, 2023b). This approach considers the number of occurrences of a word(s) in a certain document and generates a matrix that represents the overall frequency of each word across the entire document. To illustrate the application of TF, let us consider the My Talking Tom app. The initial text is:

It’s fantastic, and my kids adore playing it. I appreciate that you have this game in the shop.

Preprocessing turns the review into a fantastic game kid appreciate. I appreciate the gaming shop. In Table 4, the matrix produced by using TF is displayed.

In the proposed technique, feature selection is done using the TF/IDF (uni-, bi-, and trigrams) (Scikit Learn, 2023c). TF-IDF assigns higher weights to terms that appear frequently in a text subset while giving lower weights to words that are extremely frequent and appear in almost all documents. So, terms that are used frequently are given a lesser weight than uncommon words that are used more specifically in a given document. Table 5 displays the TF/IDF result for the aforementioned scenario.

Accuracy measures

To evaluate performance, we employed many metrics. Understanding the various accuracy characteristics of a classifier requires a comprehension of four fundamental concepts (Han, Pei & Kamber, 2011).

• True positives (TP) refer to the instances where the positive class is accurately predicted by the classifier.

• True negatives (TN) are instances where a negative class is correctly predicted by the classifier.

• False positives (FP) occur when the classifier incorrectly labels negative instances as positive.

• False negatives (FN) are the instances mistakenly marked as negative by the classifiers which are positive instances.

In many classification methods, accuracy is a key assessment factor. Accuracy is a measure of how many correctly predicted instances (both true positives and true negatives) there are among all the instances in the dataset. It provides an overall assessment of the model’s correctness. In terms of the previously defined TP, FP, TN, and FN, it is computed as follows (2)${\displaystyle accuracy=\frac{TP+TN}{TP+TN+FP+FN}\times 100}$

Accuracy measures overall correctness in predictions, while precision specifically quantifies the model’s ability to avoid false positive predictions. (3)${\displaystyle precision=\frac{TP}{TP+FP}}$

Recall, also known as sensitivity or true positive rate, measures the ratio of true positive predictions to all actual positive instances in the dataset. It tells us how many of the actual positive instances were correctly predicted by the model. Recall is calculated as: (4)${\displaystyle recall=\frac{TP}{TP+FN}}$

The accuracy and recall are taken into account while calculating the F score, which has a range of 0 to 1. It considers both precision and recall and is regarded as a more important metric, especially when the dataset is imbalanced. It is computed using (5)${\displaystyle F_1=2\times \frac{precision\times recall}{precision+recall}}$

Performance evaluation of machine learning models

Numerous experiments are conducted to assess the effectiveness of the selected machine learning classifiers, as well as the transfer learning classifiers. The experiments are divided into three groups: experiments involving TF features, experiments involving TF/IDF features, and experiments involving transfer learning models.

Table 7 displays the performance of machine learning models using TF features. The results demonstrate that the ET model outperforms other models significantly achieving a 75% accuracy score due to its ensemble-boosting architecture. Even with small amounts of data, the ET model excels in accuracy compared to all other models. XGB follows closely behind with a 74% accuracy score, indicating that linear models can also perform well with TF features. GBM, on the other hand, exhibits the lowest performance, achieving a 70% accuracy score.

The results of machine learning models using TF-IDF features are presented in Table 8. The results demonstrate that the performance of these models improves when utilizing TF-IDF features. TF-IDF, which identifies weighted features, enhances the feature vector used for training the machine learning models, whereas TF alone provides a basic feature vector. Once again, the ET model exhibits superior performance compared to other models in terms of accuracy, precision, recall, and F1 score when using TF-IDF features, achieving an accuracy rate of 85%. GBM and AB models closely follow with an accuracy of 82%. However, XGB performs poorly when employed with TF-IDF features.

In the third set of experiments, transfer learning models are employed. Four transfer learning models are utilized, and their performance is evaluated based on accuracy, precision, recall, and F1 score. The outcomes of these transfer learning models are given in Table 9. The results indicate that the transfer learning model ELMo outperforms all the other learning models used in the study, achieving an impressive accuracy of 96%. It is followed by the BERT which achieved an accuracy of 94%. Other learning models, XLNet and RoBERTa achieved accuracy of 93% and 92%, respectively.

Methodology adopted to evaluate prediction performance

Since higher ratings tend to attract more new users, user ratings on the Android Play store can be prejudiced or even extravagant. Table 10 provides examples that demonstrate a significant difference between reviews and ratings. However, a systematic evaluation of this phenomenon is required to gain a more comprehensive understanding.

This study introduces an algorithm that utilizes TextBlob to determine the points of divergence between user reviews and an app’s rating. Figure 9 presents the flowchart illustrating the methodology adopted in this study. The implementation of this technique is done using Python, resulting in the creation of a named tuple called Sentiment (polarity, subjectivity) obtained from the sentiment property of TextBlob (Loria, 2018). The subjectivity score is a floating-point number ranging from 0.0 to 1.0, where 1.0 indicates high subjectivity and 0.0 represents strong objectivity. The polarity score ranging from 1.0 to 1.0, is also a floating-point number. For example

from textblob import TextBlob

tbl = TextBlob(“He is honest. i love him ”)

tbl.sentiment()

The output of the above code is a sentiment polarity value of 0.9 and a subjectivity value of 0.7)

To detect possible biases, polarity calculations are performed on every review and then compared to the corresponding rating in the dataset. If the polarity of a review is found to be less than 0.5 and the associated rating is over 3, we consider the rating by the user to be biased. Conversely, if the polarity of the review is equal to or greater than 0.5, or the rating is less than or equal to 3, we consider the rating to be impartial. This approach allowed us to identify potential biases based on the relationship between the review’s polarity and the attached rating.

The experimental results of the proposed approach are presented in Table 11. Considering high ratings in influencing new users, this study specifically investigates the discrepancies in 3-star and above ratings and their equivalent evaluations. Among the analyzed 502,658 user ratings, the research identifies that 124,238 rating counts exhibited biases. This implies that approximately 24.7162% of the overall ratings are biased, within the selected app categories while the remaining 75.2838% represent unbiased reviews.

Google apps reviews numeric rating prediction using transfer learning models

As we can observe from Tables 7, 8 and 9, the best performing models are ElMo, BERT, XLNet, and RoBERTa. So, for phase 2 this research will make use of these transfer learning models for numeric rating prediction by analyzing user reviews. These predicted numeric ratings are compared to the overall ratings (aggregate ratings) of the respective applications on the Google Play Store. This comparison is done to identify any inconsistencies between the ratings and the user reviews. Table 12 presents the numeric ratings prediction by the transfer learning models selected for the experiment.

Bold phrases indicate classifier predictions that are closest to the actual ratings. The bold figures in Table 12 represent the terms that are expected to have the highest ratings. It is noteworthy that the ELMo classifier predicts the highest scores for 14 out of the 14 categories. Additionally, ELMo’s projected ratings are relatively closer to the aggregate ratings compared to the other classifiers.

The Google applications with the lowest expected ratings are XLNet and RoBERTa. Both these models are computationally complex and also require a large amount of data for training. XLNet also faces scalability issues with lesser data as in the current case. While RoBERTa requires fine-tuning of hyperparameters according to the nature of the data. These are the major concerns that cause the lower rating prediction of Google app data from these two models.

The results of the ELMo learning model reveal the differences between the user-specified numeric rating and the accompanying reviews. It is observed that the numeric rating provided by users is approximately 25% higher in accuracy compared to the predictions generated by ELMo.