Mobile-UI-Repair: a deep learning based UI smell detection technique for mobile user interface

Data collection

We obtained a dataset for our study from three different platforms. The first and main dataset which publicly available for research and development called Rico dataset (http://interactionmining.org/rico#quick-downloads)  (Deka et al., 2017) which is used for training and testing purposes. The second one is the crowd-sourcing platform Baidu (https://baidu.com). It is also the largest platform where users test various mobile applications. The platform’s employees are given jobs to test the applications, and then submit reports outlining their testing and observations made during testing. Between January 2021 and August 2023, we collected 1,443 of these testing tasks for Android mobile applications that were documented in our dataset. The testing assignment required the employees to provide several reports explaining the testing procedure as well as images of the user interface for the program. This dataset was chosen because it contains both screenshots of the user interface and explanations of UI bugs or problems that were discovered during testing. This makes it easier for us to search and examine any issues in the UI having display issues. The third platform is software houses which in minor in count. We consider mobile applications that have more than 5,000 downloads and at least have more than 4.3 star ratings from several areas, including news, entertainment, health, media and finance for example StarzPlay, Perfect Piano, Secure VPN, MediaFire and CEToolbox. We collected 217 UI screenshots from software houses having different UI display issues. In total, we collected 1660 UI screenshots having different UI bugs. These all UI shots were a part of the validation and real-time testing of the M-UI-R approach.

Categories and visual understandings of UI design smell

There are several categories of UI display issues for example navigation problems, component occlusion, text overlap, missing image, null values, blur screen, background image size that make foreground button text invisible, inappropriate button and icon size. The bugs other than component occlusion, text overlap, null values and missing images can be solved by following the design guidelines of application UI development (Yang et al., 2021). However, the targeted UI bugs (component occlusion, missing images, null values and text overlap) can not be identified by following the design rules. The respective UI bugs as explained in Fig. 2 and Table 1. There are multiple reasons for the selection of these bugs for example impact on user experience, frequency of occurrence and also these bugs have negative effects on the usability and functionality of mobile applications. Recent studies (Wang et al., 2022; Ali et al., 2024) explored UI bugs but they performed analysis on user reviews and feedback available on relevant forums such as Google Play and App Store (https://play.google.com/console/about/reviews/). They used textual features and reported these bugs. These are the factors that motivated us to select these visual design bugs for this research.

The above factors illustrate the severity of GUI bugs and also motivate us to design an automatic approach to successfully identify and locate these UI bugs. A commonly utilized technique for bug detection in mobile applications is to perform program analysis, building of new rules, code rewriting for different platforms such as Android and iOS and customization of code to ensure compatibility on different mobile devices (for example OPPO, Samsung, etc.). This process is time-consuming and labor expensive. Especially the task of analyzing all possible display problems and designing their detection rules is not easy.

We examined the screenshots and corresponding JSON files for these four categories of UI bugs. We discovered that text overlap, component occlusion and null value bugs can be identified by analyzing the component information extracted from JSON files (such as component text and coordinates of relevant components). We performed static analysis of JSON file data to identify these three types of bugs using static analysis of XML files. For text overlap analysis we perform analysis of all text views if the coordinates of any two text views overlap then there is a bug same as for component occlusion the coordinates of the component are analyzed if there is an intersection between any two components coordinates then component occlusion bug existed between them. Null value bug identification is very easy, we just examine text of the component if there is “null” text then it is a bug.

Furthermore, the static analysis approaches have many limitations. In certain scenarios, it becomes very challenging to detect these UI bugs just by performing the simple analysis of JSON files. For instance, in the case of component occlusion (as illustrated in Fig. 3), the font size information is not available in the JSON file, making it impossible to detect this bug where the font size is incompletely displayed in an EditText. Additionally, as depicted in Fig. 3 again, elements such as dialog box, toolbar and spinner may overlay the component, creating interference in the detection process. Similarly, for text overlap issues, there is potential interference from component occlusion. Regarding null value problems, the text extraction process may yield numerous null values, many of which stem from issues during JSON file retrieval rather than actual UI display problems. This introduces considerable noise into the bug detection process.

Considering this perspective, it is valuable to implement efficient, fast and versatile method for detecting UI display bugs. Drawing inspiration from the fact that humans can recognize these display issues visually, we suggest identifying these problematic screenshots using a visual understanding technique that mimics the human visual system. Since UI screenshots can be obtained easily (either manually or automatically) and generally do not exhibit significant variations across apps from various platforms or devices, our computer vision-based approach offers greater flexibility and ease of implementation.

Research questions

• Research question 1: How effective is our M-UI-R approach in the detection of UI display bugs?

• Research question 2: How effective is our M-UI-R approach in the localization of UI display bugs?

• Research question 3: Is the information extracted by using our M-UI-R approach useful for improving mobile applications?

Experimental setup

Since our Proposed approach “M-UI-R” is fully automated and we used an already published dataset called Rico dataset (Deka et al., 2017). We also collected data from crowd-sourcing platforms and software houses for real-time analysis of the proposed approach. To create a balanced dataset we used an equal number of images (screenshots) from each category. To make our approach effective we chose 8,000 screens from every category and in total we used 32,000 screens for analysis, as shown in Table 1. Among these 32,000 half are positive and half are negative, positive means without bugs and negative means with bugs. we used 80% data as a training dataset and 20% for testing and validation.

In addition to checking the real-time performance of our approach, we employed on real-time dataset obtained from crowd-testing apps. This dataset includes 800 screenshots including 400 buggy and 400 without bugs for testing. We exploited 5-fold cross-validation.

Baselines

We conducted a comparison between our M-UI-R approach and five baseline methods that combine machine learning and deep learning techniques. This was done to emphasize the advantages of our proposed approach. Among the baselines, three employ machine learning techniques to extract visual effects from the screenshots and subsequently apply learning methods for categorization. The remaining two baseline approaches directly employ deep learning concepts, using artificial neural networks for classification tasks. Our primary focus is to elucidate the feature extraction procedure utilized in machine learning-based techniques.

The scale invariant feature transform (SIFT) (Lowe, 2004): represents a widely used method for extracting features. It identifies and characterizes local features within an image. This technique isolates intriguing points on an object to generate a feature description that remains unaffected by uniform scaling, orientation variations, and changes in illumination.

Speeded-up robust features (SURF) (Bay, Tuytelaars & Van Gool, 2006): represents an enhancement of the SIFT technique. Within SURF, the Hessian blob detector is approximated using integer values. This approximation can be achieved with the assistance of a precomputed integral image, requiring only three integer operations for its computation.

The oriented fast and rotated brief (ORB) (Rublee et al., 2011): this technique characterized by its speed in both feature point extraction and description, is detailed in the ORB article. ORB builds upon the foundation of BRIEF, a swift binary descriptor. Notably, ORB incorporates rotation invariance and robustness against noise interference. Following this, we apply four widely used machine learning methods utilizing these extracted features: support vector machine (SVM) (Kotsiantis, Zaharakis & Pintelas, 2007), k-nearest neighbor (KNN) (Sumiran, 2018), naive Bayes (NB) (Kotsiantis, Zaharakis & Pintelas, 2007), and random forests (RF) (Breiman, 2001). These methods are employed to classify screenshots depicting UI display issues.

MLP (Zhu et al., 2021; Driss et al., 2017): the multilayer perceptron (MLP) forms a feedforward artificial neural network. This network architecture comprises an input layer, hidden layers, and an output layer. Neurons within this structure operate with a non-linear activation function similar to the rectified linear unit (ReLU). During the training process, the connection weights are adjusted while considering the output’s deviation from the ground truth. In our approach, we employ an eight-layer neural network with the following neuron counts in each layer: 190, 190, 128, 128, 64, 64, 32, and 2.

OwlEye (Liu et al., 2020): OwlEye utilizes a convolutional neural network (CNN) as its foundation to recognize the screenshots containing UI display bugs. Additionally, it employs Gradient weighted Class Activation Mapping function (Grad-CAM) to pinpoint the areas where UI design issues occur.

Evaluation metrics

We employ three commonly used evaluation metrics: precision, recall, and F1-score, to assess the effectiveness of our proposed research in addressing the challenges outlined in RQ1. These metrics find widespread application in the domains of pattern recognition and image classification (Zein, Salleh & Grundy, 2016; Chen et al., 2019a). A higher value always indicates good performance.

Precision and recall are typically calculated by aggregating the counts of true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN). Within the context of issue identification, TP corresponds to correctly labeled as buggy during prediction, FN to inadvertently mislabeled non-bug images as buggy, TN to correctly identified normal screenshots, and FP to incorrectly labeled as non-buggy screenshots.

Precision signifies the ratio of correctly predicted UI display issue-containing screenshots to all the screenshots predicted as buggy: (1)${\displaystyle Precision=\frac{TP}{TP+FP}.}$

Recall denotes the ratio of accurately predicted buggy screenshots from all the screenshots that genuinely exhibit UI display issues. (2)${\displaystyle Recall=\frac{TP}{TP+FN}.}$

The F1-score (also known as F-measure or F1) represents the harmonic average of precision and recall, effectively combining both of the aforementioned metrics. (3)${\displaystyle F-measure=\frac{2\times Precision\times Reccall}{Precision+Recall}.}$

For issue localization, we utilize two frequently employed evaluation metrics such as average precision (AP) and average recall (AR) which are commonly used in the context of object detection (Ren et al., 2015). These metrics are applied to assess the efficacy of our proposed approach in addressing the localization challenges described in RQ2. AP and AR offer a more precise and comprehensive evaluation of M-UI-R localization capabilities. Larger values consistently denote improved performance. While precision and recall are analogous in image categorization, AP and AR serve distinct evaluation purposes.

We start by selecting prediction boxes with confidence values exceeding 0.55 (Ren et al., 2015). Subsequently, we assess the overlap between the predicted UI visual smelly region and the actual smelly region by using the intersection over union (IoU) ratio. This ratio is calculated using the following formula: IoU equals the intersection of the predicted UI smelly region and the actual UI smelly region divided by the union of these two. IoU effectively addresses coverage issues. The TP refer to the count of identification boxes with an IoU of 0.55 or higher. The same false positives encompass detection boxes, including redundant ones identified within the same ground truth box, that have an IoU less than 0.55. The term FN represents the number of ground truth boxes that are not detected.

UI smell detection performance (research question 1)

Table 2 presents the initial assessment of our proposed Mobile-UI-Repair (M-UI-R) approach, both in terms of fault detection and its performance across four distinct types of display issues of mobile UI within the training dataset and the real-world dataset (collected from crowd-sourcing and software houses). Figure 5 represents the precision and recall performance of UI bug detection on different size of training data. M-UI-R attains an average precision (AP) of 0.877 in the training dataset, denoting that 87.7% (702 out of 800) of the predicted problematic screenshots genuinely exhibit UI display faults. Furthermore, the average recall (AR) stands at 0.865, signifying that M-UI-R adeptly identifies 86.5% (692 out of 800) defective screenshots. In the real-world dataset, M-UI-R yields an average precision of 0.856. This signifies that, on average, 85.6% percent (186 out of 217) of the identified screenshots with UI display faults do indeed exhibit such issues. The average recall for the real-world dataset stands at 0.843, highlighting that M-UI-R effectively detects 84.3% (183 out of 217) of the problematic screenshots.

Even though M-UI-R underwent training due to a very small dataset that’s why its average precision and recall on the real dataset were slightly lower, measured at 0.021 and 0.022, respectively, compared to those in the training dataset. This observation serves to validate the effectiveness of our M-UI-R system.

We shift our attention to the upper section of Table 2, where we delve into the performance metrics for each distinct category of UI display difficulties. It is worth highlighting that the precision and recall measurements for all four categories are quite commendable. While the lowest recorded precision is 82.5% and the recall value is 77.8%, respectively, the highest precision value is 91.9% and the recall value is 93.8%. Same like the “null value” category of UI smell stands out with the highest F-measure, showcasing a remarkable balance between precision (0.87) and recall (0.93). This observation could be attributed to the recurring nature of null value issues in screenshots, which often makes identifying the problematic areas relatively straightforward. In contrast, the “component occlusion” category of UI smell attained the lowest value of precision and F-measure respectively. This distinction is due to the diversified pattern of each category of bug, as well as the relatively smaller size of the buggy area for example in the case of component occlusion the buggy area is only 8.5%. While in case of missing images, it is visible and can be identified easily. In future work, we will try to improve the identification and localization performance with image magnification.

Furthermore, a detailed analysis of screenshots that were erroneously labeled as bug-free has been carried out, and illustrative instances are explained in motivational study section. These images often consist of very few problematic regions that are challenging to discern, even for human observers. Our upcoming efforts will concentrate on incorporating attention mechanisms and image magnification techniques to enhance the detection performance for such types of screenshots.

Issue localization performance (research question 2)

Illustrations showcasing our problem localization are displayed in Fig. 6, concentrating on the precise positions of the identified bugs. OwlEye (Liu et al., 2020) localization outcomes are visualized as heat maps, prompting us to employ image binarization for delineating the bounding box encompassing the highlighted region within the heat map. This facilitates a direct comparison of its efficacy with the novel approach proposed by M-UI-R. The efficiency of M-UI-R problem localization is outlined in Table 3. The average values for issues localization for real and testing datasets are AP (average precision) and AR (average recall) achieved by M-UI-R are 71.5% and 69.7%, respectively.

Let us now thoroughly examine the performance of problem localization for each grouping of UI display problems. Across all four types, both AP and AR demonstrate notably high values. Within the real dataset, the lowest recorded average precision and average recall stand at 67.3% and 65.5%, respectively. Notably, the “missing image” category showcases the most outstanding performance, displaying statistically significant AP (75.7%) and AR (74.5%) values. This could be attributed to the substantial and conspicuous buggy regions present in screenshots with missing image problems. Conversely, the “component occlusion” category displays comparatively lower performance, with AP and AR values of 71.5% and 69.5%, respectively. This discrepancy could be attributed to the significantly smaller buggy regions found within this category.

As depicted in Fig. 6, the red bounding box represents the ground truth data, whereas the same bounding box represents the prediction. While the projected bounding box size might be slightly larger than the actual one, leading to an IoU value below 0.55, the indicated buggy area is mostly accurate. This offers valuable insights to developers.

It is important to note that the differing assessment criteria significantly contribute to baseline considerably poorer AP and AR outcomes. In prior studies, participants were tasked with evaluating whether the localized region by OwlEye (Liu et al., 2020) coincided with the actual issue area, serving as a measure of UI display issue localization. The highlighted problematic area was required to share at least 45% of its similarities with the real issue region, whereas AP and AR metrics necessitate an IoU area exceeding 0.55. Notably, we observe three instances where AP/AR metrics suggest lower localization precision, yet human evaluation indicates strong performance. In the initial case, OwlEye highlights only one out of three potential issue sites, while M-UI-R accurately identifies all three, yielding superior AP/AR metrics compared to OwlEye. In the second scenario, there is localization noise, and OwlEye incorrectly designates a different location, leading to decreased AP/AR values compared to M-UI-R. The third situation involves localization drift, where OwlEye highlighted area is not perfectly aligned with the target, while M-UI-R adeptly identifies the entire problem area.

Performance comparison with baselines

The comparison of performance of our proposed approach with baselines is presented in Table 4. The initial focus is on opposing M-UI-R against the five baselines explained in the experimental setup section. The results indicate the superiority of our proposed approach M-UI-R over the five baselines. Specifically, M-UI-R boasts a precision 28.8% higher and a recall 21.7% higher than the baseline article multilayer perceptron (Zhu et al., 2021) and also outperform 10.8% in precision and 11.7% in recall then OwlEye (Liu et al., 2020). The other three machine learning baselines demonstrate their performance in terms of precision and recall and f-measure as shown in Table 4. M-UI-R stand superior at least 30% in precision and more than 30% in recall among all three machine learning based baselines (SIFT, SURF and ORF). We reaffirm the effectiveness of M-UI-R, underscoring its adeptness in identifying problematic screenshots from a pool of candidates. Notably, multilayer perceptron (MLP) achieves the highest precision, F-measures, and recall from all the baselines, suggesting the superior performance of this deep learning approach in recognizing faulty screenshots.

Furthermore, a comparison between the performance of the freshly designed model within M-UI-R and the prior model used in OwlEye, utilizing the same data for training and testing, is conducted. The findings demonstrate that the M-UI-R model attains enhanced performance, achieving a precision of 87.7% versus 79.0% and a recall of 86.5% versus 77.0%. The evident improvement in precision and recall within M-UI-R can potentially be attributed to the detection of object assignment, which offers more precise bounding boxes for UI display problems during model training. This refined data provision facilitates the model in learning relevant features more effectively. In contrast, other baseline methods rely solely on category labels for their training data. M-UI-R robust performance in issue detection, coupled with its capacity to adapt without requiring manual data labeling, makes it particularly adept at accommodating the varied styles of Android UI.

Usefulness evaluation (research question 3)

To comprehensively assess the effectiveness of our M-UI-R solution, we carried out a random sampling of 3,200 Android applications, 1,600 from FDroid6 and 1,600 from the Google Play store. These applications included newer releases of mobile applications for 2019-2023. Notably, none of these apps were included in our training dataset.

We examined these mobile applications and captured screenshots of their UI pages using DroidBot, a well-known and lightweight Android test input generator (Li et al., 2017). Out of the 3,200 applications, only 40% (1,280 out of 3,200) generated multiple screenshots, while 60% (1,920 out of 3,200) were successfully evaluated using DroidBot. The remaining portion of applications, for which DroidBot could not proceed further, typically required user registration or authentication steps. On average, eighteen screenshots were gathered for each of the 1,280 applications. Subsequently, we fed these screenshots into M-UI-R to identify potential UI display issues. Upon detecting an issue, we created a comprehensive bug report outlining the problem and included the relevant problematic UI screenshot. The development teams of these applications were then notified of these bug reports via email or issue tracking systems. M-UI-R identifies and provides a comprehensive listing of all the detected issues, and this approach offers supplementary detailed information regarding the identified defects. A total of 97 UI display bugs were detected in F-Droid applications, with 27 of them already fixed and 18 having received acknowledgments from developers. Similarly, in the case of the Google Play store, 76 UI display issues were uncovered, resulting in 37 fixes and nine developer acknowledgments. The rectification of these reported imperfections stands as tangible evidence of the practicality and effectiveness of our proposed approach for detecting UI display problems.

The findings reveal that M-UI-RT not only identifies 79 additional issues that OwlEye misses but also encompasses all the issues detected by OwlEye. The precision of M-UI-R stands at 87% (171 out of 197), marking a 16% improvement over OwlEye precision (87% compared to 71%). M-UI-R demonstrates superior performance in detecting problems with relatively smaller problematic regions. The significance of UI display issues on user experience is further highlighted by developers validation of the issue reports provided by M-UI-R, underscoring the crucial role of M-UI-R in identifying issues.

We also conducted the survey from eight app developers to analyze whether the information extracted by our method can provide help for improving the UI of apps. Initially, we selected 10 buggy screenshots from each category of bugs randomly and applied M-UI-R to identify and localize UI display issues. Subsequently, we presented the buggy screenshots along with the acquired information to developers and pose them questions outlined in Table 5. Specifically, the research question (RQ1) assesses the identification of UI display issues, while the research question (RQ2) evaluates the localization of UI display issues. Each question was accompanied by five options (1 strongly disagree, 2 disagree, 3 neither, 4 agree, and 5 strongly agree).

The developer’s feedback is also explained in Table 5. Examining the response to RQ1, it is evident that seven out of eight developers agreed that M-UI-R is beneficial for the identification of UI display issues, with six of them expressing strong agreement regarding the usefulness of the information. Only one developer expressed a conservative response. Similarly, for RQ2, positive feedback was received from six developers out of eight who agreed with the usability of M-UI-R for issue localization. When developers watch the bugs that are detected by M-UI-R they said that these UI bugs have a lot of impact on usability of mobile apps and need to be fixed immediately. These suggestions from developers also validate the usability of M-UI-R.

Threats to validity

We acknowledge potential threats to the construct validity of our proposed approach, particularly concerning the choice of evaluation metrics. The metrics we employ are precision, recall, and F-measure which are widely used in the research community. However, it is important to recognize that the extensive reliance on these metrics may introduce limitations in terms of construct validity.

Furthermore, we are aware of the susceptibility of classification algorithms to validity threats stemming from parameter values. To address this concern, we conducted experiments to determine variations in hyperparameter settings such as batch size or learning rate rather than relying on default values. Nevertheless, it should be noted that adjustments to these parameters could potentially impact the results.

The use of the Faster-RCNN model for the identification and localization of UI bugs introduces a potential threat to construct validity. While there are alternative tools available, we opted for Faster-RCNN due to its superior performance compared to other models available at the time. Nonetheless, it is crucial to acknowledge that the absence of comprehensive object detection tools for UI bug detection could affect the overall performance of our proposed approach.

Internal validity concerns arise from the implementation of our approach. To mitigate this, we perform cross-checks to ensure the accuracy of our methodology. However, there remains a possibility that some errors may have been overlooked.

In terms of external validity, there are considerations regarding the generalization of our proposed approach. Although our analysis is confined to mobile application UI bugs, our approach could be applicable to all platforms for the detection of UI bugs.

Lastly, the limited number of projects in our study poses an external validity threat. Deep learning algorithms often require fine-tuning parameters and substantial training data to achieve optimal performance. The restricted number of projects may therefore limit the generalizability of our results and our ability to fully explore. We would also discuss the different categories of applications, like testing a streaming application might be different with respect to an accountability, utility, social or chat application.

A limitation of our proposed study is that it can not identify the design smell related to the style of UI, layout bugs of UI, button size, typography, iconography and navigation design flaws. These style bugs also have a great impact on the user experience of mobile applications. We will consider these bugs in our future research.