JDroid: Android malware detection using hybrid opcode feature vector

Overview

This study used hybrid feature vector generation and stacked model-based classifier design for Android malware detection. The details of this design are shown in Fig. 1. The methodology mainly consists of dataset collection, preprocessing, feature engineering, and classifier design. In this study, a unique feature engineering approach using a static analysis-based code analysis approach and hybrid feature vector generation and stacked model-based classifier design for Android malware detection has been proposed. The details of this design are shown in Fig. 1. The methodology mainly consists of dataset collection, preprocessing, feature engineering, and classifier design. The first stage of the model, a comprehensive dataset containing three benign and two malicious applications, has been created. Thus, it is aimed to analyze the proposed method without any dependency on the data or dataset. Each application in the dataset is an application file packaged with an APK extension, and it is not possible to perform direct code analysis on these files. At this stage, the application must be opened with reverse engineering methods, and the source code must be accessed. In this study, unlike the studies in the literature, code analysis was performed on SMALI code instead of analysis on Java code. For this purpose, the apktool tool (Arslan, Doğru & Barişçi, 2019) was used. To extract the Java bytecode and Dalvik bytecodes, each application was decompiled using apktool. With the decompilation process, assets, lib, smali, original, res, META-INF, and Androidmanifest.xml files are obtained. The Smali folder contains the files used for the proposed model. Depending on the application design, different numbers of *.smali files are received in the Smali folder. On the Smali codes, which of the 257 different Dalvik OpCodes (invoke-virtual, sub-long/2addr, etc.) were used at least once were scanned and analyzed, and a feature vector was obtained. The feature value was determined as 1 for the used OpCodes, and the numerical value of the feature was defined as 0 for the unused features. Then, the same analysis process was repeated for 204 Java ByteCodes, and their numerical values were set as 1 or 0 according to the usage situation. Thus, two different feature vectors were obtained for the same application. Then, these two feature vectors were combined by concatenation, and a feature vector containing a total of 461 features, the values of which were determined as 0 or 1 for each application, was obtained. Combining both feature vectors has made a hybrid feature vector design with more distinctive features about the application.

The hybrid feature vector was used in the training and testing processes of the stacking generalized model. The model is in a two-layer architecture, and the first layer, random forest, extra tree, multilayer perceptron, and gradient boosting classifier are used. The XGB classifier is used as the decision maker in the second layer. The main reason for these choices is that the performance increase cannot be achieved in the ensemble structure with models such as KNN and SVM, which received high results in the first tests. For this reason, an architecture consisting of four models and 1 MLP structure in a tree architecture was designed and tested. Although tree-based models normally perform less than others, much higher accuracy values have been achieved within the designed architecture. The architectural structure was created entirely according to the performance value obtained from preliminary tests. An artificial neural network-based classifier was selected and added to the proposed 2-layer architecture to prevent all classifiers from being tree-based. In addition, in the 2-layer structure proposed in this study, the samples used for training in the 1st layer and those used for testing in the 2nd layer must be completely separated. Because in the testing phase of the 2nd layer, the training examples in the 1st layer are learned. If the training samples in layer 1 are used for testing in layer 2, it will cause overfitting. For this reason, training and test samples are separated from the beginning, as shown in Fig. 1. With complex machine learning models, it’s sometimes easy not to pay enough attention and use the same data at different pipeline steps. This can often lead to good but unrealistic performance or cause strange side effects in others. Therefore, test samples are never used in the training processes of four different classifiers in the 1st layer and the classifier in the decision-making layer in the 2nd layer. In this evaluation, the results obtained when Dalvik OpCode, Java Byte code, and hybrid feature vector are used are evaluated comparatively, and the results are shown in detail in the next section.

Opcode extraction and feature engineering

Dalvik Virtual Machine (DVM) is a virtual environment in which applications coded in Java and packaged with *.apk extension are run for Android operating system devices. Google prefers DVM in the Android operating system instead of the traditional JVM. The reasons for this are (1) being an Apache-licensed virtual machine rather than GPL, (2) the JVM not being suitable for IoT devices due to high resource needs, and (3) the DVM needing less memory and processing time than the JVM. DVM contains processing commands similar to JVM, but needs to be compiled once with the dex compiler after generating the Java byte code. On the other hand, JVM is a virtual environment where software coded in Java can run. It understands Java byte codes and enables them to work on different platforms. Javac is responsible for converting Java code to Java byte codes.

Static analysis refers to extracting and understanding the file containing metadata, source code, and other information within the application package. The Dalvik Executable (DEX) format, the basic format of Android operating systems, includes the code embedded in the Android application file. Different algorithms to parse this compiled code require a linear scan or recursive flows. As a result of this disassembling process, additional files such as assets, libs, smali, res, META-INF, and Manifest.xml are obtained for each Android application. Since the disassembling process requires understanding the compiled code, it involves using the appropriate tool, depending on the architecture. Apktool is one of the most common tools used in this field and was also used in this study. After the disassembled code is obtained, it is possible to use different analysis approaches to analyze this code. These are heuristic approaches: Call Graph, Control-Flow-Graph, N-gram, image-based approach, and code-based modeling methods. This study obtained the *.smali file due to the disassembling process, and a code-based attribute approach was adopted.

In this study, a hybrid feature vector proposal has been introduced with the idea that it can be examined in Java byte codes (Leroy, 2003) together with Dalvik OpCode (Li et al., 2023) in the classification of Android applications created in the Java language and can provide information about the application. Although *.apk files are packaged to run on DVM in the Android operating system, they are in Java bytecode in the previous stage. Moreover, *.smali files produced after decompiling APK files contain Dalvik and Java byte code sequences. Analyzing smali files allows both opcodes to be searched and converted into features. The transaction processes followed in this regard are shown in Fig. 2.

To extract the application features, the smali files of the applications were accessed using the apktool at the first stage. The critical point is that depending on the application’s design, many smali files are in separate folders in the application folder. For this process, software was prepared that automatically scans and combines smali codes and investigates the use of Dalvik opcodes and Java bytecode. Thus, all applications were quickly analyzed, and feature vectors were produced.

In the Dalvik OpCode-based feature vector, a 1 × 257 vector was created for each application, with each of the 257 OpCodes being one if used at least once and zero otherwise. In the Java Bytecode-based feature vector, a 1 × 205 vector was created with the same process. Thus, a fixed-length vector for each application and standardized for classification was obtained. In the last stage, the hybrid feature vector production proposed in this study was performed. The vector contains 461 features for each application.

When other studies in the literature are examined, either of the classes.DEX files have been analyzed to determine the use of Dalvik OpCodes, or only the Dalvik OpCode has been searched in the smali operation code files and used in the classification phase. However, as shown in Fig. 3, Java Byte codes are in the same file. Based on this idea, a hybrid feature vector was created and used for classification. It aims to outperform Dalvik OpCode-based classification by obtaining more application features. At this point, it should be noted that the 1 × 461 hybrid feature vector will require more processing time than Dalvik and Java byte code. To overcome this problem, feature selection has been made instead of evaluating all features, and classification with fewer features has been preferred. Thus, a hybrid feature vector with features that contribute more positively to classification is obtained from both Dalvik OpCode features and Java Bytecode features. As a result, a vector with 256 features for Dalvik OpCode, 204 features for Java Byte code, and 461 features for the Hybrid feature vector is obtained. The primary purpose of this study is to achieve higher performance thanks to more features, and the feature selection process is performed on a huge number of features obtained. Extra Tree classifier is used for this process. Accordingly, the best feature is selected so that the decision tree can be divided according to the Gini index. The feature selection process is performed using this index calculation. In this way, higher malware detection is achieved through fewer and more meaningful features. Accordingly, the best feature is selected so that the decision tree can be divided according to the Gini index. The feature selection process is performed using this index calculation. In this way, higher malware detection is provided thanks to fewer and more meaningful features. There are many feature selection methods available other than ExtraTree, but ExtraTree is quite successful in understanding complex and non-linear relationships between features and classes. In the problem we are addressing, we aim to capture a relationship between the application and the opcodes it uses, and this relationship cannot be linear. However, approaches such as LASSO regression and chi-square assume the existence of a linear relationship, and parameter adjustment is quite sensitive. In addition, ExtraTree is computationally efficient compared to recursive feature elimination (RFE) and, more importantly, it is less affected by noise than the mutual information (MI) feature selection method in high-dimensional datasets. In addition, ExtraTree models have a structure that can be used to embed during training. It instantly ranks and selects features according to importance, so unlike principal component analysis (PCA), the selected features are clear and interpretable. Selected features can be analyzed collectively for malware and benign software. The selected features are directly fed into JDroid’s ensemble model structure.

Environmental tools

The Python language, which researchers widely use, was used for hybrid feature vectors and stacking ensemble model design and classification tests. The following libraries were mainly used to develop in the Python language:

• Numpy: used to manipulate matrices.

• Panda is a data processing library; this work is used to process features obtained from Android application files.

• Scikit-learn (Fabian et al., 2011): It is an open-source library that hosts machine learning classifiers. This study was used to implement the proposed 12 different classifiers and one stacked ensemble model design.

In addition, Java-based coding was also used to extract approximately 14 thousand applications and automatically generate feature vectors. The experiments used a notebook and CPU (Core i7 Intel 11390H), 32 GB RAM, and a Windows 10 operating system.

Hyperparameter tuning

The proposed model was tested with different classifiers, and the results were compared. GridSearch-CV in the scikit-learn library was used to evaluate TFP and FPR rates automatically and adjust hyperparameters. The CV property was set to 10, and 10-fold cross-validation was performed. For cross-validation, it is necessary to ensure that there are no test examples during training. For this reason, in this study, cross-validation was performed separately for each dataset. In this way, accessing data from the test set during training is prevented, and data leakage is prevented. As a result, the optimal parameters were determined for each classifier and are given below.

• Logistic regression (LR): C:1.0, kernel:linear, probability=True, gamma:scale

• k-nearest neighbors (KNN): n_neighbors=5, p=2, metric=‘minkowski’

• Support vector machines (SVM): kernel=‘linear’, C=1.0, random_state=0

• Decision tree (C 4.5):

• Gaussian naive Bayes (GNB): var_smoothing:1e−9

• Multilayer perceptron (MLP): max_iter=1,000, activation=‘relu’, alpha=0.5, hidden_layer_sizes= (10, 20, 10), learning_rate= ‘adaptive’, solver= ‘adam’

• AdaBoost (AB): n_estimators:50, learning_rate:0.1

• Gradient boosting (GB): loss:‘log_loss’, n_estimators:50, learning_rate:0.1

• Random forest (RF): criterion:‘gini’, n_estimators:50, random_state:3

• ExtraTree Classifier (ET): criterion= ‘entropy’, max_depth=8

• XGBoosting (XGB): learning_rate = 0.01, n_estimators=10, max_depth=5, min_child_weight=1, gamma=0

• Linear discriminant analysis (LDA): solver: svd, shrinkage: auto

Dataset design

A heterogeneous and balanced dataset consisting of different datasets was created to be used in this study. Selected datasets have been used in many studies before, and the results have been shared. The number of applications related to the datasets, APK file size, file size after conversion to Java code, file size after conversion to Smali code, and total number of files in Java language were shown in Table 2 in detail. Java ByteCode sequences and Dalvik ByteCode sequences of all applications have been extracted and tested using only Smali files.

Beneficial applications refer to reliable applications that are made available to meet user requirements. In this study, MalDroid2020, CICInvesAndMal2019, and Omer, which have been used and validated in various studies, were used. As given in Table 2, there are a total of 8,176 applications in these datasets. The main problem with applications is that they can have similar Dalvik bytecode and Java bytecode calls. This may cause the training and test samples to be the same and cause overfitting. Other apps may have the same feature sets. For this reason, the same ones were eliminated. When the data was examined, it was determined that 1,151 of them were the same and repetitive applications were removed from the dataset, and as a result, a benign dataset containing 7,025 APKs was created. Since there was no severe sample selection imbalance during the random selection of the training and test sets, stratified sampling was not used. Because when randomly selected, the number of samples for training and test samples is approximately the same.

While malicious apps meet user requirements, they mainly aim to harm users. Often, these purposes are hidden in a part of the application. Ways of harm can be executing malicious program code without the user’s knowledge, repeated execution, or lingering on the device for longer, making it difficult to detect. This study used 6,774 of the 6,780 applications in the Drebin and Genome datasets.

The dataset, compiled from multiple malware datasets and comprising approximately 13,799 APKs (7,025 Benign and 6,774 Malware), was randomly divided into two subsets for training and testing the models. In this context, 70% of the data was allocated to the training set to facilitate the model’s learning process. In comparison, the remaining 30% was assigned to the test set to assess the model’s accuracy and its ability to generalise.

Malware detection with Java feature vector, Dalvik feature vector and Hybrid vector for all classifiers

This study investigated the effect of hybrid feature vector design on Android malware detection. Unlike other studies, using both Dalvik OpCode and Java Byte code in the same file was analyzed and used for classification.

To evaluate the classification performance of the proposed model, the test results of only Java Bytecode, only Dalvik OpCode, and the hybrid feature vector in the same classifier were taken comparatively and are given in Table 3. Tests were repeated with 13 different algorithms in different classifier types, and the results were measured according to accuracy, precision, recall, and F-score metrics. The Hybrid feature vector and the Stacked Generalized ensemble model achieved the highest success. In addition, the hybrid feature vector design performed better in all classifiers than the Java and Dalvik-based feature vectors. In addition, with the hybrid feature vector, a 1.5% performance increase was achieved in some classifiers, while it was 1% in others.

As a result, the hybrid feature vector, created by taking the most meaningful features for Java Byte code and Dalvik OpCode, showed higher classification success despite using a similar number of features, thanks to feature selection.

A cross-validation (CV) process was carried out to solve overfitting, bias, and generalization problems regarding the results obtained in this study. Thus, it was intended to reduce bias and demonstrate data independence. As it is known, CV provides the ability to predict performance on data that is not used and seen during training. In addition, another problem that CV solves is to prevent sampling instability. Randomization due to a single selection of training and test data may cause experiments to produce unstable results in the case of a different division. Cross-validation tried to demonstrate overfitting control, bias reduction, and data independence by evaluating all these situations together. Cross-validation graphs obtained with the model proposed are given in Fig. 4.

As can be seen in graph A, high classification performance can be achieved for all classifiers except GNB, and the results are similar in repeated tests. The highest classification success was obtained with the stacking ensemble classifier proposed in this study. Graph B shows the classifier’s results based on Java bytecode. As can be seen in the results, a lower classification performance is obtained than the Dalvik OpCode. As can be seen in the (C) graph, the highest classification success is obtained with the hybrid feature vector, independent of the classifier, since only the significant feature is taken. These results show that the proposed hybrid feature vector and stacking ensemble model design contribute positively to the classification.

The ROC curve for all classifiers is taken and presented in Fig. 5. Accordingly, in parallel with the evaluations made according to the Accuracy value, the highest value was obtained with the hybrid feature vector and stacking classifier, with a rate of 99.6% in the area under the ROC curve (AUC) measurements. The results confirm each other and demonstrate the success of the proposed model.

Detection of malware with proposed stacked ensemble model

Accuracy and AUC measurements do not allow for the evaluation of the results on a class basis. Three different feature vectors were tested with the stacking ensemble classifier, and the confusion matrices are given in Fig. 6. The hybrid feature vector captured the lowest FP and FN values, as shown in Fig. 6C. Only 57 false detections were made in 4,140 test samples, and the classification performance was 98.6%. In addition, it has been observed that malicious applications are classified with higher success than benign applications. While there were only 13 false detections in malicious applications, 44 false detections were made in benevolent applications. For this reason, while the recall value increased to 99.2%, it was above the accuracy value, while the precision value decreased to 97.9%.

The different test results obtained in this section support each other, and it can be seen that the FP and FN values of the hybrid feature vector and stacking ensemble classifier are low. In addition, due to the hybrid feature vector having more features, it can be thought that there will be an increase in training and testing times, which is one of the disadvantageous points of the proposed model. Figure 7 shows the measurement results of training and test times. Measurements were made for all three feature vectors with the stacking ensemble classifier, where the highest performance was achieved in the tests.

There was no significant change in training times for all three feature vectors. The measurement is shown proportionally, not in ms. Thus, the effect of the computer infrastructure change was eliminated. Accordingly, during the training period, the Java byte code requires approximately 20% less processing time than the Dalvik OpCode and Hybrid feature design. The test obtained balanced results regarding processing time in three different feature vectors. The most important reason for this is the use of a similar number of features in all three vectors due to feature selection.

Discussion and limitations

Android app developers use various techniques to prevent malware from being caught by detection tools or reduce their code’s readability. Because with multiple devices such as apktool and dex2jar, there is a chance to open almost all applications with reverse engineering methods and access their source codes. However, it is much more challenging to analyze the states of applications packaged in an obfuscated form and make them easy to read. The limitations of the proposed model in terms of such cases are evaluated in this section.

The obfuscation process refers to how some transformations (renaming, junk code injection, control flow-based obfuscation, etc.) are made by preserving the operation and meaning of the source code. Different codes emerge each time the same source code is repeatedly obfuscated. However, capturing some similarities within these other codes is beneficial in terms of classifying the application. In most cases, changing opcodes or bytecodes is impossible, while manual code analysis is prevented. The transaction code is needed if any action is to be taken on the system.

The model proposed in this study can resist various obfuscation techniques such as name changing, junk code injection, and control flow-based obfuscation. Especially in cases of junk code injection, the operation code search time may be prolonged, and negative situations may arise regarding operation time. This will be seen in the details section of the dataset table. The size of smali file sizes is critical. This problem was solved by terminating the search process once the relevant transaction code was reached. However, this may cause problems for some applications. In addition, the inherent limitations of machine learning models apply to the model proposed in this study. If real-time data is fed to the proposed model, the model cannot continuously train. The most basic way to overcome this will be to periodically include new applications in the dataset and update the proposed model by retraining. Therefore, further research is needed in the future.

In this study, as given in Table 4, more than one malware dataset was brought together heterogeneously to obtain a dataset containing sufficient benign and malicious applications. This use means that training and testing processes are carried out based on the assumption that the applications are equivalent in all datasets. Sensitivity analyses may need to be performed depending on the datasets’ weight. Whichever database is more dominant, the training of the data may be more dependent on it. To overcome this limitation, a dataset with larger samples and sufficient numbers for both benign and malicious applications will be created.

In the evaluation of the computational complexity of the proposed model, the utilisation of Java byte code and Dalvik OpCode features as a hybrid approach is employed. This means that more features are extracted from an APK file. This is a factor that increases the computational complexity of the proposed model. However, in this study, unlike other static analysis methods such as N-gram and API-Call, in the feature extraction phase, the research is analyzed on “smali” files (instead of Java codes) over a limited OpCode list (461 different OpCodes), and only the existence of the relevant OpCodes is investigated. For this reason, it is not necessary to scan the entire file, and the research is terminated when the relevant OpCode usage is encountered. All these operations provide a generalizable method in all applications, and it is possible to complete the research quickly. In addition, since researching the use of API calls over the smali code requires deep analysis (filtering the instructions starting with “invoke”, etc.), the processing time is quite long. When the analysis is done on Java code, the computational cost increases, and more importantly, it is quite problematic to do this research on obfuscated Java code. In the N-gram algorithm, the feature dimensionality tends to increase exponentially rapidly depending on the n value. The necessity of N-gram shifting and the extraction of the related feature vectors is a situation that increases the computational complexity. As a result, the model proposed in this study, which works on an opcode basis, is an approach that increases the computational complexity due to the use of hybrid feature vectors, and it is also in an advantageous position compared to other static analysis methods, such as N-gram and API call analysis. In order to further increase the computational efficiency and scalability of the proposed model, it is possible to use a distributed parallel computing architecture as in the studies (Al-Andoli et al., 2022, 2023, 2024). This situation will be especially taken into consideration in our subsequent studies.

JDroid works based on static analysis. Consequently, it does not require any sandbox, etc., operating environment as in dynamic analysis. This makes JDroid integratable to mobile antivirus programs or leading target enterprise application distribution platforms (Google Play Protect, etc.) or mobile antivirus solutions (Bitdefender Mobile, etc.). For example, during the application installation phase, the Java byte code and Dalvik OpCodes of the application are extracted, and the malicious status of the application can be analyzed by testing it with the previously trained model. In addition, investigating both Java byte code and Dalvik OpCodes on mobile devices and analyzing their usage will bring additional computational cost due to limited resources. In order to reduce this, JDroid works with smali files instead of using the source code, which consists of Java files. Then, the features are reduced before classification with feature selection algorithms on the features. In addition, the computational cost can be reduced by optimizing the feature extraction methods or by methods such as distributed computing (Al-Andoli et al., 2022, 2023, 2024). This will allow JDroid to be used for real-time analysis. The main distribution target of JDroid is enterprise companies that provide application market management and expect high-accuracy detection, and allow cloud-based analysis. However, a version that performs fast detection and works with fewer attributes can also be used on end-user Android devices, thanks to the structures that can be connected to a full-featured system in the cloud for in-depth analysis when needed. In the future, we plan to develop a prototype to evaluate the efficiency and scalability of JDroid on Android platforms.

The use of Grid-SearchCV for hyperparameter selection cannot always guarantee the best parameter selection because the search is performed for a specified range. For this reason, it cannot be guaranteed that the results obtained are always the best value.

Comparison of similar works

The problem to be solved in this study has been studied for many years, and a comprehensive review is given in the literature summary section. High-performance studies are presented in Table 4, compared with the JDroid application, and their advantages and disadvantages compared to other studies are highlighted.

When the studies are evaluated in general, it is understood that Android malware detection is done by feeding different classifiers with code-based features such as OpCode, api calls, permissions, and bytecode based on static analysis. In our study, a similar opcode-based feature extraction process was carried out. However, in all studies in the literature, opcodes were investigated by considering the Dalvik OpCode list. In this study, however, research was conducted on the same code together with the Dalvik OpCode list and also in the Java Bytecode list. Although both are expressed as OpCodes, they are two completely different code structures. Our main purpose in doing this is to obtain more features about the application. As can be seen in the studies in the literature, it is understood that more meaningful features about the application are intended to be obtained by extracting different features, such as api calls, permissions, together with the opcode in many studies. In this study, instead of looking at api calls and permissions, we considered Java byte codes. In addition, in most of the studies, different statistical or LLM-based feature transformation algorithms such as word2vec, codebert, textcnn, graph2vec, n-gram, and skip-gram were used in the feature analysis phase. Because obtaining OpCode sequences is not enough for direct classification, feature vectors need to be obtained from these features. In this study, without using such a method, only the vector conversion process was performed in a way that would be one if the relevant OpCode is used by the application and zero if it is not used. Thus, feature vectors were obtained quite effectively. As a result, for JDroid, there was no need for complex structures such as N-gram, CFG, image conversion, or text-based analysis.

When evaluated in terms of tested datasets, there are studies in the literature that use specific datasets, as well as applications that test with 10 thousand or more application pools. In this study, tests were performed with a very high number of application pools, such as 14 thousand. Therefore, it was shown that the JDroid tool does not have any dependency on any dataset.

In the selection of classifiers, it is understood that in recent years, either CNN-based GNN or DNN architectures have been used. For CNN-based architectures, OpCode sequences need to be converted to images or graphics with different methods, which means extra workload. In this study, a traditional machine learning-based ensemble model is used, and no image transformation, etc., is required. After scanning the OpCode sequences, feature vectors are obtained directly. In terms of performance, the high AUC and Accuracy values obtained by JDroid are better than many studies in the literature.