Cyber threats such as malware have become a significant challenge to digital security in recent years, affecting individuals, organizations, and critical infrastructure worldwide. As these threats evolve and become increasingly sophisticated, traditional signature-based detection methods are becoming less effective [1]. In response, Machine Learning (ML) has emerged as a powerful tool to automate malware detection, offering the ability to classify large volumes of data to identify patterns that might otherwise go unnoticed [2].
However, despite the growing use of machine learning, selecting the most appropriate algorithm for malware classification remains a difficult task due to the complexity of the data and the need for high accuracy and stability of the model in different data splits [3]. To address this challenge, this study conducts a comprehensive evaluation of eight widely used machine learning algorithms for malware detection, including Random Forest, Extreme Gradient Boosting (XGBoost), Support Vector Machine (SVM), Logistic Regression, Naive Bayes, Light Gradient Boosting Machine (LightGBM), Decision Tree, and k-Nearest Neighbors [4]- [6]. These models are assessed using 5-fold stratified cross-validation to ensure robust performance estimation across multiple data splits [7]. The evaluation is based on key performance metrics, including accuracy, precision, recall, F1 score, AUC-ROC, and variance, allowing a detailed comparison of the predictive power and stability of each model [4], [8].
For our experiments, we leverage a refined version of the Microsoft Malware Classification Challenge (BIG 2015) dataset [9], which contains feature-engineered representations of malware binaries [10]. The dataset encapsulates both static features, such as Portable Executable (PE) headers and entropy profiles, and dynamic features, including API call sequences and assembly opcode distributions [11]. These features enable robust classification of malware into distinct families. By analyzing attributes such as section-wise entropy differences ent_q_diffs, importing table dependencies (Imports) and opcoding frequencies, we aim to develop an interpretable machine learning model for malware detection [12]. The dataset's rich feature space not only facilitates accurate classification but also enables anomaly detection, providing insights into evolving malware evasion techniques [13].
The primary objective of this study is to identify the most effective machine learning model for malware classification by balancing predictive accuracy with model stability. While ensemble based methods, like Random Forest and XGBoost, are known for their strong predictive capabilities, their performance must be assessed in comparison to simpler models, like Decision Tree and KNN, which may offer competitive results with lower computational cost. Furthermore, we explore the role of variance-aware evaluation, which is crucial in cybersecurity applications where model reliability across different datasets is essential. Our findings reveal that RF, LightGBM, DT, and KNN achieve near-perfect classification performance with minimal variance, demonstrating their robustness in malware detection tasks. In contrast, XGBoost and SVM exhibit slightly lower accuracy and higher variance, while LR and Naive Bayes perform moderately, struggling to capture complex decision boundaries in the data. These insights provide valuable guidance for researchers and practitioners in cybersecurity, helping them select reliable models for malware classification.
The structure of the paper is as follows. Section II reviews related work in machine learning-based malware detection. Section III briefly introduces the eight ML models utilized in this work. Section IV depicts the modeling procedure and results for the malware detection. Section V discusses the findings. We conclude with Section VI.
The application of machine learning in cybersecurity, particularly for malware detection, has gained significant attention in recent years. Salem et al. [1] provided a comprehensive review of Artificial Intelligence (AI)-driven detection techniques, highlighting the evolution from traditional signaturebased methods to sophisticated machine learning approaches. Similarly, Dasgupta et al. [2] conducted an extensive survey on machine learning applications in cybersecurity, emphasizing the critical role of automated detection systems in addressing the growing complexity of cyber threats.
Several studies have focused on comparative analysis of machine learning algorithms for malware classification. Rahul et al. [4] analyzed various machine learning models for malware detection, demonstrating the effectiveness of ensemble methods in capturing complex malware behavior patterns. Singh and Singh [5] assessed supervised machine learning algorithms using dynamic API calls, providing insights into the importance of feature selection and extraction techniques. Their work highlighted the challenges of balancing accuracy with computational efficiency in real-time detection systems.
The Microsoft Malware Classification Challenge dataset [11] has served as a benchmark for numerous studies in this domain. Aslan and Samet [9] provided a comprehensive review of malware detection approaches, categorizing methods into static, dynamic, and hybrid analysis techniques. Ghouti and Imam [10] specifically focused on malware classification using compact image features and multiclass support vector machines, demonstrating the potential of visual representation techniques. More recently, Connors and Sarkar [12] explored machine learning approaches for detecting malware in PE files, while Lin and Chang [13] addressed the interpretability challenges in ML-based automated malware detection models. These studies collectively underscore the ongoing evolution of machine learning techniques in cybersecurity applications, setting the foundation for our comprehensive comparative analysis of eight state-of-the-art algorithms.
Classification algorithms, a cornerstone of machine learning, have demonstrated exceptional performance across various domains, including cybersecurity applications such as malware detection [3], [14]. Beyond cybersecurity, these algorithms play a crucial role in disease diagnosis [15], where they help detect conditions like cancer [16], [17], diabetes [18], [19], and cardiovascular diseases [20] through medical imaging and clinical data analysis [21]. In finance, classification models are widely used for fraud detection, identifying suspicious transactions and preventing financial crimes [22]. Additionally, they contribute to spam filtering in email systems, sentiment analysis in natural language processing, and customer churn prediction in business analytics [23]. The versatility and effectiveness of classification algorithms make them indispensable across diverse fields where pattern recognition and decision making are essential. This study evaluates eight state of the art classification models, namely, Random Forest (RF), XGBoost, LightGBM, Support Vector Machine (SVM), Logistic Regression, Naive Bayes, Decision Tree, and k-Nearest Neighbors (KNN) to predict malware classes using static and dynamic features. Performance is assessed via five metrics: Accuracy, Precision, Recall, F1-Score, and AUC-ROC, with variance analysis across stratified 5-fold cross-validation to quantify stability.
Given a labeled dataset D = {(x i , y i )} n i=1 where x i represents feature vectors (e.g., API calls, entropy values) and y i ∈ {0, 1} denotes benign/malicious labels, we formalize each model's prediction ŷ for a new sample x.
RF is an ensemble method that aggregates predictions from multiple decision trees, reducing overfitting through majority voting. For malware detection, it has proven to be effective [6].
where f i is the i-th tree's prediction, and N is the total number of trees in Equation (1). RF excels at handling high-dimensional feature spaces (e.g., API call sequences).
XGBoost iteratively improves predictions by combining weak learners (decision trees) with gradient descent optimization.
where γ i is the learning rate. XGBoost's regularization (L1/L2 penalties) mitigates overfitting, critical for imbalanced malware datasets.
LightGBM uses histogram-based splitting for efficiency, optimizing memory usage for large-scale malware data.
where α i weights leaf outputs. Its Gradient-based One-Side Sampling (GOSS) is ideal for sparse features (e.g., n-gram opcodes).
SVM finds the optimal hyperplane to separate malware benign classes via maximum margin optimization.
where ŷ is the predicted class label for a given input x, w is the weight vector learned by the SVM during training, w T denotes the transpose of the weight vector w, ϕ(x) is a non-linear transformation of the input vector x into a higher-dimensional feature space, performed using a kernel function, b is the bias term that shifts the decision boundary, and sign(•) is the sign function, which returns +1 if the argument is positive and -1 if it is negative. The kernel function ϕ(•) enables SVM to handle non-linearly separable data by implicitly mapping inputs into a high dimensional space. A common choice is the Radial Basis Function (RBF) kernel. The effectiveness of SVM is highly dependent on the scaling of features, as it ensures that each feature contributes proportionally to the boundary of final decision.
A linear model for probabilistic classification
where I(•) is the indicator function. It is Interpretable but limited to linear feature relationships.
Naive Bayes is a probabilistic classifier that Leverages Bayes' theorem with feature independence assumptions. ŷ = arg max y P (y)
where ŷ is the predicted class label for a given input instance, y represents a possible class label (e.g., malware or benign), P (y) is the prior probability of class y, x j is the j-th feature of the input vector x, P (x j | y) is the conditional probability (likelihood) of observing feature x j given class y, d is the total number of features in the input, and arg max selects the class label y that maximizes the posterior probability. Naive Bayes is computationally efficient and effective for high-dimensional data. However, its performance can degrade when features are highly correlated, such as in the case of dependent API calls in malware behavior analysis.
A single tree recursively partitions the feature space.
where θ denotes split thresholds. It is prone to overfitting but useful for interpretability.
The k-Nearest Neighbors (KNN) algorithm classifies samples based on majority labels of the k closest training instances.
where N k (x) are the k-nearest neighbors. Sensitive to feature scaling and distance metrics (e.g., Hamming distance for binary features).
Five metrics evaluate model performance, with variance calculated across folds.
1. Accuracy is the proportion of correct predictions over total predictions [24].
where TP (True Positives) represents the number of correctly predicted positive instances; TN (True Negatives) is the number of correctly predicted negative instances; FP (False Positives) is the number of negative instances incorrectly predicted as positive; and FN (False Negatives) is the number of positive instances incorrectly predicted as negative [25]. 2. Precision is the proportion of correctly predicted positive instances among all predicted positives [24]. Precision = TP TP + FP (10) 3. Recall is the proportion of actual positive instances that were correctly identified [24].
4. The F1-Score is the harmonic mean of precision and recall [24].
5. AUC-ROC (Area Under the Receiver Operating Characteristic Curve) plots True Positive Rate (Sensitivity) against False Positive Rate (1-Specificity) across all classification thresholds. The closer the curve approaches the top-left corner (0,1), the better the model's discriminative ability [26].
6. The variance of each performance metric is calculated across cross-validation folds to assess the model's stability. A lower variance indicates a more consistent and reliable model, while a higher variance suggests performance fluctuations across different training sets [27].
TABLE I RESULTS OF MODELS. Model Accuracy Accuracy Variance Precision Precision Variance Recall Recall Variance F1-Score F1-Score Variance AUC-ROC AUC-ROC Variance Random Forest 0.991833 2.57E-06 0.991972 2.39E-06 0.991833 2.57E-06 0.991814 2.57E-06 0.999819 1.90E-08 XGBoost 0.979296 9.65E-06 0.980132 8.70E-06 0.979296 9.65E-06 0.938477 0.000201884 0.999429 2.57E-08 SVM 0.979296 9.65E-06 0.980132 8.70E-06 0.979296 9.65E-06 0.938477 0.000201884 0.999429 2.57E-08 Logistic Regression 0.938575 0.000206963 0.943028 0.000117831 0.938575 0.00020696 0.938477 0.000201884 0.976317 3.25E-05 Naive Bayes 0.938575 0.000206963 0.943028 0.000117831 0.938575 0.00020696 0.938477 0.000201884 0.976317 3.25E-05 LightGBM 0.991833 2.57E-06 0.991972 2.39E-06 0.991833 2.57E-06 0.991814 2.57E-06 0.999819 1.90E-08 Decision Tree 0.991833 2.57E-06 0.991972 2.39E-06 0.991833 2.57E-06 0.991814 2.57E-06 0.999819 1.90E-08 KNN 0.991833 2.57E-06 0.991972 2.39E-06 0.991833 2.57E-06 0.991814 2.57E-06 0.999819 1.90E-08
The experiments were conducted using a refined version of the Microsoft Malware Classification Challenge (BIG 2015) dataset [9]. The dataset contains 21,741 samples with balanced class distribution across nine malware families and benign files. Feature engineering yielded 2,381 static features including PE header information, entropy profiles, import table dependencies, and assembly opcode frequencies. All models were evaluated using stratified 5-fold cross-validation to ensure robust performance estimation across different data splits [28].
Table I presents the comprehensive performance evaluation of eight machine learning models across five key metrics. The results reveal distinct performance tiers among the evaluated algorithms.
Tier 1 -Exceptional Performers: RF, LightGBM, DT, and KNN achieved identical near-perfect performance with accuracy of 0.9918, precision of 0.9920, recall of 0.9918, F1-score of 0.9918, and AUC-ROC of 0.9998. These models demonstrated remarkably low variance (10 -6 to 10 -8 ), indicating exceptional stability across cross-validation folds.
Tier 2 -Strong Performers: XGBoost and SVM achieved accuracy of 0.9793 with identical performance metrics. While still demonstrating strong classification capability, these models showed slightly higher variance (≈ 10 -6 ) compared to Tier 1 performers.
Tier 3 -Moderate Performers: Logistic Regression and Naive Bayes exhibited lower accuracy (0.9386) and significantly higher variance (≈ 10 -4 ), indicating less consistent performance across different data splits.
The variance analysis reveals critical insights into model reliability [29]. The exceptionally low variance (< 10 -6 ) observed in RF, LightGBM, DT, and KNN indicates these models maintain consistent performance regardless of training data variations-a crucial requirement for cybersecurity applications [27].
In contrast, the higher variance exhibited by Logistic Regression and Naive Bayes (≈ 10 -4 ) suggests potential instability when deployed across different malware datasets or network environments. This stability assessment is particularly important in cybersecurity where reliable performance across diverse threat landscapes is essential.
The superior performance of tree-based ensemble methods (Random Forest, LightGBM) and the Decision Tree can be attributed to their ability to capture complex, non-linear feature interactions inherent in malware behavior patterns [30]. These models effectively handle the high-dimensional feature space (2,381 features) without suffering from the curse of dimensionality.
Ensemble Method Advantages: Random Forest's bootstrap aggregating reduces overfitting while maintaining high accuracy. LightGBM's Gradient-based One-Side Sampling (GOSS) efficiently handles sparse features common in malware detection, such as n-gram opcodes and API call sequences.
KNN's Unexpected Success: The exceptional performance of KNN (identical to ensemble methods) suggests that malware and benign samples form distinct, well-separated clusters in the feature space. This clustering behavior indicates that the extracted features effectively capture discriminative patterns between malware families and benign software.
XGBoost Underperformance: Despite its reputation for strong performance, XGBoost's lower F1-score (0.9385) compared to simpler tree-based methods suggests potential overfitting or suboptimal hyperparameter configuration. This highlights the importance of hyperparameter optimization using techniques such as GridSearchCV or Randomized-SearchCV [31].
The moderate performance of Logistic Regression and Naive Bayes stems from their fundamental assumptions that are incompatible with malware detection requirements. Logistic Regression assumes linear decision boundaries, which cannot adequately model the complex, non-linear relationships between malware features and class labels. Similarly, Naive Bayes relies on the feature independence assumption, which is violated in malware analysis where features such as API call sequences and opcode patterns exhibit strong dependencies. However, these models offer computational efficiency and interpretability advantages, making them suitable for resource-constrained environments or scenarios requiring explainable decisions.
Computational Complexity: While ensemble methods provide superior accuracy, they introduce computational overhead. Real-time malware detection systems may require model optimization or hardware acceleration for practical deployment.
Scalability Analysis: KNN's instance-based learning requires storing all training samples, making it memory-intensive and computationally expensive for large-scale deployments. Despite its excellent accuracy, scalability concerns limit its practical applicability.
Model Selection Recommendations: For production environments, Random Forest and LightGBM offer the optimal balance of accuracy, stability, and computational efficiency.
Several challenges were encountered during this study: Feature Engineering Constraints were evident as this study relied primarily on static features extracted from malware samples. Incorporating dynamic behavioral features such as API call sequences and network traffic patterns could further enhance classification performance.
Dataset Generalization presents another concern since while the Microsoft dataset provides a solid foundation, real-world malware detection faces continuously evolving threats. Future work should evaluate model performance on contemporary malware samples and emerging attack vectors.
Class Imbalance Considerations must also be addressed, as although our refined dataset maintains balanced class distribution, real-world scenarios typically exhibit significant class imbalance where benign samples vastly outnumber malware instances. Addressing this through cost-sensitive learning or advanced sampling techniques represents an important future direction.
Hyperparameter Optimization limitations were apparent since the current study employed default hyperparameters for most algorithms. Systematic hyperparameter tuning using GridSearchCV or RandomizedSearchCV could potentially improve performance, particularly for XGBoost and SVM models.
Future investigations should explore several promising directions. Deep learning integration through evaluating Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) for malware detection represents a natural evolution of this work. Dynamic feature incorporation by including behavioral analysis features such as API call sequences and runtime behavior patterns could significantly enhance detection capabilities. Adversarial robustness assessment against malware samples specifically designed to evade detection systems presents a critical research challenge. Additionally, realtime performance optimization through developing lightweight model variants suitable for edge computing and real-time detection systems would address practical deployment requirements in cybersecurity environments.
This study systematically evaluated eight machine learning models for malware detection using stratified 5-fold crossvalidation, assessing both accuracy and stability through variance analysis. The results demonstrate clear performance hierarchies among the evaluated algorithms with significant implications for cybersecurity applications.
Tree-based models, particularly RF, LightGBM, DT, and KNN, achieved exceptional performance with accuracy of 0.9918 and AUC-ROC of 0.9998, while maintaining minimal variance (< 10 -6 ). These models demonstrated remarkable stability across data splits, effectively capturing complex feature interactions in malware behavior patterns. Conversely, Logistic Regression and Naive Bayes underperformed with accuracy of 0.9386 and higher variance (∼ 10 -4 ) due to their linear assumptions, which fail to model complex malware characteristics.
Notably, KNN's exceptional performance suggests that malware and benign samples form distinct clusters in the feature space, validating our feature engineering approach. XGBoost's moderate F1-score (0.9385) indicates potential overfitting, highlighting the importance of hyperparameter optimization for gradient boosting algorithms.
For practical deployment, we recommend Random Forest and LightGBM due to their optimal balance of accuracy, stability, and computational efficiency. Our analysis emphasizes the critical importance of variance-aware evaluation alongside accuracy metrics for cybersecurity applications, ensuring consistent performance across diverse threat environments.
Future research should focus on integrating deep learning approaches, incorporating dynamic behavioral features, and developing adversarial robustness against evolving evasion techniques to advance practical malware detection capabilities.
Copyright (c) IARIA, 2025. ISBN: 978-1-68558-284-5 Courtesy of IARIA Board and IARIA Press. Original source: ThinkMind Digital Library https://www.thinkmind.org IARIA Congress 2025 : The 2025 IARIA Annual Congress on Frontiers in Science, Technology, Services, and Applications