There has been an increased interest in Unmanned Aerial Vehicles (UAVs) for civil applications over the last decade [1]. Several tasks of UAVs, including navigation, positioning, and return-to-home, are dependent on Global Positioning System (GPS) devices. The civilian GPS signals are not encrypted, and as a result, can be easily spoofed which is a problem for UAV's safe flight operations [1].
For this purpose, a number of techniques have been proposed to detect and mitigate such attacks. Some of these techniques are hardware-based [2 -4]. For example, in [2], the authors proposed a method using the off-the-shelf global navigation satellite system (GNSS) antennas to detect GPS spoofing attacks. This method is able to detect malicious signals from diverse locations. In [3], the authors presented a GPS spoofing detection algorithm using the Doppler frequency difference of arrival in a dual-antenna receiver, that exploits the regularity between the signal features and the navigational information. In [4], the authors proposed a detection technique and removal method which relies on the existing Cooperative Adaptive Cruise Control system to provide intervehicle ranging and data sharing.
Other studies proposed GPS spoofing attack detection techniques using artificial intelligence (AI) methods, including machine learning (ML) [5][6][7][8]. For instance, in [5], the authors presented an ML technique based on Support Vector Machines in detection of malicious signals on UAVs. In [6], the authors proposed an anomaly detection technique based on K-nearest Neighbors (KNN) to detect GPS spoofing on UAVs. In [7], the authors provided a comparison performance of tree-based Supervised machine learning models for detecting GPS spoofing attacks on UAVs. In [8], the authors compared the performance of two ML models, namely KNN and Naive Bayesian (NB) classifiers, to detect attacks on UAVs.
All these hardware and AI-based techniques show good accuracy. However, a performance comparison should be made in terms of different performance metrics, such as probabilities of misdetection and false alarm, which are missing from several studies. In addition, most of the works did not employ hyperparameter tuning techniques to improve the performance of the algorithms. This study fills the existing gap by enhancing the performance of ML models using ensemble-based ML models with a particular hyperparameter technique. These ensemble-learning techniques are bagging, stacking, and boosting. Each of these models incorporates the decisions of several machine learning models to increase the detection performance. To identify the correlated features in the dataset, we choose Pearson's Correlation Coefficient. In addition, to optimize the results, we use the hyperparameter tuning technique, Grid search, to find the best hyperparameters for each model. These ensemble models are compared based on the evaluation metrics: accuracy, probability of detection, probability of misdetection, probability of false alarm, memory size, processing time, and prediction time per sample.
The following summarizes the contribution of this study:
• Identification of the most important features by using two feature selection methods.
• Comparative analysis of ensemble learning methods using seven evaluation metrics, namely accuracy, probability of detection, probability of misdetection, probability of false alarm, memory size, processing time, and pre-diction time per sample.
The remainder of this paper is organized as follows: Section II outlines the materials and the methodology used in this study. The results are described in Section III. Finally, the conclusion and future work are drawn in Section IV.
This section describes the dataset collection, features, and data preprocessing techniques, including data cleaning and normalization. We also briefly describe the feature selection methods and classification models.
To collect GPS signals, a software-defined radio receiver was employed to collect GPS signals at different speeds, positions, and altitudes [7]. Three types of spoofing attacks were simulated, namely simplistic, intermediate, and sophisticated. In simplistic spoofing attacks, the broadcasted signals are not synchronized with the authentic GPS signals; therefore, these attacks can be easily detected. In intermediate spoofing, attack estimate the target receiver antenna location and velocity before broadcasting the fake GPS signals. Sophisticated spoofing attacks are the most advanced type of GPS spoofing attacks, in which multiple synchronized phaselocked intermediate spoofers are used to avoid the detection by the target receiver [7].
The dataset for training and testing is built by identifying 13 features. These features, along with their equations, and descriptions are discussed in detail in Table I.
In this work, we used a dataset consisting of 10,056 samples. This dataset includes 4,764 attack samples and 5,382 authentic samples. Data corresponding to GPS spoofing attacks are encoded as 1, and the remaining are encoded as 0. The final step is to normalize data into the compatible form for modeling. Several techniques exist to normalize raw data, including Mean and Standard Deviation Based Normalization Methods, Decimal Scaling Normalization, and Median and Median Absolute Deviation Normalization. In this study, we used the min-max scalar to normalize data. The min-max normalization inserts the data into a common scale, which increases the performance of the classifiers [8,9]. Each feature's value is scaled to a number between 0 and 1. The min-max scalar is calculated as [9]:
where x is the initial value, Min(x) and Max(x) are the minimum and maximum values of the feature vector.
Feature selection is a crucial step to identify the significant features to obtain high-performance results. Correlated or irrelevant features in the dataset can affect the performance of the classification models. In this study, we employed a correlation technique, Pearson Correlation, which is a filterbased method that measures the correlation between two variables and ascertain the strength of the linear connection ranging from -1 to 1. When the result is close to 1 or -1, the characteristics have a strong connection, either positive or negative. A positive correlation coefficient indicates positive linear correlations, while a negative correlation coefficient indicates negative linear correlations [5]. The correlation coefficient is given by:
where n is a sample size, x and y are two variables, x and ȳ are the means of the two variables, and x i and y i are the individual sample points indexed with i. In general, a weak correlation corresponds to a value of R p less than or equal to 0.39 and a moderate correlation is defined as an R p value between 0.40 and 0.89. In this study, we considered a correlation coefficient threshold of 0.88 and -0.88 for highly correlated features.
In addition to feature selection methods, a static relationship is needed for a proper model since ML algorithms cannot handle non-stationary data modification. We search for characteristics that follow a non-stationary distribution and use analysis to transform the raw data to stationary data, includes determining the consecutive deviations between samples. This method is equated below [7]:
where R is the rate of change and n i+1 -n i is the distance between two instances, which is equal to 1.
Traditional machine learning techniques may not always produce high performance results, particularly when the data is composite or unbalanced [10]. One possible way is to use an ensemble learning model, which is a set of training models that work together to enhance the accuracy of a single model's predictions [11 -13]. In general, these techniques can be classified into three categories, namely bagging, stacking, and boosting models. The stacking technique employs several classification algorithms by using their outputs as inputs of a final estimator to obtain high accuracy. This technique is executed at two levels. In first level, the algorithm mainly trains various models including their prediction results, while in second level the model evaluates the best estimate of previous level predictions [11]. In the bagging model, multiple evaluations are calculated, and the average of them is used to make the prediction. This model has several ML estimators that use decision trees and individual learners to make a prediction [12]. One advantage of such a method is to reduce the base algorithm's choice and increase the accuracy of the model. In the boosting model, weak learners are converted into strong ones by collecting algorithms. At each iteration, the learning of this model is done according to the training weights, which is updated based on the previous iteration's performance. To improve the classification results, the boosting model uses a technique known as decision trees that combines several models with varying levels of performance [13].
In this study, we combine five different traditional classification methods, namely KNN, NB, decision tree (DT), random forest, and logistic regression, in the stacking technique. In the bagging method, to achieve the best results, we apply DT classification technique. Finally, we implement the Gradient Tree Boost ensemble model in the boosting model.
To train and test the proposed algorithms, we employ a 10fold cross-validation method. With this technique, the models are trained with 80% and tested with 20% of data in the given dataset. A comparison between these algorithms is carried out based on several evaluation metrics: the accuracy, probability of detection P d , the probability of misdetection P md , and the probability of false alarm P f a . We calculate these metrics by applying the following equations:
where T P is the number of correct predicted malicious attacks, T N is the number of correct predicted normal attacks, F P is the number of incorrect predicted malicious attacks, and F N is the number of incorrect normal attacks.
Due to the size, weight, and power (SWaP) constraints, subsidiary three metrics are employed: memory size of each model, the processing time to perform all steps, and the average prediction time per instance. The memory size observes the consumption of the memory for each model separately, as well as line-by-line investigation of memory use. The processing time refers to the prerequisite time to train and test the models, and it highly depends on the used ML classifier. The average prediction time for each instance is prerequisite to predict whether the current sample is authentic.
The results of the investigation are presented in Figs. 1 to 2 and Tables II to IV. After implementing the models and metrics, we applied the Grid search as a hyperparameter tuning technique to obtain the best results for each model. These hyperparameters are described in Table II for each of the three ensemble models. Fig. 1. illustrates the results of Pearson's correlation coefficient for each pair of the features. As one can observe, few features are highly correlated. We selected the threshold 0.9 to identify highly correlated features. As a result of this method, RX and DO, are considered highly correlated with TOW, and TCD, respectively. Due to their lower importance than that of DO and TOW, RX and TCD were discarded from the list of features. Finally, eleven features, namely DO, TOW, PD, CP, CN0, PRN, PC, PQP, PIP, LC, and EC are considered relevant and uncorrelated features for classifying GPS spoofing attacks on UAVs. Moreover, subsidiary data preprocessing step is The results of the three selected algorithms are shown in Fig. 2 in terms of accuracy, probability of detection, probability of misdetection, and probability of false alarm. Fig. 2a shows the accuracy of the bagging, stacking, and boosting models. As one can see, the stacking model has the highest accuracy (95.43%), followed by the bagging (95.28%), then the boosting model (94.61%). Therefore, these results show that the stacking model provides the best accuracy for detecting GPS spoofing attacks. However, the accuracy is not sufficient to compare the efficiency of ML models in detecting GPS spoofing attacks. The number of falsely detected alarms, and misdetected samples can degrade the performance of ML models.
Fig. 2b shows the results of the selected models in terms of the probability of detection. As one can see, the stacking classifier has the highest detection probability of 99.56%, the bagging classifier has a detection probability of 99.24%, and the boosting model has a detection probability of 96.55%, which is considered the lowest result compared to the two other ensemble models. Fig. 2c shows the probability of misdetection of the selected ensemble models. As one can see, the stacking classifier has a probability of misdetection of 0.36%, the bagging model shows a probability of misdetection of 0.64%, and the boosting model has a probability of misdetection of 2.95%. Consequently, the stacking model has the lowest probability of misdetection, whereas the model has the highest and worse probability of misdetection. Fig. 2d illustrates the results of the probability of false alarm of the selected models. As one can see, the stacking classifier has the best result in terms of the probability of false alarm (0.43%), followed by the bagging model (1.07%) and then boosting classifier (5.08%). The summary of the performance results of the proposed models in terms of the four evaluation metrics are given in Table III. Table IV gives the results of the memory size of each model, the processing time, and average prediction time of each sample for each model. As one can see in this table, the stacking classifier presents the worst outcomes in terms of processing time and average prediction time compared with the other ensemble techniques. In addition, the stacking classifier employs the biggest proportion of memory size (191.3 megabytes), followed by the bagging model (190.4 megabytes), then the boosting method (190.5 megabytes). The stacking model has a processing time of 13.06 seconds, the bagging model has 0.74 seconds, and boosting model has 1.5 seconds. As a result, the bagging classifier provides the best results in terms of processing time, followed by bagging and stacking models. Finally, the stacking classifier has the worst average prediction time of 0.24 seconds per instance, followed by the bagging with a prediction time of 0.02 seconds and the boosting model with 0.01 seconds.
In summary:
• The correlated features, namely RX and TCD are discarded from the corresponding dataset.
• The Grid search hyperparameter tuning method is used to find the best hyperparameters for each model.
• Among the ensemble models, the stacking model gives the best results in terms of probabilities of detection, misdetection, and false alarm. Whereas it has the highest processing time and average prediction time for each instance.
• The boosting classifier provides the lowest results compared to the bagging and stacking models.
• The bagging model has good detection, misdetection, and false alarm probabilities that are slightly below those of the stacking algorithm. However, its memory size, processing time and average prediction time per sample are 1, 18, and 12 times smaller than those of the stacking algorithm.
IV. CONCLUSION GPS spoofing attacks are among the most important threats that target UAVs. In this paper, we presented a performance comparison of the bagging, stacking, and boosting algorithms in detecting GPS spoofing attacks in terms of accuracy, probability of detection, probability of misdetection, probability of false alarm, memory size, processing time, and prediction time per sample. First, we identified the most relevant and uncorrelated features using Pearson's Correlation feature selection technique. The results show that RX and TCD are highly correlated and have lower importance scores, thus they are discarded from the corresponding dataset. In addition, we implemented the Grid search technique for hyperparameter tuning to determine the optimal hyperparameters for each model. The simulation results show that the stacking-based ensemble learning model has the best results compared to the bagging and boosting classifiers. In contrast, the boosting classifier provides the lowest results among all ensemble models. For future work, we plan to investigate the performance of deep learning models in detecting GPS spoofing attacks on UAVs.
Authorized licensed use limited to: UNIVERSITY OF NORTH DAKOTA LIBRARIES. Downloaded on September 05,2022 at 20:53:51 UTC from IEEE Xplore. Restrictions apply.