Applying Machine Learning on RSRP-based Features for False Base Station Detection

False base stations -- IMSI catchers, Stingrays -- are devices that impersonate legitimate base stations, as a part of malicious activities like unauthorized surveillance or communication sabotage. Detecting them on the network side using 3GPP standardized measurement reports is a promising technique. While applying predetermined detection rules works well when an attacker operates a false base station with an illegitimate Physical Cell Identifiers (PCI), the detection will produce false negatives when a more resourceful attacker operates the false base station with one of the legitimate PCIs obtained by scanning the neighborhood first. In this paper, we show how Machine Learning (ML) can be applied to alleviate such false negatives. We demonstrate our approach by conducting experiments in a simulation setup using the ns-3 LTE module. We propose three robust ML features (COL, DIST, XY) based on Reference Signal Received Power (RSRP) contained in measurement reports and cell locations. We evaluate four ML models (Regression Clustering, Anomaly Detection Forest, Autoencoder, and RCGAN) and show that several of them have a high precision in detection even when the false base station is using a legitimate PCI. In our experiments with a layout of 12 cells, where one cell acts as a moving false cell, between 75-95\% of the false positions are detected by the best model at a cost of 0.5\% false positives.


Introduction
A false base station [3GPP, 2022a] can be used as an attack tool that poses as a legitimate base station and performs various security and privacy attacks. As such, detection of false base stations is among the hottest topics in mobile networks security. In [Borgaonkar et al., 2017], the authors present their White-Stingray framework to assess the capabilities of different detection apps for the Android platform. Further, in [Park et al., 2019], the authors have presented a survey on app-based, sensor-based, and network-based detection methodologies proposed by research studies and commercial products.
One of the detection techniques proposed by researchers is to use radio frequency properties, with the premise that false base stations use cheap hardware with identifiable hardware imperfections. For instance, FBSleuth [Zhuang et al., 2018] uses variance in the modulation errors, instantaneous frequency, and phases of the electromagnetic signals Applying Machine Learning on RSRP-based Features for False Base Station Detection A PREPRINT to generate fingerprints of false base stations. A statistical approach has been devised by Ali and Fischer [Ali and Fischer, 2019a], who model the radio frequency noise as a univariate Gaussian distribution. The authors in [Ali and Fischer, 2019b] used the idea of cellular network-based synchronization to look for higher carrier frequency offset values. Another detection technique involves using location information. e.g., Huang et al. [Huang and Wang, 2018] propose using the location of mobile phones and legitimate base stations for checking the received signal strength, taking into consideration path loss, shadowing effect, and small-scale fading.
In general, detection techniques that are not network-based are either not reliable due to lacking up to date network topology information; or not scalable due to additional hardware requirements. Hence, in this paper, we focus on the network-based approach that uses 3GPP standardized measurement reports and has been adopted by the 5G security standard [3GPP, 2022b]. Using this approach, the validity of Physical Cell Identifiers (PCIs) present in the measurement reports can be analyzed and is an effective rule-based detection technique [Nakarmi et al., 2021]. This technique works well when the attacker, for stability of attack [Rupprecht et al., 2019], operates the false base station with a PCI that is not a legitimate neighbor. Nevertheless, a more advanced attacker might scan the neighborhood and determine all the legitimate PCIs, and then operate the false base station with one of those legitimate PCIs. In such a case, the rule-based detection will produce false negatives. Therefore, we investigate Machine Learning (ML) techniques that can be applied to provide the necessary discriminatory power to fill the gap where the rule-based detection technique fails. Jian et al. in [Jin et al., 2019] describe using Gradient Boosting Machine algorithm on features like Reference Signal Received Power (RSRP) and information elements in System Information Block. Since this work uses supervised learning and treats false base station detection as a classification problem, it is unlikely to be practical because hardware and software configurations of both the false and legitimate network will vary widely in reality. Furthermore, regulatory issues often make it infeasible to obtain labeled malicious data needed for supervised learning. In [van Thuan et al., 2016], the authors use anomaly detection algorithm named Seasonal Hybrid ESD (S-H-ESD) on features like handovers, location updates, and International Mobile Subscription Identity to International Mobile Subscription Identity mappings. Although this work addresses the issue as an anomaly detection problem, it requires more varied data sources, which limits its practicality. Further, to our knowledge, there is no work yet that aims at detecting a false base station using a legitimate PCI.
To that end, the aim of our work is to apply ML and make the network-based detection technique -using 3GPP standardized measurement reports [3GPP, 2022b, Nakarmi et al., 2021 -robust even when the false base station uses a legitimate PCI. The basic intuition behind our work is that a false base station affects the surrounding radio environment. It is so because one of the techniques that false base stations utilize is to provide a higher-than-normal signal strength to the mobile phones (e.g., by getting physically closer) in order to entice them to connect to it. For training and testing, we used the measurement reports from a simulated LTE network using ns-3 [ns 3, 2022a] which is a free and publicly available discrete-event network simulator ( § 2). We did feature engineering based on the measurement reports and explored three different options, i.e., COL, DST and XY ( § 3). We evaluated four ML models, i.e., Regression Clustering, Anomaly Detection Forest, Autoencoder, and RCGAN ( § 4). The results we obtained indicate that applying ML is an effective technique that can be added to the arsenal of network-based detection of false base stations ( § 5). In our experiments with 12 cells where one of the cells acts as a false base station, the strongest ML model, ADF, detects 75-95% of the false positions at a false positive ratio of 0.5%.
Our main contributions are: • We describe a ns-3 based LTE simulation setup for validating the effectiveness of false base station detection techniques. • We present three different features derived from RSRPs in measurement reports and base station locations. • We provide and evaluate four ML models -without the need for malicious data during training -for detecting false base stations even when the attacker is using one of the legitimate PCIs. One of the models (based on Regression Clustering) is novel. • We provide extensive experiments evaluating the performance of one ML model per serving cell and also include aggregated results from multiple serving cells.

General
The intuition behind our approach to using ML for detecting a false base station is that the presence of the false base station generates deviating data in the measurement reports received by the legitimate base station from mobile phones. Architecture wise, we build on the system components from [Nakarmi et al., 2021]. Our primary data are the measurement reports on the network side that contain PCIs, Reference Signal Received Power (RSRP), and Reference Signal Received Quality (RSRQ) of neighboring cells. RSRP and RSRQ respectively represents strength and quality of received signals from a cell [3GPP, 2022c]. Our auxiliary data is the cell topology that consists of location of cells (cartesian coordinates in ns-3). We simulate the system using ns-3 as described below.

ns-3 Simulation
Using the LTE module in ns-3, we simulated a network with 12 cells in a 3 x 4 layout as shown in Figure 1 (bigger circles represent cells; smaller circles represent mobile phones; and Cyan colored circles represent core network functions). ns-3 uses the Cartesian coordinate system. We setup the cells layout using the Grid Position Allocator with DeltaX=500, DeltaY=500, GridWidth=3, and LayoutType=RowFirst. We set the propagation loss model to Okumura Hata [ns 3, 2022b], number of RA preambles to 10, and SRS periodicity to 320. The mobile phones moved within the specified boundary (-250, 1250), (-250, 1750) according to the RandomWalk2dMobilityModel in which they moved with a specified speed of 5 m/sec and direction chosen at random every 20 seconds. We kept many other configurations in the ns-3's LTE module to their defaults, e.g., A2-A4-RSRQ is the handover algorithm which takes into account the RSRQ of serving cell getting worse (event A2) and that of neighboring cell getting better (event A4); transmission power of cell is 30 dBm.
During generation of training data, we created 200 mobile phones and kept all 12 cells operational with Automatic Neighbor Relation (ANR). All cells were placed at the height of 25 m and the mobile phones were placed at the height randomly chosen between 1.5 to 2 m. The simulation was run for 1000 seconds with random seed set to 1.
For test data generation, ideally, we would have to introduce a new cell with existing PCI. But this was not supported by ns-3 because ns-3 natively avoids PCI collision by automatically assigning PCIs up to 65,535 distinct PCIs. Therefore, we envision a story as follows. We assume that one of the cells has been decommissioned, but the cell topology is not yet updated. The attacker capitalizes on this and starts its own false cell with the PCI of the decommissioned cell and moves along a path. Mind that even though we could not simulate the exact scenario we wanted with PCI collision, the final effect is that the radio conditions (RSRP and RSRQ) will be affected and our main goal still remains valid in the sense that we want to detect changes in the radio conditions even though there is no new PCI.
We generated separate testing data with each cell as decommissioned (which we achieved by setting AdmitHandover-Request to false). The attacker's cell started at the original position of the decommissioned cell; stayed there for 200 seconds (but at a lower height of 2 m assuming that the attacker had its equipments in a van); and after that, moved for 120 seconds along the red lines shown in Figure 1, i.e., (250, -250), (250, 1750), (750, 1750), and (750, -250). We note that the idea behind the movement of the false cell in the simulation is to contribute to the difference in radio environments and is irrelevant otherwise to our detection technique.
We created 100 mobile phones and ran the test simulation with random seed 9. We generated validation data in the same way as the testing data except for using random seed 1 and only using the 200 seconds (without the cells moving).

Training and Testing Data
We preprocessed the data by removing records without any neighbors as these cannot contain false base station information. We separated the rest of the data according to the specified serving cell of each record. This way, for each serving cell, we obtained one training set, and 11 test and validation sets for each simulation where one of the other cells in the layout was moving. Figure 2 shows the dynamics of the training data generated by our ns-3 simulation. For clarity of plots, we only chose the serving cell in the middle, i.e., cell 8. The first column shows two box plots RSRPs and RSRQs of all the cells reported to cell 8 (marked with S in the label and blue color in the plot). In general, it can be seen that the serving cell (8) has higher RSRP/RSRQ followed by the immediate neighboring cells (5,7,9,11). It is also observable that although the serving cell clearly has better RSRQ, the RSRQs of neighboring cells are more erratic compared to the Applying Machine Learning on RSRP-based Features for False Base Station Detection A PREPRINT RSRPs. The second column depicts how the reported RSRPs/RSRQs vary with the distance of mobile phones with each cells. There is a visibly strong negative correlation, which is also intuitive, that the power and quality decrease with increasing distances. However, the RSRQ plot is more erratic. The third column shows plots of RSRP and RSRQ correlation between the serving cell (cell 8) and a cell above it (i.e., cell 5). Strong correlations are not clearly visible, except that, many a times, for the UEs below the cell 8 (orange color), the RSRP/RSRQ of cell 5 is 0. This may be attributed to how Okumura Hata propagation loss model works.
Some dynamics of the testing data are depicted in Figure 3. The first figure shows how many measurement reports become anomalous from the perspective of serving cells 1, 8, and 12, when cell 5 moves. The other two figures show the plots of RSRP and RSRQ correlation between the serving cell 8 and cell 5 above it. It is clearly visible that the dynamics are not as they used to be for the training data ( Figure 2).

Feature Engineering
We extract robust features suitable for applying ML to detect false base stations. From the measurement reports, the RSRQ values are excluded because they are erratic, as shown in Figure 2. We combine the RSRP values of cells with the locations of those cells to build a model of how these values typically vary around each serving cell. When a false base station, disguised with a legitimate PCI, appears in a measurement report, the RSRP values reported for that PCI will typically be different than if the PCI belonged to the legitimate base station with the same PCI.
Since the serving cells are stationary, we use the data collected by each cell to create a model of the reported neighbor values. We have explored three different options for extracting data to train models of neighbor RSRP values around each serving cell. Each option includes the RSRP of the serving cell, the number of neighbors, and the RSRP of the existing neighbors in one of the following ways: COL One fixed column per neighbor PCI with the RSRP-value of that neighbor.
DST Two columns of mixed neighbor values per existing neighbor, one for the distance of the neighbor from the serving cell and one for the RSRP of the neighbor. The maximum number of DST features therefore be twice the number of concurrent neighbors in any report. XY Same as DST but containing two columns modelling the cartesian coordinates of the neighbor relative to the serving cell instead of just the distance. The number of neighbor derived features will here be three times the maximum number of concurrent neighbors in any report.
Note that all of these features will contain missing elements as the number of neighbors in reports varies. The feature sets DST and XY enable mixing different neighbors in the same column by adding additional positional features to distinguish different neighbor PCIs instead of extracting the RSRP value of each neighbor PCI into separate features. The COL feature does not mix values from different neighboring cells and thereby has the greatest capability in modelling exact correlations between the RSRP values of the serving cell and specific neighbors. However, in a setting where the maximum number of concurrent cells is much smaller than the number of different neighbors this may not work as well due to the quickly growing set of missing values in the data matrix as seen in Figure 4.

ML Models for Novelty Detection
Novelty detection usually implies semi-supervised learning, where data from the normal class (i.e., one known label) is used to train a model. This is then used as a reference when determining if new input belongs to the training distribution or if it is previously unseen (e.g., a novelty). In our experiments we have evaluated one new model based on Regression Clustering and three existing novelty detection methods based on Anomaly Detection Forest (ADF) [Sternby et al., 2020], traditional Autoencoders (AE) [Salahuddin et al., 2020], and Regularized Cycle Consistent Generative Adversarial Network (RCGAN) [Yang et al., 2020].
Regression Clustering. In Regression Clustering [Zhang, 2003], a data set is first grouped into different clusters and regression models are trained for each cluster in order to be able to predict values. We use the COL features, and build clusters and regressor models on the training data, such that RSRP of a cell (y) is trained using the RSRPs of other cells around that cell (x). Then, during the test phase, the trained models are used to predict the RSRP of the cell. If the predicted RSRP differs significantly from the observed RSRP, we determine that there is some anomaly. Now, when does such anomaly occur (i.e., when is the predicted RSRP value is not close to the observed RSRP value)? It is when a false base station pollutes the data. In other words, if there is a false base station, the prediction will not be correct; and if a false base station is absent, the prediction will be correct. We can use this intuition and iterate through cells. Then, a cell in whose absence the prediction becomes correct is the false cell. Training logic is shown in Algorithm 1 and test logic is shown in Algorithm 2. We used KMeans for clustering and Random Forest for regression from scikit-learn [scikit learn, 2022].
ADF. ADF is an ensemble of binary decision trees built from subsets of the training set selected at random. The training consists of two phases where the first phase splits a training subsample into smaller regions in input space and the second phase sets feature-wise anomaly borders for each region. The implementation of ADF is adopted from [Sternby et al., 2020] with hyperparameters: subsample size |X| = 512, anomaly margin a = 1, isolation level η = 0.05 and max depth D = 14. Each model contains 150 anomaly detection trees.
AE. Autoencoders is a common choice of algorithm for anomaly detection. An autoencoder is defined by a selfmapping network that is trained by minimizing the reconstruction error on a training set. The basic motivation for using reconstruction error for anomaly detection is that encoding and decoding has been trained with a certain training distribution and thus the reconstruction may not work as well for elements that are dissimilar to the training distribution. The autoencoder architecture tested in our experiments used the same hourglass shaped structure presented in [Salahuddin et al., 2020] with three internal dense layers. The number of nodes in the hidden layers were [0.8F, 0.5F, 0.8F ], where F is the number of input features (this varies with serving cell as well as the type of features used). Training of the autoencoder models was conducted with the Tensorflow framework using the Adam optimizer. The loss function was mean square error (MSE) and the batch size was 50. Each model was trained for 150 epochs with a starting learning rate of 0.002 updated every 50 epochs with a decay of 0.8.
RCGAN. RCGAN is one of several recent GAN-inspired methods for anomaly detection. The main components of the RCGAN is an Encoder E() which is analagous to the first layers of the autoencoder and a Generator G() which maps the encoded input back to input space (from encoded/latent space) and thereby corresponds to the final layers of the autoencoder. So the actual structure is very similar to the autoencoder but with some significant differences in the training process. The implementation of RCGAN in this paper used the bi-direcitonal GAN setup from ALAD [Zenati et al., 2018] but with an added latent penalty distribution as proposed in [Yang et al., 2020]. The Encoder used one hidden layer with 0.7F number of nodes where F is the number of input features and the latent dimension (the dimension of output from the Encoder) was 0.3F . The Generator used two hidden layers with [0.5F, 0.7F ] nodes. The three discriminators D xx , D xz , D zz had one hidden layer each with 0.5F nodes. Training was conducted as in [Yang et al., 2020] apart from not using exponential moving average or spectral normalization. Similar to AE, the Tensorflow framework was used to train the RCGAN models for 150 epochs with a starting learning rate of 0.002 updated every 50 epochs with a decay of 0.8.

Algorithm 1 Regression Clustering: Training
Predict yP CL with x CL residueY ← |y CL − yP CL | end for F LAG P ASS 1 [C, nC] ← residueY > T HRESHOLD end for end for /*From perspective of each cell, flag others*/ for C in CELLS do F LAG C ← F LAG P ASS 1 [C, * ] F LAG P ASS 2 ← T rue if only one 0 in F LAG C , F alse otherwise end for F LAG P ASS 2 contains the flagged/unflagged verdict

Results and Discussion
Training and test data are described in Section 2.3. The number of records for each serving cell are listed in Table 1. To evaluate the success of the novelty detection methods we tuned the threshold on the anomaly scores for each method on the benign samples of the validation set such that only 0.5% their scores were above the threshold. In Table 2, we show the recall of the true positives (ratio) with this threshold.
Note that, as the false cell starts moving in the simulation, it leaves a vacancy such that surrounding serving cells start receiving measurement reports from mobile phones previously reporting to the moved cell and this causes novelties. This phenomenon is due to the limitation in ns-3 that PCI collision was not supported. The effect is clearly visible in Figure 5 that serving cell 3 gets false positives when cell 2 starts moving, but not when cell 1 moves since movement of cell 1 does not cause mobile phones previously reporting to cell 1 to report to cell 3.  Furthermore, Figure 5 indicates that the presence of a false base station may be further corroborated by multiple mobile phones in different locations picking up the false cell and causing a cluster of measurement reports with higher anomaly scores. On the other hand, the false positives in the absence of a false base station seem to occur more isolated as would be the case for a limited number of phones showing up in a new location.

Data-derived Static Novelties
During training of the ML models we obtain an updated list of present neighbors for each serving cell. Neighboring cells never seen for a serving cell during training will imply that these should be marked as suspicious if they start appearing during test (see Table 1). This way, we obtain data-derived static novelties to combine with our ML methods for detection. As these novelties are easy to detect, we note that the attacker would need to avoid inducing such novelties to evade detection.

Serving Cell Results
One novelty detection model for each evaluated algorithm was trained on the training data without anomalies listed in Table 1. In order to evaluate the possibility of detecting false base stations using models built on these features in a realistic setting, we determined a threshold for the anomaly score based on an acceptable rate of false positives on a validation set of benign records. Although we evaluated more combinations of features and ML-algorithms, only a selection including the best performing combinations is listed in Table 2 to avoid clutter. For these algorithms we show the novelty recall results when setting a threshold implying a 0.5% false positive rate on the benign validation set. For each model of serving cells we show recall results both with static anomalies as well as with those excluded (in parenthesis) in Table 2.
The COL features performed well in our simulated setting where the distinctly occurring neighbor cells is fairly few (we had 12 cells in total), but we believe that the XY features could prove more useful in the diversity of real deployment settings (where there are hundreds of cells). Although the XY features imply an extra attribute for every occurring neighbor, the added possibility to distinguish the direction of the neighbor seems to be superior to just using the distance of the DST features. The RCGAN consistently performs better than the AE and we stipulate that this is due to the lack of guarantee that the AE will have high reconstruction errors for the anomalies as previously noted in [Zong et al., 2018]. The success of ADF may be owed to its even more explicit modelling of the benign distribution. Unlike AE and RCGAN which both perform better with the XY features it does not contain any dimensionality reduction (encoding) which may be the explanation as to why the COL features provides the best results for ADF. The Regressor does not perform well in this setting.

Aggregated Serving Cell Results
In terms of being able to detect a false base station, the numbers for recall of true positives (the records where a moved/altered cell occurs as a neighbor) in Table 2 are overly conservative. Depending on its location, a false base station would be picked up by the reports of numerous mobile phones reporting to different serving cells and it is enough that one of these serving cell models detects the malicious activity. This is illustrated in Table 3, where each row contains the PCI of the false cell (instead of the serving cell as in Table 2). For each false cell, we categorize the false cell positions by the number of serving cells it is visible to, and for each such category the percentage of positions of the false cell detected as anomalous by at least one serving cell.

Conclusion and Future Research
We show that applying ML using RSRP-based features for network-based detection of false base station is promising. In our simulations with 12 cells where each cell alternated as a moving false cell, the best ML model obtained between 75-95% recall of false cell's positions. Furthermore, analysis of the obtained anomaly scores indicates that a true false base station would cause a cluster of anomalies as multiple phones pick up the false cell in the measurements sent to different serving cells. This could make it feasible to distinguish such anomalies from false positives, for instance caused by an unusual phone location. We intend to continue the research, and validate our models and features on real operator data.