Privacy-Preserving Credit Card Fraud Detection using Homomorphic Encryption

Credit card fraud is a problem continuously faced by financial institutions and their customers, which is mitigated by fraud detection systems. However, these systems require the use of sensitive customer transaction data, which introduces both a lack of privacy for the customer and a data breach vulnerability to the card provider. This paper proposes a system for private fraud detection on encrypted transactions using homomorphic encryption. Two models, XGBoost and a feedforward classifier neural network, are trained as fraud detectors on plaintext data. They are then converted to models which use homomorphic encryption for private inference. Latency, storage, and detection results are discussed, along with use cases and feasibility of deployment. The XGBoost model has better performance, with an encrypted inference as low as 6ms, compared to 296ms for the neural network. However, the neural network implementation may still be preferred, as it is simpler to deploy securely. A codebase for the system is also provided, for simulation and further development.


Motivation
Homomorphic Encryption (HE) is a technology which allows operations to be performed on encrypted data without needing to decrypt it. This concept has long been referred to as the "holy grail" of cryptography [1], but has historically struggled with speed and accessibility issues. Recent improvements in these areas has led to HE being proposed as a practical solution to real-world problems, such as in the healthcare [2] or finance [3] domains. I initially read about HE while working at encryption services startup Evervault, and decided to search for some existing problems which it could potentially be a solution for. The intention was to become familiar with the state-of-the-art practices in HE, while solving an important problem. Given the sensitive nature of credit card (CC) data, it stood out as a domain which could benefit from private computation.
CC fraud is the use of someone's payment card fraudulently, typically involving the theft of funds via transfer, or the purchasing of goods. The European Central Bank found that in 2019 alone, the value of fraudulent transactions amounted to over €1.8 billion [4]. CC fraud is an important issue for banks, as they must make large amounts of payouts to customers each year. It is also a great frustration for customers when one's card has had funds stolen from it, or is frozen due to fraudulent activities. This makes CC fraud detection a vital issue for banks and customers, and a difficult problem to be solved in the financial world. Existing fraud detectors have found success by using rule-based methods or Machine Learning (ML) methods on data surrounding the transactions. This data may include information such as the location at which the transaction was made, or the type of goods or services being purchased.
Although the detectors have resulted in a relative reduction in CC fraud, it has led to the need for users to share additional data, in plaintext form. The upward trend of data breaches motivates a reduction in the amount of sensitive data gathered from users. Additionally, some users with high demands for privacy may not want to share this data with their bank, let alone with potential hackers. This calls for a solution in which user data can be kept secure, while also allowing it to be used in fraud detectors. This paper explores using HE to build a CC fraud detector which meets this need for privacy. As the main focus is on the privacy preservation, the aim is not to make reach state-of-the-art performance at the fraud detection task, but to make a model which performs well as a fraud detector in the HE domain.
ML models in various domains process vast amounts of sensitive user data each day. This provides an additional motivation to produce research which converts standard ML models to models which use HE, as parts of this research can then be used in domains unrelated to CC fraud. Hence, the methods implemented in this paper will be ML based.
An additional motivation is to produce an accessible, open source codebase for this project. It is currently quite difficult to find tutorials for ML inference using HE, or information on how to adapt on previous work. Novel technologies such as HE are accelerated by community contributions. It is hoped that the codebase released with this paper will be valuable for further research.

Approach
The approach of the paper is to create HE implementations of CC fraud detectors and make latency and storage comparisons with their original plaintext versions. The produced HE models will also be compared, to see which would be most practical to use in the real-world. Comparisons made include transaction encryption latency, model size, transaction size, and inference latency.
The choice of which methods to implement in HE is based on how effective their plaintext versions are at detecting fraud. The number of models chosen for comparison, two in this case, is limited by the amount of time allocated to the project.

Objectives
The objective of this research is to produce a model which, on the selected datasets, achieves the following standards: 1. Transaction data is encrypted at 128 bits of security, and operated on using Homomorphic Encryption.
2. The model achieves high performance detecting the fraudulant transactions in the dataset.
3. The HE implementation has the same detection performance as the plaintext model. 4. The inference time of the model is reasonable for use in a service requiring real-time performance.
5. The time taken to encrypt the model is reasonable for use in a service which may have many users.
6. The size of the encrypted model is reasonable for use in a service which may have many users.
If possible, multiple models which achieve this standard will be produced. Having models with different characteristics, pros, and cons can open up opportunities for different use cases.

Report Structure
Section 2 outlines the necessary background information on the topics used, and previous relevant work. Section 3 details the design of the proposed privacy-preserving CC fraud detection system, with section 4 describing the technical details of the implementation. In section 5 the results of CC fraud detection performance are shown, along with latency and storage size benchmarks. Section 6.3 discusses the performance of the models, limitations of the system, and proposals of further work on the system. The conclusion of the paper is stated in section 7.

Credit Card Fraud Detection
Credit card (CC) fraud is an umbrella term used for any time a payment card (debit or credit) is used against the will of the owner. CC fraud consists of card-present fraud, which is the use of a physical card, and card-not-present fraud, which involves the use of stolen card details for online transactions [4]. CC fraud detection is the problem of predicting whether a given transaction is fraudulent or legitimate. Designing an effective detection system has been a consistent subject of research for the past three decades [5,6,7,8].

Transaction Data
Access to previous transaction data allows for detection models to be based on patterns found in existing data, and then tested on that data. Datasets typically contain a list of transactions in chronological order. Each transaction contains a number of transaction features, such as transaction amount, or product type purchased. They are also labelled as fraudulent or legitimate transactions, represented by 1 or 0 respectively.
Datasets which are made publicly available must undergo a process of anonymization to ensure that information about who made the transactions cannot be inferred. A typical method for anonymization is Principal Component Analysis (PCA) [9]. Some prior works use anonymized, publicly available datasets [10], and others use private commercial datasets [11]. A simulator for CC fraud data has also recently been implemented [12], which holds potential to enable further testing of models on generated data.
A small percentage of CC transactions are fraudulent, for instance, only 0.024% of SEPA transactions [4]. This skew is also typically present in datasets used for fraud detection, in which often less than 5% of transactions are fraudulent. Managing this unbalanced data is a challenging problem which must be dealt with by practitioners [13].

Machine Learning Techniques
In this section, Machine Learning (ML) methods and techniques used in previous works on CC fraud detection are discussed, with two families of ML models being the focus: Decision Trees and Neural Networks. Previous work on CC fraud detetors have also used classical ML algorithms such as Support Vector Machine (SVM) [14] and logistic regression [15]. Some ML terminology is required, with definitions provided for Loss Function 1, Overfitting 2, and Feature Engineering 3.
Definition 1 (Loss Function). When training ML models, the goal of the training process is to decrease the value of a function which usually represents the difference between the current outputs of the model and the intended outputs. This function is known as the loss function.
Definition 2 (Overfitting). If an ML model is trained on existing data, this data is known as the training set. Overfitting is a term used for when a model learns to perform correctly on the training set, but performs significantly worse when introduced to new data.
Definition 3 (Feature Engineering). Feature engineering is the process of adding extra features to a dataset with the intention of improving model training.

Decision Trees
A Classification and Regression Tree [16] is a tree structure in which a comparison is made at each node, and the leaf node reached contains a result. These are often referred to as CARTs, regression trees or decision trees. For simplicity, in this paper are referred to as trees. In 2007, Shen et al. attempted to use a single tree to detect CC fraud [17], but it was outperformed by other models.
An ensemble of trees is when the result of multiple trees is combined to produce the output. Random Forest, an ensemble method, was proposed as a method for CC fraud detection, but was unable to adequately handle the unbalanced data [18]. Gradient boosting [19] is a method which has been used to improve the training of the trees. Models using gradient boosting have found success for CC fraud detection, such as XGBoost [20] and LightGBM [10]. An example of how a tree ensemble model could be used to predict a probability for if a transaction is fraudulent is shown in Figure  2.1.

Neural Networks
An Artificial Neural Network (NN) is an ML model which processes data using a combination of functions called neurons, which calculate a learned linear function on its input. The simplest form of NN is called a feedforward NN. In a feedforward NN, numerical inputs are passed through a series of layers to produce an output. The depth of a feedforward NN corresponds to the number of layers it has. Each layer contains a set of neurons. The output of each layer passes through an activation function before being passed to the next layer. Activation functions are used to introduce non-linearity to the model. The most popular activation function is the Rectified Linear Unit (ReLU) [21].
A Classifier NN is one which produces a single output which is either a binary value or a continuous probability. This architecture was used in the first implementation of an NN for CC fraud detection [22]. An example of how a feedforward classifier NN could be used to predict a probability for if a transaction is fraudulent is shown in Figure  2.2. Other types of NNs previously used for CC fraud detection include Autoencoder [23,24], Convolutional Neural Network (CNN) [11] and Long Short Term Memory [25].

Homomorphic Encryption
Encryption is the process of converting data into another form such that the original value can only be revealed by the holder of a secret key. The encrypted form is called a ciphertext and the plain form is called a plaintext. Homomorphic Encryption (HE) is a way of encrypting data such that some or all standard operations (addition, multiplication, division etc.) can be performed using ciphertexts, with the decrypted result showing the correct value. An example of multiplication using HE is shown in Figure 2.3, as a demonstration of this concept.  [26], with Fully Homomorphic Encryption (FHE) being the end goal. FHE schemes are those which allow for any operation to be evaluated any number of times. The FHE problem was cracked in 2009 by Gentry [27], but choosing an FHE scheme is not always optimal for performance.
Partially or leveled HE schemes are often chosen, which have limitations on, respectively, the type of operations allowed or the number of operations allowed, but may provide lower latency. A comprehensive survey by Acar et al. contains information on the history and specifics of various schemes [28] and also found that the dominant topics of research for HE are in the health and finance domains.
Definitions for Symmetric Encryption 4 and Asymmetric Encryption 5 are given to aid in understanding the uses of the HE schemes discussed.
Definition 4 (Symmetric Encryption). In a symmetric encryption scheme a single secret key is used to encrypt and decrypt.
Definition 5 (Asymmetric Encryption). In an asymmetric encryption scheme a public key and a secret key are generated. The public key can be safely sent to anyone, and anyone can use it to encrypt values. However, only the holder of the secret key can decrypt.

Decision Tree Inference using Homomorphic Encryption
The first implementation of a Decision Tree using HE was by Bost et al. in 2014 [29]. They used the Quadratic Residuosity scheme [30] and a leveled HE scheme implemented by HElib [31], with optional ensembling using the Paillier scheme [32]. Inference time of approximately 1 second, depending on security parameters and tree depth.
In 2015, Khedr et al. demonstrated an encrypted tree ensemble model [33] which used a Ring Learning With Errors variant of the GSW scheme [34]. They achieved a much better inference time of 0.027s.

Neural Network Inference using Homomorphic Encryption
Significantly more research has been done on HE implementations of NNs than Decision Trees. This is likely due to the trend of deep learning, and the applicability of NNs to problems in various domains. CryptoNets [37] in 2016 was the first implementation, which used the YASHE scheme [38] and a square activation function. CryptoNets performed well on its classification task, but still had a high latency of 250 seconds.
Latency improvements have since been made by using a GPU in HCNN [39], and by more efficient encoding in LoLa [40], which both used the BFV scheme [41]. nGraph-HE2 [42] then achieved state-of-the-art throughput using the CKKS scheme [43], which is a fast HE scheme for approximate computation.
Clet et al. made a comparison on the BFV, CKKS and TFHE [44] schemes and found that the CKKS scheme was the fastest for inference using a small architecture [45]. Some works have used polynomial approximations of activation functions such as ReLU [46] for improved performance in exchange for added latency.

Different methods for private computation
Alternative methods to HE for private computation have also been proposed. A Trusted Execution Environment (TEE) [47] is a compute environment in which code can be run in a confidential manner and the code running can be remotely attested. Secure Multi-Party Computation (MPC) [48] is a method for sharing the compute of a function between parties, without any single party knowing the full information.

Anonymization
If data can be anonymized in such a way that detection models using the anonymized data reach the same level of performance of models which use private data, then simply anonymizing the data would result in privacy-preserving detection.

Using Non-Sensitive Data
Similarly to anonymization, if adequate performance can be reached using data which is deemed non-sensitive, then there is no need for privacy-preserving methods.

Related Work
HE for CC fraud detection was explored in two previous works. Canillas

Privacy-Preserving Credit Card Fraud Detection
This section proposes a system for private CC fraud detection using HE, including use cases, the methods implemented, and the system architecture of each method.

Use Cases
It is important to understand the use cases of the proposed system to ensure that it has practical applications. Each use case has (1) Host, which is the party hosting the HE CC fraud detetor, and (2) Client, which is the party sending an encrypted transaction to Host in order to find out if the transaction(s) is fraudulent or not.

Customer & Bank
In this use case, a bank is Host. Customers of the bank are Clients. A customer attempts to make a CC transaction. Before the transaction goes through, its features are encrypted and sent to Host for private fraud detection. If the transaction is predicted as legitimate, the transaction goes through. If not, the transaction is blocked.
Every customer of the bank could potentially be a Client, or perhaps only a subset of customers who are willing to accept extra transaction latency or payment costs, in exchange for a higher guarantee of privacy. A limitation on this use case is discussed in Section 6.2.1.

Bank & Third-Party
In this use case, a bank is Client. Host is any third-party which has created a (plaintext) CC fraud detection model which has better detection performance than the model currently available to the bank. The bank does not trust the third-party with plaintext customer data, so the third-party hosts a HE implementation of their model.
The bank can then send encrypted transaction data to the third-party to make use of the superior fraud detector. The third-party could be another bank, or a company set up to provide this service. Note that in this use case, the Client bank could theoretically store their customers transaction data in plaintext, which would not protect from data breaches.

Methods Implemented
A Decision Tree model (XGBoost) and a NN model (Feedforward Classifier) were both selected for use in the system. Selecting from two different families of ML algorithms can demonstrate the usability of each and can reveal features of each which may be more suited to certain use cases. If both are performant, it also opens up further opportunities for a combination via ensemble.

HE XGBoost
XGBoost was chosen due to the high performance of its plaintext implementation on CC fraud detection tasks. In addition, the open-source implementation of PPXGB 1 meant there would be little experimentation required for successfully converting a model to HE.
An example of what the XGBoost model previously shown in Figure 2.1 would look like after conversion to HE using PPXGB is shown in Figure 3.1.

HE Feedforward Classifer
A Feedforward Classifier NN model was chosen due to its previous success for CC fraud detection, and its simple architecture. The HE implementation used is the CKKS scheme, as it was found to be have the fastest inference time for a similar architecture [45], and open-source implementations are supported in various programming languages. The CKKS scheme works well for NNs, as it has fast latency by using approximate arithmetic. As Feedforward Classifer NNs produce a fuzzy output, using approximate arithmetic can result in a slightly different output, but show the same prediction after conversion to binary.
An example of a CKKS implementation of the NN previously shown in Figure 2.2 is shown in Figure 3.2.

System Architecture
The sequence of events for a CC fraud detection system using the HE NN is shown Protocol 1, with a visualisation shown in Figure 3

Datasets Chosen
The datasets chosen for model training are the Vesta and ULB datasets. The reasons for choosing these is that they are the two largest public datasets available, and have both been widely used in previous work since their respective releases.
No extra feature engineering was done on the datasets, as the focus of this paper is not on analysing the datasets. Using the datasets in their existing form with little pre-processing demonstrates how the models can be implemented directly on existing data.
Both of the datasets used have undergone some level of anonymization. It may seem impractical to use them in a privacy-preserving way, as they don't contain any private data. However, they still work as a proof-of-concept for privacy-preserving inference.
The reason for this is that anonymizing the data can only result in a level of entropy which is less than or equal to the amount of entropy in the original data. Hence, if the level of entropy in the anonymized data can be used to accurately predict if transactions are fraudulent, it follows that there would be enough entropy present in the original data to reach at least the same level of performance.

"Credit Card Fraud Detection" (ULB)
The ULB dataset 2 contains 284,807 transactions, 492 of which are labelled as fraudulent (0.172%). Each transaction has 28 numerical features V1-V28, along with the relative time since the first transaction and a label of whether it was fraudulent or legitimate. PCA has been applied to the dataset before release for anonymization, with V1-V28 being the first 28 principal components.

Python
Python 3 was chosen as the programming language for the implementation, as this is the language used in PPXGB, and the CKKS scheme also has Python implementations.

Weights & Biases
Weights & Biases (W&B) [51] is a tool which aids in training ML models, via effective logging and visualisation of training runs. Including W&B in the training pipeline allows for logging of metrics to ensure models are training properly and to keep track of the performance of each model.
W&B also provides a free service for doing Sweeps of model hyperparameters. These are the parameters associated with the structure of a model such as the size of a model layer, or the number of decision trees in a model. A Sweep is a sequence of training runs, each using different hyperparameters. The results of each training run are logged to W&B for comparison. The choice of which hyperparameters used in each training run can be either a random choice, or using a bayesian method to try converge toward better performance on a chosen metric.

Jupyter Notebook
Jupyter Notebooks [52] are interactive Python notebooks which allow researchers to display their results in an interactive manner, by running inline code alongside standard markdown text. They are used in the simulation to aid in understanding which code would be run by each of the parties involved.

Model Training Packages
The XGBoost Python Package [53] is used for training the XGBoost models. PyTorch [54] was found to be suitable for converting models to HE, as it uses an object oriented approach.

TenSEAL
TenSEAL [55] is a library which contains Python bindings for the C++ CKKS scheme implementation in Microsoft SEAL [56]. TenSEAL have also provided tutorials 2 on converting plaintext NNs to HE models, which was helpful for writing this implementation.

Model Training
Each time a dataset is used for training, the first 65% of transactions are used as the training set, the next 15% are used as the validation set, and the final 20% are used as the test set. The model is first trained on the training set with various hyperparameters. The choice of hyperparameters is then based on performance on the validation set. Final results are then taken on the test set.
The purpose of splitting the dataset in this manner is to ensure the resulting model performs well across different distributions, i.e. to prevent overfitting. If the model performs well on the validation and test sets, we can be confident that it would perform well on further new transactions.
Two scripts train_xgb.py and train_nn.py are provided to allow for simple training of each model. The dataset flag can be used when running the scripts to choose the dataset to train on. The wandb flag can be used to log the metrics from the training run to W&B. Logging the training run is important to ensure that the loss function is converging towards a minimum on the training set. If it is not converging, there is likely a problem with either the loss function, or perhaps the learning rate is too high. The full output of an XGBoost training run on the ULB Dataset is shown in Appendix A1.3.

Undersampling
As the datasets are highly skewed towards legitimate transactions, improvements may be found in a model's performance by using a method called undersampling [57]. In undersampling, a subset of the training set is selected which increases the total percentage of fraudulent transactions. The intention of undersampling is to achieve better recall on the fraudulent transactions, as the model is less skewed toward labelling transactions as legitimate. A hyperparameter undersampling_num_negatives is implemented in the training pipeline which determines the number of legitimate transactions included in the training set.

Use of Categorical Variables
NNs typically use learned embedding layers to handle categorical features [58]. However, little work has been done on HE implementations of embedding layers. XGBoost does not currently support any extra optimizations for the use of categorical variables. For these reasons, the categorical features in the Vesta dataset are converted to numerical data.
For conversion to numerical, the pre-processing steps given in the starter code 3 provided by the organisors of the competition mentioned in Section 3.4.2. A simple label encoding converts the categorical features to a corresponding number and the NaN features are replaced with -999.

Choosing Model Hyperparameters
The wandb-sweep flag can be used to perform a hyperparameter Sweep of the model. These Sweeps were performed on each model to determine the hyperparameters which performed best on the validation set. num_estimators controls the number of decision trees in the model. Note that max_depth was hardcoded as 7 for the Vesta dataset, as it was found that using trees any deeper resulted in impractical model encryption times.

HE XGBoost
PPXGB encrypts the model by iterating through each of the trees and encrypting each node in the tree recursively using OPE. Note: An additional requirement when using the PPXGB model is that the max and min values in the dataset must be known in order to use the OPE scheme. Once a leaf node is reached, the encryption scheme switches to Paillier. A HMAC is used as a pseudorandom function to obfuscate the names of the features in a transaction [59] for additional privacy.
An improvement was made to PPXGB to allow for parallelising the tree encryption. As each tree can be encrypted independently, the encryption can be spread across multpile CPU cores. In this implementation, the number of cores can be specified using the cores flag. The list of trees is then evenly split into cores lists, and encryption for each list is run on a seperate core. In theory, if the number of trees T is greater than the number of cores chosen C , there should be a speedup by a factor of C over the original PPXGB function. In practice, thread management adds a cost which reduces the speedup.

HE Neural Network
A layer in a NN performs a matrix mutiplication of the layer input with the layer weights, and then adds the bias of the layer to produce the output. This evaluate the output of all the neurons in a layer in one operation. The CKKS scheme supports ciphertext-plaintext multiplication. This means the NN layer weights and biases do not need to be encrypted to be used as HE layer weights. Converting from a plaintext NN to a HE NN involves using a CKKS matrix multiplication on the encrypted input with the plaintext layer weights, followed by adding the plaintext layer bias.
A comparison between the model classes showing the corresponding layer operations is shown in Appendix A1.1. A debug option was also implemented to display the output of each layer, as shown in Appendix A1.2.

HE Inference
For the HE XGBoost model, Client must be supplied with the OPE key, the HMAC function, and the Paillier private key. Client hashes the transaction feature names with the HMAC, then encrypts the transaction features using the OPE key. This ciphertext is then sent to Host. When the encrypted result is received from Host, Client uses the Paillier private key to decrypt and view the result.
For the HE NN model, Client generates a CKKS public/private key pair. The transaction features are encoded as a CKKS vector and encrypted using the public key. This ciphertext is then sent to Host, along with the encryption context used, so Host can use the same context for the CKKS layer operations. The context could either be the same for every Client, or perhaps some Clients could choose higher levels of security. When the encrypted result is received from Host, Client uses the private key to decrypt and view the result.

Security
PPXGB uses the python-paillier library [60] for its implementation of the Paillier scheme, which chose a default key length of 3078 to provide 128 bits of security. The symmetric OPE key and the seed used for the HMAC are both 128 bits in length, providing 128 bits of security.
Microsoft SEAL requires that the security context is set up with 128 bits of security by default, by ensuring their encryption contexts are in accordance with the Homomorphic Encryption Security Standard [61]. This ensures that if the HE NN runs, it will have 128 bits of security.

Simulation
A simulation is provided to demonstrate the order of events in a Client-Host use case as described in Section 3. Next, the Secure Middle Server loads the XGBoost models from Host Files. Encryption/decryption keys are generated, and each model is encrypted. The encrypted models are saved to Host Files, and the keys are saved to Client Files. In practice, the Secure Middle Server would not have direct access to these file systems, and the transmission of the models and keys would be done over secure channels. For simplicity of the simulation, access to both file systems is granted.
Host provides Client with the specification for encoding and encrypting their transaction data, so that it is sent to Host in the correct format for input to the HE Model. Now that Host and Client have the necessary models and keys, the standard protocols for transaction inference described in Section 3.3 is done over HTTP.
Each part of the inference process is timed for comparison: encryption time, inference time, and the round trip time from the encryption of the input to decryption of output. This round-trip-time is the most accurate representation of how fast the system performs. However, as this simulation is designed to be run on the same machine, the typical latency involved with transmission over the internet is not included. Appendix A1.4 shows an example of the Client and Host interaction in the Simulation.

Results
In this section the results of HE conversion testing, fraud detection, latencies and storage sizes are presented.

Similarity Test
When the chosen XGBoost and NN models are selected, a test is done to ensure that their HE versions function correctly. A random subset of the test set is selected and inference on is performed using both the plaintext and HE models. The results from the HE model are decrypted, converted to binary, and compared with the results of the plaintext model. If they are not the same, the test is deemed a failure.
A subset is used rather than the full test set, as encrypting the full test set would take a lot of compute and time. An even balance of 75 fraudulent and 75 legitimate transactions are used in the test. This style of testing for HE inference is discussed in Section 6.3.6.
The HE XGBoost passed this test with all tested combinations of hyperparameters. The HE NN failed this test whenever more than one hidden layer was used.

Metrics Used
There is no consensus in the fraud detection literature on which metrics are best for evaluation. Using accuracy (percentage of transactions labelled correctly) is not practical due to the unbalanced data. Labelling every transaction as legitimate would result in very high accuracy.
In these results, the threshold used for converting a probability prediction into a binary value is 0.5, i.e. any transaction given at least 50% chance of being fraudulent is labelled as fraudulent. Different Hosts/Clients may choose this threshold to be lower or higher, depending on how sensitive they want their detector to be.
Threshold-free metrics evaluate models on their output before conversion to binary, and threshold-based metrics evaluate the binary decision. There is little consensus in the fraud detection literature on which metrics are best. In Reproducible Machine Learning for Credit Card Fraud Detection -Practical Handbook a combination of both is recommended [12] . In these results, the AUC ROC score and the Average Precision score are the threshold-free metrics chosen, and the percentage of each type of transaction labelled correctly are chosen as the threshold-based evaluation.

Models Chosen
The fraud detector to use on the test set is chosen based on how it performs on the validation set. Selecting for different metrics should provide different results on the test set. A comparison is made between the results obtained by the fraud detectors when each of the different metrics are selected for. Figure 5.1 shows selecting the NN model which had the best Average Precision score in a Sweep on the ULB dataset. An extra model which was selected by inspection is also included in each case, to check if a person's judgement could result in better performance than by using a metric. The model which was deemed to have the best balance of correct labels on the validation set was selected in each case. An example of the XGBoost model selected by inspection on the Vesta dataset is shown in Figure 5.2.  Table 5.1.

Detection Results
The results for each dataset are shown in Tables 5.2

Benchmarking
Next, the results from the benchmarking of various latencies and storage sizes in the system are shown. As the NN model for the Vesta dataset failed to work as a fraud detector, it was excluded from further HE implementation benchmarking.

Conditions for Benchmarking
Latency benchmarking was run on an Intel Core i7-6700K CPU @ 4.00GHz with an NVIDIA GeForce GTX1080 GPU. The models used for benchmarking were those with the overall best performance, as shown in Figures 5.2 and 5.3.
The CKKS scheme's encryption context can be tuned to achieve better performance. Using higher poly modulus degree and coefficient modulus results in better precision, but also higher latency. The HE NN for the ULB dataset had 47 neurons in its layer, which allowed for a bit precision of 30 to pass the Similarity Test. 16384 was chosen as the poly modulus degree, with a coefficient modulus of [40,30,30,30,40].
The HE XGBoost used for benchmarking on ULB had a max depth of 9 and 51 trees, while the model used for on Vesta had a max depth of 7 and 92 trees. A maximum depth of 7 used for all XGBoost models considered for Vesta, as testing showed that using a higher depth resulted in model encryption times that were impractical (over 12 minutes). Using a fixed depth of 7 kept the model encryption time around 3 minutes.

Latency Results
The training time for the NN depends on the number of epochs chosen, while XGBoost automatically chooses when to stop training. The number of epochs trained on the ULB dataset was 100. Although it still showed poor detection performance, more epochs (300) were used to train on the Vesta dataset. Early stopping could potentially be added to the NN when a targeted validation loss is reached, however, training time is less critical than the other latencies.   Table 5.5 compares the sizes of transactions and models in their plaintext and ciphertext forms. These are important due to storage usage and loading times. The difference in the size of plaintext transactions between the XGBoost and NN models is because they are stored as a Pandas DataFrame [62] for XGBoost, whereas the transactions for the NN are stored as NumPy arrays [63].

Discussion
In this section the performance of the models are discussed, limitations of the system are explained, and potential further work is proposed.

Performance on ULB Dataset
On the ULB dataset, the XGBoost model was successfully implemented as a CC fraud detector, with the best model correctly classifying 89% of the fraudulent transactions and 98.1% of legitimate transactions. The NN model also found success, with the best model achieving results of 77% and 99.93%. XGBoost was better at detecting fraudulent transactions, but the NN model produced less false positives.
In the latency benchmarking, HE XGBoost had an encrypted inference time of 6ms, while HE NN had a time of 296ms. When the encryption time of the transaction is added, the total latencies become 180ms and 306ms respectively. As the storage size of an encrypted HE XGBoost transaction is only 2.4kB, this would also be transmitted faster over the internet than the 641 kB transaction encrypted for HE NN.
Both models were successful on this dataset. HE XGBoost performs much faster, but the simpler deployment of HE NN may still make it the preferred choice.

Performance on Vesta Dataset
On the Vesta dataset, HE XGBoost found worse performance than on ULB, but the best model still correctly classified 47% of fraudulent transactions and 97.89% of legitimate transactions. The NN model failed to work as a fraud detector, likely due to the limited depth. Although it detected 72% of the fraudulent transactions, it also labelled 42% of legitimate transactions as fraudulent, which is unacceptable for a fraud detection system.
HE XGBoost had a fast inference time of 23ms, but saw a transaction encryption time of 2 seconds. Better encryption times could make HE XGBoost a decent model for this dataset.

Metric Selected For
No single basis for model choice between AUC ROC, Average Precision and Inspection consistently provided the best detection performance on the test set. For XGBoost on ULB, selecting for Average Precision produced the best results, while on the Vesta dataset, selecting by Inspection was the better option. The best metric for NN selection on ULB was probably Average Precision again, but very similar performance was found by Inspection. Overall, using Average Precision and the Inspection method proved to be the best option for these datasets.

Encrypted Outputs
The models proposed above produce an encrypted output, which can only be decrypted using the secret key, which is held by the Client. This creates a limitation on the Client & Bank use case, as the bank Host will be unable to determine if a given transaction is fraudulent, as the result is encrypted. Two possible solutions for this problem are as follows: 1. The Client's secret key is used in a TEE to decrypt the output. This would involve Client encrypting their secret key using a key shared with Host and transmitting it to a TEE. Inside the TEE, the secret key is decrypted, and used to decrypt the output of the HE model. The secret key is discarded, and the Host receives whether the transaction was deemed fraudulent.
2. The output of the model is not encrypted. This is possible using the XGBoost model, and other decision tree models. The inputs to the model are still encrypted, and still traverse the tree using OPE, but the leaf nodes are in plaintext, rather than using Paillier. This results in a model which takes encrypted inputs, and produces plaintext outputs.
This does however introduce further security concerns. Host could theoretically derive information about an encrypted transaction by producing the same output of the model via brute-force, using transaction data generated by Host.
One might ask why should HE be used at all if TEEs are going to be used in the pipeline anyways. The reason is that deploying complex attested code in TEEs is difficult to implement, especially at scale. Deploying a whole model inside a TEE could be very difficult, but using them for simple encryption/decryption steps to aid in the deployment of a HE model seems feasible.

Lack of Real Data
Using anonymized datasets means that our evaluation is undertaken on data that is not a direct representation of what would be used for training and inference in a production environment. Solutions to this would involve applying to gain access to private datasets, or using a simulator which accurately represents the distributions found in real data.
It is possible that some of the unknown features in these datasets could contain features engineered via aggregation, i.e. a value in a given transaction may contain information about previous transactions. This would undermine the Client & Bank use case, which would need transaction features to be independent from other transactions.

NN Model Depth
The model depth of the NN model was limited to one hidden layer by the Similarity Test. Even when using the highest CKKS precision of 60 bits, deeper NNs still failed. As each hidden layer adds another square activation function, it is likely this is the issue.
This limitation means that if potential detection performance increases are available with increasing model size, that performance is unavailable to this implementation.

Lack of Failure Assessment
In a plaintext fraud detection system, production datapoints which result in false-positives or true-negatives can be reviewed, and changes to the model can be made accordingly. As the Host is unable to view the transaction data, they are unable to review why failures in the system occur.
One potential solution to this is that users could be asked to share information about a specific transaction if it was a transaction that caused a failure. If the transaction was not particularly private to the Client, they may be willing to give the plaintext version to improve the system. This is similar to how software applications may ask for the local system logs after a crash.

HE XGBoost Information Leakage
As the tree nodes of the model use OPE to make comparisons on the input data, if the plaintext model is known to the Host, they could watch the path taken by the input data down each of the trees. Knowing the result of each of the comparisons made allows Host to determine a range of possible values for each feature in the input data. This attack could be done by Host, or by a third-party attacker if they have access to the plaintext model, the encrypted model, and encrypted transaction data. A prevention for this attack could be to also run the training in the Secure Middle Server, as it was in the original algorithm proposed [35]. But if Host has the data used for training and the information on how the model was trained, they could potentially make a replica model.

Deeper NN
To increase the depth of the CKKS NN, an interactive bootstrapping protocol could be introduced. Sending data back for decryption after every N layers in the model, but this would likely be too slow. It is also possible that the practical multiplicative depth could be increased by using other activation functions such as a polynomial approximation of ReLU.

HE Inference speedups
Parallelization across cores was implemented for the encryption of the PPXGB model, but it still struggled with long transaction encryption times. Further improvements could be reached by using GPU, FPGA, or ASIC.

Implementation on Live Data
As the datasets used have been anonymized, a true proof that this system is viable would be to train it on a private commercial dataset, and then test it on production transactions.

Ensemble
An ensemble of the two models could possibly be implemented, where the output of the HE XGBoost model and the HE NN model are combined to produce the final result. The leaf nodes of the XGBoost model could be encrypted using CKKS using the same key used for input to the NN model. This would allow the result of the two models to be added together. Alternatively, the two results could be returned to Client, and they would be combined after decryption.

Feature Engineering using HE
Feature engineering is widely used in CC fraud detection systems. This could potentially be implemented for encrypted transactions. An implementation would probably involve aggregating features across consecutive transactions via homomorphic addition, and producing mean values using homomorphic division.

Testing of HE Models
As HE models are slower than plaintext models, testing their performance on the full test set used for testing the plaintext model may be infeasible, as that also involves encrypting the full test set. Further work to build a confidence score of how well a HE model performs, or how similarly it performs to its plaintext equivalent could be valuable across domains.

Blockchain Use Case
Another proposed use case of the system relates to blockchain transaction fraud detection [64]. A transaction is made and added to a blockchain, along with extra transaction features. A smart contract [65] then runs a HE fraud detector on these encrypted features. If the transaction is predicted as fraudulent, the funds are not transferred.

Conclusion
In this paper, a system for private credit card fraud detection using homomorphic encryption was proposed. Two models, XGBoost and a feedforward classifier neural network, were trained as fraud detectors on plaintext data. Models were chosen based on their performance on a validation set. Choosing models based on their average precision score and by inspection produced the best performance on the test set. They were then converted to models using homomorphic encryption, for inference on encrypted transactions.
A comparison of the models found that the XGBoost model had better detection performance and faster latency on the chosen datasets than the neural network model. However, the complexity of deploying the encrypted XGBoost model may result in the neural network being preferred in some cases.