Smartphone Impostor Detection with Built-in Sensors and Deep Learning

Abstract: In this paper, we show that sensor-based impostor detection with deep learning can achieve excellent impostor detection accuracy at lower hardware cost compared to past work on sensor-based user authentication (the inverse problem) which used more conventional machine learning algorithms. While these methods use other smartphone users' sensor data to build the (user, non-user) classification models, we go further to show that using only the legitimate user's sensor data can still achieve very good accuracy while preserving the privacy of the user's sensor data (behavioral biometrics). For this use case, a key contribution is showing that the detection accuracy of a Recurrent Neural Network (RNN) deep learning model can be significantly improved by comparing prediction error distributions. This requires generating and comparing empirical probability distributions, which we show in an efficient hardware design. Another novel contribution is in the design of SID (Smartphone impostor Detection), a minimalist hardware accelerator that can be integrated into future smartphones for efficient impostor detection for different scenarios. Our SID module can implement many common Machine Learning and Deep Learning algorithms. SID is also scalable in parallelism and performance and easy to program. We show an FPGA prototype of SID, which can provide more than enough performance for real-time impostor detection, with very low hardware complexity and power consumption (one to two orders of magnitude less than related performance-oriented FPGA accelerators). We also show that the FPGA implementation of SID consumes 64.41X less energy than an implementation using the CPU with a GPU.

\section{Introduction}

Machine learning (ML) and deep learning techniques (DL) have become widely used for data analysis for various purposes in the real world. To support the computation and memory intensive deep learning computation, specialized accelerators have been studied for certain commonly used deep network architecture such as convolutional neural network (CNN) and recurrent neural network (RNN). These model-specific accelerators (MSA) are designed with the knowledge of the certain model and can thus leverage techniques like data reuse and data sparsity to deliver higher performance or energy efficiency than the general-purpose computing devices.

However, the data analysis in the real world is not always so simple that one specific model can fulfill the analysis task in all cases. In one application, different ML and DL models can be needed depending on the actual user case. It is also possible that the output of a model needs subsequent processing for better final results. If a specialized MSA is used, there are difficulties when the application has to switch to another model that the MSA cannot support or it has to transmit its results back to general-purpose computing devices for subsequent processing.

When the accelerators are built to optimize the trade-off between computation efficiency and design cost, there are two directions from the function perspective in which they can proceed to overcome the limitation of MSAs. The first one is to still pursue maximum power efficiency when the system, e.g. the cloud computing center, is not sensitive to the size or kinds of accelerators that will be used. The system designer can provide versatility by integrating multiple MSAs dedicated to different network architectures. The second direction is designing a versatile accelerator for models that will potentially be used. This direction is important for the Internet of things (IoT) or embedded systems because they are usually resource-constraint and not able to incorporate multiple MSAs. On the other hand, IoT or embedded systems always have a fixed range of functionality so that hardware designers can explore prior knowledge to decide the level of versatility.

We explore in the second direction and identify three critical challenges to build a hardware module that provides support for the entire analysis and produces the final result without the need for subsequent processing. These challenges make directly integrating
MSAs infeasible for versatile data analysis.
\begin{itemize}
    \item Challenge 1:
    A real application can require different ML/DL models depending on the real use cases.
    \item Challenge 2:
    A real application can apply auxiliary data processing techniques which gets processed after or even between model inference.
    \item Challenge 3:
    A real application can involve the control flow having various layers of loops
\end{itemize}

We propose an algorithm-hardware exploration methodology to design a minimalist but versatile hardware module using the example of impostor detection.

Smartphones are one of the most frequently used mobile devices that people use in their daily life. Users store their confidential information such as photos and videos in the smartphones. The smartphone is also used for remote access by financial or social applications. The impostor is the kind of attacker who controls the smartphone by stealing or cheating and pretends to be the real user. All the security properties of critical applications in the smartphone are threatened by the potential abuse from impostors. Therefore it is important to authenticate a user before a sensitive usage.

Explicit authentication such as password and PINs has been integrated into smartphones to prevent potential impostors from using the smartphone. Biometric-based authentication is also implemented in real smartphone using fingerprint~\cite{clancy2003secure} or facial information~\cite{jafri2009survey}. However, these protection mechanisms are all one-time checks. It is possible for an attacker to take control of the phone after the real user passes the primary authentication.

Sensor-based user authentication~\cite{lee2017implicit} has then been proposed as a second defense in addition to the primary authentication. The authentication system collects the readings from sensors like the accelerometer and the gyroscope which are built into typical smartphones. The system contains a machine learning model that is trained off-line with multiple users' data to distinguish the real user from others. With the model, the system is able to implicitly read the sensors and authenticate the user without explicitly asking for the user's input.

However, there are still problems when a machine learning (ML) or deep learning (DL) model is deployed for sensor-based impostor detection. The continuous detection on sensor readings causes periodic interruptions to the applications running on the smartphone processor and have a negative impact on user experience. A dedicated hardware impostor module is needed but existing performance oriented accelerators may not have the functional support for the whole analysis and require subsequent processing or they are over-designed to be too gigantic to fit in the smartphone and work under the power constraint of the sleep mode.
Therefore, we develop a methodology to design a lightweight hardware module on FPGA implementing only the necessary functional units for ML/DL models with the goal to provide enough performance using the least amount hardware resources and power consumption.

The methodology consists of two main steps. In the first step, we explore the necessary ML/DL models and auxiliary processing techniques by evaluating their accuracy. We leverage the prior knowledge of use cases and decide what models and techniques are useful in each case and should be supported by the module for impostor detection. In the second step, we decompose the models and processing techniques into a set of computation primitives. We then build a minimalist module,~SID, that is able to support the necessary primitives. We also consider the control flow of nested loops involved in some algorithms and provide relevant hardware support.

We present what models and auxiliary processing techniques are in different scenarios of impostor detection in~\gyrefsec{sec_sw_model}. We show how our~SID~module for self-contained impostor detection which is designed with the knowledge of selected algorithms and the goal to save hardware utilization and power consumption in~\gyrefsec{sec_hw}. The evaluation is given in~\gyrefsec{sec_evaluation}.~\gyrefsec{sec_related_work} contains the relationship with previous work. The whole paper is concluded in~\gyrefsec{sec_conclusion}.

\iffalse
The main contributions of this paper are:
\begin{packeditemize}

\item We summarize a set of ML/DL algorithms for impostor detection. In addition to conventional binary classification models which require other users' data for training, we investigate the model for time-series modeling to enable one-class training.

\item We identify the fundamental operation primitives needed by relevant ML/DL algorithms.

\item We show a novel lightweight and scalable implementation of a programmable hardware module for ML/DL that implements these primitives.

\item We evaluate the ML/DL models for real-time impostor detection in terms of accuracy, execution time and memory usage.

\item We define an algorithm-HW methodology that determines the best security-performance trade-off for a resource-constraint device,

\end{packeditemize}
\fi

\section{Introduction}

We rely heavily on our smartphones not only for communication but also to store our private information and access remote cloud services. This means a lot of confidential and sensitive data will be lost if smartphone theft happens\cite{wang2012smartphonetheft}. An impostor is defined as an adversary who gets hold of a smartphone that does not belong to him. The impostor may try to access information and services that he is not authorized to access, but the legitimate user of the smartphone is authorized to access. Hence, implicit impostor detection mechanism built into the smartphone would make stealing smartphones less attractive if the smartphone detects the impostor, and prevents access, at least to secret and sensitive material. Ideally, we would like to detect a potential impostor even before he/she tries to access any secret or sensitive data, code or services. Is it feasible to detect an impostor as he is just walking with the smartphone?

To solve this important security problem of smartphone impostor detection, or theft deterrence, we leverage two new opportunities in the computing landscape today: the ubiquitous inclusion of motion sensors in smartphones and the rise of deep learning algorithms. We will use both synergistically in our solution.

Built into all smartphones today is a sensor subsystem that comprises motion sensors and other sensors like light, barometer, magnetometer, etc. The most common sensors are the 3-axis accelerometer and the 3-axis gyroscope, which are found in almost all smartphones and smartwatches, not just the high-end ones. The readings from these sensors are not considered privacy-sensitive, unlike the GPS which gives the location, or cameras and microphones, which can divulge private scenes or conversations. We use data from these two widely-available sensors to characterize the motion of the person currently holding the smartphone, as he is walking away from the theft scene, etc. Past work has succeceed in using such sensors to implicitly and continuously authenticate a user\cite{lee2017implicit}. It seems that physiological behavior, like how a person walks or holds a smartphone can be captured by time-series measurements of an accelerometer or gyroscope, and analyzed by advanced algorithms.

For these sensor-based authentication systems, traditional Machine Learning (ML) algorithms like Support Vector Machine (SVM) and Kernel Ridge Regression (KRR) have been proposed. The rise of Deep Learning (DL) algorithms, where there is no need to first compute heuristically derived features, is an attractive alternative. Toward detecting the impostors, we consider two different user scenarios. In the first scenario, users subscribe to an "impostor Detection as a Service (IDaaS)". They register to a service and obtain a trained ML or DL model on their smartphone for subsequent real-time impostor detection. That model is trained with both the legitimate user's sensor data and the sensor data from other users of the IDaaS service. In the second scenario, users are more privacy-shy and do not want to send their sensor data to a cloud service. Their ML or DL model is trained locally, with only the legitimate user's sensor data. We show that there exists an important trade-off between the detection accuracy (usability) and the increased privacy of the user's sensor data.

Modern smartphones are equipped with various computation subsystems dedicated to different application domains\cite{hexagon:2014micro:codrescu2014hexagon}. For the domain of impostor detection, we investigate whether a small and energy-efficient hardware module for impostor detection can be designed for the self-contained computation. Unlike previous work on ML/DL accelerators whose goal is to get the maximum performance from one or a few specific ML or DL algorithms, like Convolutional Neural Networks (CNN), leveraging data reuse~\cite{diannao:Chen:2014:DSH:2541940.2541967}\cite{eyeriss2016isca} or matrix sparsity features\cite{han2016eie}, our goal is not to maximize performance, but sufficient performance at small size and low power consumption. Past work also looked at maximizing the energy efficiency of certain Deep Neural Network (DNN) accelerators, but this is different from our goal to pursue the minimal energy consumption.

We design Smartphone impostor Detection Engine (SIDE) to show that we can efficiently and effectively solve a real security problem (small hardware footprint and energy consumption). We call this architecture SID~for Smartphone impostor Detection engine. SID~reuses functional units where possible to reduce size and cost. Yet it is also scalable for higher performance if more parallel datapath tracks are implemented. It is designed to support not only the best deep learning algorithms we found for impostor detection in both user scenarios, but can also support other Machine Learning algorithms, calculations of empirical probability distributions and statistical tests. Programmability support provides flexibility in the choice of algorithms.

Our key contributions are:
\begin{packeditemize}

\item Investigating a comprehensive comparison of Machine Learning and Deep Learning algorithms for two user scenarios of impostor detection, where other people's data are used, or not used, in the training phase, trading off detection accuracy versus data privacy.

\item Achieving over 96\% accuracy in impostor detection for the IDaaS user scenario (2-class models), and over 88\% accuracy in the single-user scenario (1-class models).

\item Proposing~SID~ module that is versatile, scalable and efficient to perform self-contained impostor detection without subsequent processing on other devices.

\item Showing the trade-off between detection accuracy with performance and energy consumption when models are mapped to the~SID~module.

\item Showing a major difference in size between the~SID~module and existing accelerators with a similar target.

\item Showing SIDE uses 64.4X less energy than a server platform with a GPU

\end{packeditemize}
\section{Introduction}

We rely heavily on our smartphones not only for communication but also to store confidential and private data. Smartphones make it more convenient for us to access our data anywhere, anytime, on mobile devices.

However, smartphone theft is one of the most severe threats to smartphone users, leading to a loss of confidentiality and private data\cite{wang2012smartphonetheft}. Impostors are attackers who take over a smartphone and perform actions allowed for the smartphone owner but not for others. Impostor attacks are critical threats to the confidentiality, privacy and integrity of the secrets and personal information stored in the smartphone and accessible online through the smartphone. For powerful attackers who already know or can bypass the legitimate smartphone user's password or personal identification number (PIN), can a defense-in-depth mechanism be provided to detect impostors quickly before further damage is done? We show how this can be done effectively and implicitly, using an algorithm and architecture approach in this paper.

Implicit impostor detection mechanism built into the smartphone would make stealing smartphones less attractive if the smartphone can detect the impostor, and automatically prevent access to confidential information when an impostor is detected. Unlike explicit authentication with passwords, PINs or fingerprints, we would like to detect a potential impostor by looking for intrinsic differences in smartphone user behavior (also known as behavioral biometrics). For example, is it feasible to detect an impostor as he is just walking with the smartphone?

Our first insight is, the ubiquitous inclusion of motion sensors, e.g. 3-axis accelerometer and the 3-axis gyroscope, in smartphones provide a great opportunity to capture a user's motion patterns. We use data from these two widely-available sensors to characterize the motion of a potential thief currently holding the smartphone, as he is walking away from the theft scene, etc. In the literature, implicit smartphone authentication using sensors is primarily modeled as a binary classification problem using conventional Machine Learning (ML) algorithms like Support Vector Machine (SVM) and Kernel Ridge Regression (KRR)\cite{lee2017implicit}.

However, these classification approaches require both the legitimate user's sensor data and the sensor data from other users for training. This causes serious privacy issues as users must submit their sensitive behavioral data. In this paper, we explicitly consider the trade-off of security and privacy by investigating one-class anomaly detection for impostor detection. We show that using only the legitimate user's sensor data, we can still achieve very good accuracy while preserving the privacy of the user's sensor data, i.e. the user's behavioral biometrics.

Our intuition is to build a Recurrent Neural Network (RNN) based deep learning (DL) model that can represent the normal user's behavior. A large deviation of the observed behavior from the model's prediction indicates that the smartphone is used by an impostor. Different from previous work on using RNN for anomaly detection, we show that the detection accuracy of an RNN model can be significantly improved by generating and comparing empirical prediction error distributions (PEDs) rather than just comparing it with a threshold.

To reduce the attack surface and reduce the cost of impostor detection, we design a small and energy-efficient hardware module for impostor detection. Unlike previous work on ML/DL accelerators whose goal is to get the maximum performance from one or a few specific ML or DL algorithms (e.g. Convolutional Neural Networks ), our goal is not to maximize performance, but to provide \textit{sufficient performance at low cost and low power consumption}. Likewise, our goal is also different from past work that focuses on minimizing power consumption -- rather, we consider a broader goal of trade-offs in security, usability and privacy, with execution time, memory and energy consumption.

We design Smart\-phone Impo\-stor Dete\-ctor~(SID) to show that we can efficiently and effectively solve a real security problem with small hardware footprint and energy consumption.
SID~reuses functional units where possible to reduce size and cost. Yet it is also scalable for higher performance if more parallel datapath tracks are implemented. It is designed to support not only the best deep learning algorithms we found for impostor detection in both user scenarios (with and without other users' data for training), but can also support other Machine Learning algorithms, calculations of empirical probability distributions and statistical tests. Programmability support provides flexibility in the choice of algorithms and trade-offs in security, privacy, usability and cost.

Our key contributions are:
\begin{packeditemize}

\item We conduct a comprehensive comparison of ML and DL based classification and one-class outlier detection algorithms for impostor detection. We explicitly consider the trade-offs among security, privacy and usability. Our models achieve 97-98\% accuracy in impostor detection for the multi-user scenario (2-class models), and up to 92\% accuracy in the single-user scenario (1-class models).

\item We show that the detection accuracy of a Recurrent Neural Network (RNN) deep learning model, e.g. LSTM and GRU, can be significantly improved by comparing prediction error distributions.

\item We propose a hardware module that is versatile, scalable and efficient for performing impostor detection without preprocessing or postprocessing on other devices.

\item We show the trade-off between detection accuracy with performance, storage and energy consumption when ML/DL models are mapped to our~SID~module.

\item We show a major difference in size between the~SID~module and existing accelerators with a similar RNN target.

\item We show SID~uses 64.4X less energy than a platform with a GPU.

\end{packeditemize}

\section{Background}
\label{sec_background}

\subsection{Multi-class Models}
\label{sec_model_class}

We consider representative ML/DL models to classify a sample of sensor data as being the real user or a potential impostor. Logistic regression intrinsically provides the probability of a class, and thus can be naturally leveraged as the warning score. KRR alleviates over-fitting by penalizing large parameters. It is suitable for learning with limited data. We evaluate SVM because it is a light-weight and powerful linear model. MLP is a non-linear classifier and can automatically extract complex patterns from raw data, without heuristic and tedious hand-crafting.

\bheading{Regression}
Regression is a model describing the linear relationship between the input and the output. In the context of impostor detection, we use the logistic regression whose output could be treated as a probability for predicting whether a smartphone is currently under the control of an attacker. The computation for inference of a linear regression model is given in \gyrefeq{eq_linearreg}.
\begin{equation}
\label{eq_linearreg}
  Y = W \times x_{input} + b
\end{equation}

The logistic regression produces its output by applying an additional non-linear logistic function as shown in \gyrefeq{eq_logisreg}. The logistic function is actually converting a real number to an output between 0 and 1 which can be treated as a probability.
\begin{equation}
\label{eq_logisreg}
  Y = f(W \times x_{input} + b), where f(z) = \frac{1}{1+e^{-z}}
\end{equation}

\begin{figure}
    \centering
    \includegraphics[width=\linewidth]{./figures/svm_fig.pdf}
    \caption{The optimal hyperplane in SVM}
    \label{img_svm}
\end{figure}

\bheading{Support Vector Machine} The support vector machine (SVM) is one of the most commonly used models in machine learning. To perform a binary classification, it identifies a hyperplane in a high-dimensional space as the decision boundary. As shown in \gyreffig{img_svm}, the SVM model is trained to find the optimal hyperplane (the green line) which maximizes the margin between data points from two classes. It is provable~\cite{wiki:svm} that the optimal hyperplane can be decided by a set of support vectors sitting on the maximized margin in the data plane.

In the inference phase, \gyrefeq{eq_svm} is computed for testing samples. In~\gyrefeq{eq_svm}, there are $n\_sv$ support vectors, namely $\vec{x_i}$'s. $c_i$'s are coefficients obtained in the training phase. $y_i$ is either $1$ or $-1$ depending on which side of the hyperplane the support vector resides on. The angle bracket stands for the inner product of the input testing sample, $\vec{x_{input}}$, and a support vector. The class of the testing sample is decided by the sign of Y
\begin{equation}
  Y = \sum^{n\_sv}_{i=1} c_i y_i \langle \vec{x_i}, \vec{x_{input}} \rangle +
\label{eq_svm}
\end{equation}

To enable more flexible separation with a curve boundary, the inner product of $\vec{w}$ and $\vec{x_{input}}$ could be replaced by kernel functions, e.g. the Gaussian kernel in~\gyrefeq{eq_svm_kernel}. The use of kernel functions represents the inner product when the data points are projected into a higher-dimensional space so that a non-linear separation is made possible.
\begin{equation}
  \begin{split}
    &Y = \sum^{n\_sv}_{i=1} c_i y_i \kappa(\vec{x_i}, \vec{x_{input}}) + b \\
    &\kappa(\vec{x}, \vec{y}) = e^{- \gamma \norm{\vec{x}-\vec{y}}^2}
  \end{split}
\label{eq_svm_kernel}
\end{equation}

\bheading{Multilayer Perceptron}

Neural network (NN) models have been used for deep learning. Multilayer perceptron (MLP) is a basic feedforward NN model which contains at least three layers of neurons. The first layer of neurons is the input layer which collects the input data. The last layer is the output layer presenting the final results. Between the input and the output layer are the hidden layers and the depth of a NN is decided by the number of hidden layers. In the feedforward path, the neurons in the next layer is obtained by applying linear transformation and then non-linear activation functions to the neurons in the previous layer. The weight and bias used in the linear transformation are trainable with gradient descent and back-propagation. In the inference phase described in ~\gyrefeq{eq_mlp}, $W$'s and $b$'s are trained weight matrices and bias vectors superscripted with the layer index $l$.
\begin{equation}
\centering
\begin{split}
  &x^0 = x_{input} \\
  &x^{l+1} = f(W^l \times x^l + b^l
\end{split}
\label{eq_mlp}
\end{equation}

\subsection{Models for Time-Series Modeling}
\label{sec_model_time}

Deep learning models are also studied to model time-series. One usage of the model is to predict the incoming data based on the past observations. While the prediction can be directly used by forecasting applications, it is also feasible to detect anomalies by calculating the prediction error. If a model is assumed to represent the time-series generated by a certain subject, a high prediction error when predicting with this model will indicate an anomalous series or even a potential ongoing attack.

\begin{figure}
    \centering
    \includegraphics[width=\linewidth]{./figures/lstm_fig.pdf}
    \caption{The computations within a LSTM cell}
    \label{img_lstm}
\end{figure}

Long short-term memory (LSTM) has been proposed as a variant of recurrent neural network (RNN) to learn long-term dependency and avoid the problem of vanishing gradient~\cite{bengio1994learning}. Different from the stateless models mentioned in~\gyrefsec{sec_model_class}, the LSTM model has two hidden states (h state and c state) which can remember the previous input information.

When analyzing a series, a LSTM cell will update its hidden states for each input element as described in~\gyrefeq{eq_lstm} and ~\gyreffig{img_lstm}. In~\gyrefeq{eq_lstm}, the W's and U's are weight matrices, the b's are bias vectors and $x_t$ is the input feature vector at time t. Conceptually, with the previous h state $h_{t-1}$ and the input $x_t$, the LSTM cell first computes $cand_t$ as a candidate for new c state and three control gates, the forget gate $f_t$, the input gate $i_t$ and the output gate $o_t$. The new c state $c_t$ is then computed as a sum of old c state $c_{t-1}$ and $cand_t$ weighted by $f_t$ and $i_t$ respectively. The new h state $h_t$ is generated from $o_t$ and the new $c_t$.
\begin{equation}
\begin{split}
  &cand_t = tanh(W_c \times x_{t} + U_c \times h_{t-1} + b_c) \\
  &f_t = \sigma (W_f \times x_{t} + U_f \times h_{t-1} + b_f) \\
  &i_t = \sigma (W_i \times x_{t} + U_i \times h_{t-1} + b_i) \\
  &o_t = \sigma (W_o \times x_{t} + U_o \times h_{t-1} + b_o) \\
  &c_t = f_t \odot c_{t-1} + i_t \odot cand_t \\%, where f(z) is the activation function
  &h_t = o_t \odot tanh(c_t)
\end{split}
\label{eq_lstm}
\end{equation}

\section{Motivation and Threat Model}

Our threat model includes powerful attackers who can bypass the conventional explicit authentication mechanisms, e.g. personal identification number (PIN), password, voice or fingerprint. The explicit authentication bypassing could occur in different ways. For example, the PIN/password may not be strong enough. The attacker can actively figure out the weak pin/password by guessing or social engineering. Another example is the attacker taking the phone after the legitimate user has entered the secret or biometric for explicit authentication.

Once the explicit authentication is bypassed, the attacker can find and kill any suspicious impostor detection program running as a software process. Hence, we consider a hardware solution that cannot be disabled in this way.

We assume the smartphone has common, built-in motion sensors, e.g. the accelerometer and the gyroscope. We assume that the sensor readings are always available.

Our approach can be used with any explicit user authentication mechanism and can provide an extra layer of security in a defense-in-depth approach. Our detection mechanism does not conflict with the conventional explicit authentication but prevents, as much as possible, the exposure of secret information/services to an impostor even when the explicit authentication is broken.

While this paper assumes a single legitimate user of a smartphone, our detection methodology can be easily extended to allow multiple legitimate users sharing a smartphone, e.g., members in a family or tight-knit team.

\iffalse
We assume powerful attackers who can bypass the conventional authentication which asks for the personal secret explicitly. For instance, a thief or robber can spy on the smartphone owner and wait until the user operation on the smartphone which requires explicit authentication like personal identification number (PIN), voice or finger print. Then the thief may take a phone after the real user provides the information. The scenarios also include the attackers who exploit social engineering and weak password and are able to type the obtained password after a theft or robbery. The attacker can find any impostor detection application running as a normal process and kill it.

We protect users who will use the smartphone for a period that is along enough to collect user's signature motions. The smartphone for short-term use, e.g. as a demo smartphone for sale, is beyond our scope of protection.

Our impostor detection relies the motion of smartphone which is monitored through ubiquitous motion sensors like the accelerometer and the gyroscope. It provides implicit authentication which is an extra layer of security in a defense-in-depth approach. Our detection mechanism does not conflict with the conventional authentication but prevents, as much as possible, the exposure of secret information/services to an impostor even when the explicit authentication is broken.
\fi
\section{SID~Hardware Module}
\label{sec_hw}

Our goal is to design a small but versatile and program\-mable hardware module that can be integrated into a smartphone to perform impostor detection, without needing compu\-tation on another processor or accelerator. Design goals for our Smart\-phone Impo\-stor Dete\-ctor~(SID) are:
\begin{packeditemize}
\item {Flexibility for different ML/DL models and trade-offs of security, usability, privacy, execution time, storage and energy consumption},
\item {Scalability for more performance},
\item Reduced memory storage and access,
\item Minimized energy consumption, and
\item {Reduced attack surface for better security}.

\end{packeditemize}

While our primary goal is to design SID~to be able to perform the best algorithms for impostor detection, namely, MLP and SVM for the IDaaS binary classification scenario, and one-class PED-LSTM/GRU for the local detection scenario, we also want it to be flexible enough to support other ML/DL algorithms as well, possibly for future security needs. For performance scalability, we design SID~to allow more parallel tracks to be implemented, if desired. An innovative aspect of our design is that the SID~macro instructions implementing the selected ML/DL algorithm do not even have to be changed when the number of parallel tracks is increased.
We also design SID~to reduce memory traffic by reusing operands and having local storage in the execution stage.
To be feasible for implementation in a smartphone, we design SID~for reduced energy consumption to alleviate smartphone battery usage. To reduce the attack surface, SID~should be able to support detection without subsequent processing on another device like the CPU. For example, our SID~also does statistical testing, which is important to enhance ML/DL models. This includes collecting and comparing empirical probability distributions.

\subsection{Architecture Overview}
\label{sec_arch_overview}

For flexibility, we want SID~to be able to support different ML/DL algorithms, for different security/usability/privacy/cost trade-offs, as well as future security needs.

\begin{figure}[ht]
    \centering
    \includegraphics[width=\linewidth]{./figures/block_diagram.pdf}
    \caption{A SID~module implementing vector and matrix operations defined in \gyreftbl{tbl_primitives}}
    \label{img_blockdiagram}
\end{figure}

An overview of SID~is shown in \gyreffig{img_blockdiagram}. Unlike vector machines with vector registers of fixed length, we use memory (BRAM in the FPGA) to store the vector or matrix operands and results. The memory controlled by macro instructions in SID~is more efficient since it does not have fixed vector lengths and operates seamlessly with our automatic determination of the number of times to execute a vector or matrix operation. We also have a small local scratchpad memory for faster access to intermediate results during a macro operation. The scratchpad is used to reduce the memory traffic for matrix-vector multiplication and other vector operations (\gyrefsec{sec_parallel_operation}).

We propose a minimal implementation of a SID~module with four parallel tracks in the datapath, as shown in \gyreffig{img_blockdiagram}. Each track consists of a Look-up Table (LUT), a Multiplier and an Adder, which are put into three consecutive EXEcution stages. \gyrefsec{sec_macro_iteration} shows how SID~is programmed, how it controls the execution of macro instructions and how SID~is scalable for loops. We describe the operations implemented with maximal functional unit reuse and the memory access reduction in~\gyrefsec{sec_parallel_operation}. An efficient implementation of empirical probability distribution generation and comparison is given in~\gyrefsec{sec_op_kstest}.

\subsection{Instruction Encoding}
\subsection{Flexible Storage and Control}
\label{sec_inst_enc}
\subsection{Macro Iteration}

\begin{table}[ht]
\centering
\iffalse
\begin{tabular}{crcrcrclcrclcrclcr}
\multicolumn{1}{l}{\textbf{127}} & \textbf{124} & \multicolumn{1}{l}{\textbf{123}} & \textbf{110} & \multicolumn{1}{l}{\textbf{109}} & \textbf{96} & \multicolumn{2}{c}{\textbf{95}}      & \multicolumn{1}{l}{\textbf{94}} & \textbf{64} & \multicolumn{2}{c}{\textbf{63}}      & \multicolumn{1}{l}{\textbf{62}} & \textbf{32} & \multicolumn{2}{c}{\textbf{31}}      & \multicolumn{1}{l}{\textbf{30}} & \textbf{0} \\ \hline
\multicolumn{2}{|c|}{\textbf{Mode}}             & \multicolumn{2}{c|}{\textbf{Length}}            & \multicolumn{2}{c|}{\textbf{Width}}            & \multicolumn{2}{c|}{\textbf{Off\_x}} & \multicolumn{2}{c|}{\textbf{Addr\_x}}         & \multicolumn{2}{c|}{\textbf{Off\_y}} & \multicolumn{2}{c|}{\textbf{Addr\_y}}         & \multicolumn{2}{c|}{\textbf{Off\_z}} & \multicolumn{2}{c|}{\textbf{Addr\_z}}        \\ \hline
\multicolumn{2}{|c|}{\textbf{4 bits}}           & \multicolumn{2}{c|}{\textbf{14 bits}}           & \multicolumn{2}{c|}{\textbf{14 bits}}          & \multicolumn{2}{c|}{\textbf{1 bit}}  & \multicolumn{2}{c|}{\textbf{31 bits}}         & \multicolumn{2}{c|}{\textbf{1 bit}}  & \multicolumn{2}{c|}{\textbf{32 bits}}         & \multicolumn{2}{c|}{\textbf{1 bit}}  & \multicolumn{2}{c|}{\textbf{32 bits}}        \\ \hline
\end{tabular}
\fi
\resizebox{0.48\textwidth}{!}{
\begin{tabular}{crcrcrcrcrcr}
\multicolumn{1}{l}{\textbf{127}} & \textbf{124} & \multicolumn{1}{l}{\textbf{123}} & \textbf{110} & \multicolumn{1}{l}{\textbf{109}} & \textbf{96} & \multicolumn{1}{l}{\textbf{95}} & \textbf{64} & \multicolumn{1}{l}{\textbf{63}} & \textbf{32} & \multicolumn{1}{l}{\textbf{31}} & \textbf{0} \\ \hline
\multicolumn{2}{|c|}{Mode}                      & \multicolumn{2}{c|}{Length}                     & \multicolumn{2}{c|}{Width}                     & \multicolumn{2}{c|}{Addr\_x}                  & \multicolumn{2}{c|}{Addr\_y}                  & \multicolumn{2}{c|}{Addr\_z}                 \\ \hline
\multicolumn{2}{|c|}{4 bits}                    & \multicolumn{2}{c|}{14 bits}                    & \multicolumn{2}{c|}{14 bits}                   & \multicolumn{2}{c|}{32 bits}                  & \multicolumn{2}{c|}{32 bits}                  & \multicolumn{2}{c|}{32 bits}                 \\ \hline
\end{tabular}
}
\caption{SID~Macro Instruction with Automatic Scalable FSM Contro
}
\label{tbl_inst_fmt}

\iffalse
\begin{tabular}{|c|c|c|c|c|c|c|c|c|}
\hline
\textbf{Mode}              & \textbf{Length}            & \textbf{Width}            & \textbf{Off\_X}         & \textbf{Addr\_X}         & \textbf{Off\_Y}         & \textbf{Addr\_Y}         & \textbf{Off\_Z}         & \textbf{Addr\_Z} \\ \hline
\textbf{4 bits}              & \textbf{14 bits}            & \textbf{14 bits}            & \textbf{1 bit}         & \textbf{31 bits}         & \textbf{1 bit}         & \textbf{31 bits}         & \textbf{1 bit}         & \textbf{31 bits} \\ \hline
\textbf{Inst{[}127:124{]}} & \textbf{Inst{[}123:110{]}} & \textbf{Inst{[}109:96{]}} & \textbf{Inst{[}95{]}} & \textbf{Inst{[}94:64{]}} & \textbf{Inst{[}63{]}} & \textbf{Inst{[}62:32{]}} & \textbf{Inst{[}31{]}} & \textbf{Inst{[}30:0{]}} \\ \hline
\end{tabular}
\fi
\end{table}

\subsection{Scalable FSM Control with Macro Operations}
\label{sec_macro_iteration}

We provide a scalable programming interface to allow ML/DL models with different operations and sizes. The software in SID~is a sequence of macro-instructions, which encode a flexible number of iterations for an entire vector or matrix operation.
Each macro instruction initializes the finite state machine (FSM) state of the control unit to indicate the number of iterations of the specified operation. Each cycle, the FSM (in decode stage in~\gyreffig{img_blockdiagram}) updates the number of uncomputed iterations, according to the number of parallel tracks, to decide when a macro-instruction finishes (details in the following paragraphs).
This means the same code can run on the SID~hardware modules with a different number of parallel tracks without modification

The format of a SID~macro instruction is shown in~\gyreftbl{tbl_inst_fmt}. The \textbf{Mode} field specifies one of the operation modes in~\gyreftbl{tbl_primitives}, and the~\textbf{Length} and~\textbf{Width} fields initialize the control FSM and indicate when the execution of the macrio instruction finishes.
The control FSM has three state registers,~\textbf{reg\_le\-ngth},~\textbf{reg\_wid\-th} and~\textbf{reg\_wid\-th\_copy}, to track the number of iterations that have not been finished and automatically decide when to stop the FSM and move to the next instruction (which initiates a new FSM). The FSM can be configured by instructions in two ways: the one-dimension iteration and the matrix-vector iteration

\begin{table*}[ht]
\centering
\resizebox{\textwidth}{!}{

\iffalse
\begin{tabular}{|l|l|c|c|c|c|c||c|c|c|c|c|c|}
\hline
\multirow{2}{*}{\textbf{Operations}} & \multirow{2}{*}{\textbf{Description}}                                                                           & \multicolumn{5}{c||}{\textbf{IDaaS}}                                                                                & \multicolumn{5}{c|}{\textbf{LAD}}                                                                                & \textbf{}                                                         \\ \cline{3-13}
                                     &                                                                                & \textbf{KRR}              & \textbf{LR}               & \textbf{MLP}              & \textbf{\begin{tabular}[c]{@{}c@{}}SVM\\ Basic\end{tabular}} & \textbf{\begin{tabular}[c]{@{}c@{}}SVM w/\\ Gaussian\\ Kernel\end{tabular}} & \multicolumn{1}{c|}{\textbf{\begin{tabular}[c]{@{}c@{}}OCSVM w/\\ Gaussian\\ Kernel\end{tabular}}} & \textbf{LSTM}             & \textbf{KS-test}          & \textbf{\begin{tabular}[c]{@{}c@{}}PED-LSTM/\\ GRU-OCSVM\end{tabular}} & \textbf{\begin{tabular}[c]{@{}c@{}}PED-LSTM/\\ GRU-Mvote\end{tabular}} & \textbf{\begin{tabular}[c]{@{}c@{}}Support\\ Status\end{tabular}} \\ \hline
\textbf{Vadd}                        & \textbf{\begin{tabular}[c]{@{}l@{}}Element-wise addition\\ of two vectors\end{tabular}}                         & \checkmark & \checkmark & \checkmark & \checkmark                                    & \checkmark                                                   & \checkmark                                                                          & \checkmark & \checkmark & \checkmark                                         & \checkmark                                         & Yes                                                               \\ \hline
\textbf{Vsub}                        & \textbf{\begin{tabular}[c]{@{}l@{}}Element-wise subtraction\\ of two vectors\end{tabular}}                      &                           &                           &                           &                                                              & \checkmark                                                   & \checkmark                                                                          & \checkmark & \checkmark & \checkmark                                         & \checkmark                                         & Yes                                                               \\ \hline
\textbf{Vmul}                        & \textbf{\begin{tabular}[c]{@{}l@{}}Element-wise multiplication\\ of two vectos\end{tabular}}                    & \checkmark &                           &                           &                                                              & \checkmark                                                   & \checkmark                                                                          & \checkmark & \checkmark & \checkmark                                         & \checkmark                                         & Yes                                                               \\ \hline
\textbf{Vsgt}                        & \textbf{\begin{tabular}[c]{@{}l@{}}Element-wise set-greater-than\\ of two vectors\end{tabular}}                 & \checkmark & \checkmark & \checkmark & \checkmark                                    & \checkmark                                                   & \checkmark                                                                          & \checkmark &                           & \checkmark                                         & \checkmark                                         & Yes                                                               \\ \hline
\textbf{Vsig}                        & \textbf{Sigmoid function of a vector}                                                                           & \checkmark &                           & \checkmark &                                                              &                                                                             &                                                                                & \checkmark &                           & \checkmark                                         & \checkmark                                         & Yes                                                               \\ \hline
\textbf{Vtanh}                       & \textbf{Tanh function of a vector}                                                                              &                           &                           & \checkmark &                                                              &                                                                             &                                                                                & \checkmark &                           & \checkmark                                         & \checkmark                                         & Yes                                                               \\ \hline
\textbf{Vexp}                        & \textbf{Exponential function of a vector}                                                                       &                           &                           &                           &                                                              & \checkmark                                                   & \checkmark                                                                          &                           &                           & \checkmark                                         &                                                                   & Yes                                                               \\ \hline
\textbf{Mvmul}                       & \textbf{\begin{tabular}[c]{@{}l@{}}Multiplication of a\\ matrix and a vector\end{tabular}}                      & \checkmark & \checkmark & \checkmark & \checkmark                                    & \checkmark                                                   & \checkmark                                                                          & \checkmark &                           & \checkmark                                         & \checkmark                                         & Yes                                                               \\ \hline
\textbf{VSsgt}                       & \textbf{\begin{tabular}[c]{@{}l@{}}Set-greater-than to compare a\\ scalar and a vector's elements\end{tabular}} &                           &                           &                           &                                                              &                                                                             &                                                                                &                           & \checkmark & \checkmark                                         & \checkmark                                         & Yes                                                               \\ \hline
\textbf{Vmaxabs}                     & \textbf{\begin{tabular}[c]{@{}l@{}}Find the maximum absolute\\ value of a vector\end{tabular}}                  & \checkmark &                           &                           &                                                              &                                                                             &                                                                                &                           & \checkmark & \checkmark                                         & \checkmark                                         & Yes                                                               \\ \hline
\textbf{Vsqnorm}                     & \textbf{Squared L2-norm of a vector}                                                                            &                           &                           &                           &                                                              & \checkmark                                                   & \checkmark                                                                          & \checkmark &                           & \checkmark                                         & \checkmark                                         & Yes                                                               \\ \hline \hhline{|=|=|=|=|=|=|=|=|=|=|=|=|=|}
\textbf{Loop}                        & \textbf{Start a loop of operations}                                                                             &                           &                           &                           & \checkmark                                    & \checkmark                                                   & \checkmark                                                                          &                           & \checkmark & \checkmark                                         & \checkmark                                         & Yes                                                               \\ \hline
\textbf{RegAddi}                     & \textbf{\begin{tabular}[c]{@{}l@{}}Add an immediate number\\ to a control register\end{tabular}}                &                           &                           &                           & \checkmark                                    & \checkmark                                                   & \checkmark                                                                          &                           & \checkmark & \checkmark                                         & \checkmark                                         & Yes                                                               \\ \hline
\textbf{RegStore}                    & \textbf{Store a control register(s)}                                                                            &                           &                           &                           & \checkmark                                    & \checkmark                                                   & \checkmark                                                                          &                           & \checkmark & \checkmark                                         & \checkmark                                         & Yes                                                               \\ \hline
\textbf{RegLoad}                     & \textbf{Load a control register(s)}                                                                             &                           &                           &                           & \checkmark                                    & \checkmark                                                   & \checkmark                                                                          &                           & \checkmark & \checkmark                                         & \checkmark                                         & Yes                                                               \\ \hline \hhline{|=|=|=|=|=|=|=|=|=|=|=|=|=|}
\textbf{Vargmax}                     & \textbf{\begin{tabular}[c]{@{}l@{}}Find the index of the\\ maximum in a vector\end{tabular}}                    & \checkmark &                           &                           &                                                              &                                                                             &                                                                                &                           &                           &                                                                   &                                                                   & No                                                                \\ \hline
\textbf{Vmin}                        & \textbf{Find the minimum in a vector}                                                                           & \checkmark &                           &                           &                                                              &                                                                             &                                                                                &                           &                           &                                                                   &                                                                   & No                                                                \\ \hline
\textbf{Vmax2}                       & \textbf{\begin{tabular}[c]{@{}l@{}}Find the second largest\\ number in a vector\end{tabular}}                   & \checkmark &                           &                           &                                                              &                                                                             &                                                                                &                           &                           &                                                                   &                                                                   & No                                                                \\ \hline
\textbf{VFFT}                        & \textbf{\begin{tabular}[c]{@{}l@{}}Compute the Fourier\\ transform of a vector\end{tabular}}                    & \checkmark &                           &                           &                                                              &                                                                             &                                                                                &                           &                           &                                                                   &                                                                   & No                                                                \\ \hline
\textbf{Vsqrt}                       & \textbf{\begin{tabular}[c]{@{}l@{}}Compute the square root\\ of each element in a vector\end{tabular}}          & \checkmark &                           &                           &                                                              &                                                                             &                                                                                &                           &                           &                                                                   &                                                                   & No                                                                \\ \hline
\end{tabular}
\fi

\begin{tabular}{|l|l|c|c|c|c||c|c|c|c|c|c|c|}
\hline
\multirow{2}{*}{\textbf{\begin{tabular}[c]{@{}l@{}}Operation\\ Modes\end{tabular}}} & \multirow{2}{*}{\textbf{Description}}                                                                           & \multicolumn{4}{c||}{\textbf{IDaaS}}                                                                                & \multicolumn{6}{c|}{\textbf{LAD}}                                                                                & \multirow{2}{*}{\textbf{\begin{tabular}[c]{@{}c@{}}Support\\ Status\end{tabular}}} \\ \cline{3-12}
                                                                                &                                                                                & \textbf{LR}               & \multicolumn{1}{c|}{\textbf{KRR}} & \textbf{\begin{tabular}[c]{@{}c@{}}SVM w/\\ Gaussian\\ Kernel\end{tabular}} & \multicolumn{1}{c||}{\textbf{MLP}} & \textbf{\begin{tabular}[c]{@{}c@{}}OCSVM w/\\ Gaussian\\ Kernel\end{tabular}} & \textbf{LSTM}             & \multicolumn{1}{c|}{\textbf{GRU}} & \textbf{KS-test}          & \textbf{\begin{tabular}[c]{@{}c@{}}PED-\\ LSTM/GRU\\ -OCSVM\end{tabular}} & \textbf{\begin{tabular}[c]{@{}c@{}}PED-\\ LSTM/GRU\\ -Mvote\end{tabular}} &                                                                                \\ \hline
\textbf{Vadd}                                                                       & \textbf{\begin{tabular}[c]{@{}l@{}}Element-wise addition\\ of two vectors\end{tabular}}                         & \checkmark & \checkmark         & \checkmark                                                   & \checkmark         & \checkmark                                                     & \checkmark & \checkmark         & \checkmark & \checkmark                                             & \checkmark                                             & Yes                                                                                \\ \hline
\textbf{Vsub}                                                                       & \textbf{\begin{tabular}[c]{@{}l@{}}Element-wise subtraction\\ of two vectors\end{tabular}}                      &                           &                                   & \checkmark                                                   &                                   & \checkmark                                                     & \checkmark & \checkmark         & \checkmark & \checkmark                                             & \checkmark                                             & Yes                                                                                \\ \hline
\textbf{Vmul}                                                                       & \textbf{\begin{tabular}[c]{@{}l@{}}Element-wise multiplication\\ of two vectos\end{tabular}}                    &                           & \checkmark         & \checkmark                                                   &                                   & \checkmark                                                     & \checkmark & \checkmark         & \checkmark & \checkmark                                             & \checkmark                                             & Yes                                                                                \\ \hline
\textbf{Vsgt}                                                                       & \textbf{\begin{tabular}[c]{@{}l@{}}Element-wise set-greater-than\\ of two vectors\end{tabular}}                 & \checkmark & \checkmark         & \checkmark                                                   & \checkmark         & \checkmark                                                     & \checkmark & \checkmark         &                           & \checkmark                                             & \checkmark                                             & Yes                                                                                \\ \hline
\textbf{Vsig}                                                                       & \textbf{Sigmoid function of a vector}                                                                           &                           & \checkmark         &                                                                             & \checkmark         &                                                                               & \checkmark & \checkmark         &                           & \checkmark                                             & \checkmark                                             & Yes                                                                                \\ \hline
\textbf{Vtanh}                                                                      & \textbf{Tanh function of a vector}                                                                              &                           &                                   &                                                                             & \checkmark         &                                                                               & \checkmark & \checkmark         &                           & \checkmark                                             & \checkmark                                             & Yes                                                                                \\ \hline
\textbf{Vexp}                                                                       & \textbf{Exponential function of a vector}                                                                       &                           &                                   & \checkmark                                                   &                                   & \checkmark                                                     &                           &                                   &                           & \checkmark                                             &                                                                       & Yes                                                                                \\ \hline
\textbf{Mvmul}                                                                      & \textbf{\begin{tabular}[c]{@{}l@{}}Multiplication of a\\ matrix and a vector\end{tabular}}                      & \checkmark & \checkmark         & \checkmark                                                   & \checkmark         & \checkmark                                                     & \checkmark & \checkmark         &                           & \checkmark                                             & \checkmark                                             & Yes                                                                                \\ \hline
\textbf{VSsgt}                                                                      & \textbf{\begin{tabular}[c]{@{}l@{}}Set-greater-than to compare a\\ scalar and a vector's elements\end{tabular}} &                           &                                   &                                                                             &                                   &                                                                               &                           &                                   & \checkmark & \checkmark                                             & \checkmark                                             & Yes                                                                                \\ \hline
\textbf{Vmaxabs}                                                                    & \textbf{\begin{tabular}[c]{@{}l@{}}Find the maximum absolute\\ value of a vector\end{tabular}}                  &                           & \checkmark         &                                                                             &                                   &                                                                               &                           &                                   & \checkmark & \checkmark                                             & \checkmark                                             & Yes                                                                                \\ \hline
\textbf{Vsqnorm}                                                                    & \textbf{Squared L2-norm of a vector}                                                                            &                           &                                   & \checkmark                                                   &                                   & \checkmark                                                     & \checkmark & \checkmark         &                           & \checkmark                                             & \checkmark                                             & Yes                                                                                \\ \hline
\hhline{|=|=|=|=|=|=|=|=|=|=|=|=|=|}
\iffalse
\textbf{Loop}                                                                       & \textbf{Start a loop of operations}                                                                             &                           &                                   & \checkmark                                                   &                                   & \checkmark                                                     &                           &                                   & \checkmark & \checkmark                                             & \checkmark                                             & Yes                                                                                \\ \hline
\textbf{RegAddi}                                                                    & \textbf{\begin{tabular}[c]{@{}l@{}}Add an immediate number\\ to a control register\end{tabular}}                &                           &                                   & \checkmark                                                   &                                   & \checkmark                                                     &                           &                                   & \checkmark & \checkmark                                             & \checkmark                                             & Yes                                                                                \\ \hline
\textbf{RegStore}                                                                   & \textbf{Store a control register(s)}                                                                            &                           &                                   & \checkmark                                                   &                                   & \checkmark                                                     &                           &                                   & \checkmark & \checkmark                                             & \checkmark                                             & Yes                                                                                \\ \hline
\textbf{RegLoad}                                                                    & \textbf{Load a control register(s)}                                                                             &                           &                                   & \checkmark                                                   &                                   & \checkmark                                                     &                           &                                   & \checkmark & \checkmark                                             & \checkmark                                             & Yes                                                                                \\ \hline
\hhline{|=|=|=|=|=|=|=|=|=|=|=|=|=|}
\fi
\textbf{Vargmax}                                                                    & \textbf{\begin{tabular}[c]{@{}l@{}}Find the index of the\\ maximum in a vector\end{tabular}}                    &                           & \checkmark         &                                                                             &                                   &                                                                               &                           &                                   &                           &                                                                       &                                                                       & No                                                                                \\ \hline
\textbf{Vmin}                                                                       & \textbf{Find the minimum in a vector}                                                                           &                           & \checkmark         &                                                                             &                                   &                                                                               &                           &                                   &                           &                                                                       &                                                                       & No                                                                                \\ \hline
\textbf{Vmax2}                                                                      & \textbf{\begin{tabular}[c]{@{}l@{}}Find the second largest\\ number in a vector\end{tabular}}                   &                           & \checkmark         &                                                                             &                                   &                                                                               &                           &                                   &                           &                                                                       &                                                                       & No                                                                                \\ \hline
\textbf{VFFT}                                                                       & \textbf{\begin{tabular}[c]{@{}l@{}}Compute the Fourier\\ transform of a vector\end{tabular}}                    &                           & \checkmark         &                                                                             &                                   &                                                                               &                           &                                   &                           &                                                                       &                                                                       & No                                                                                \\ \hline
\textbf{Vsqrt}                                                                      & \textbf{\begin{tabular}[c]{@{}l@{}}Compute the square root\\ of each element in a vector\end{tabular}}          &                           & \checkmark         &                                                                             &                                   &                                                                               &                           &                                   &                           &                                                                       &                                                                       & No                                                                                \\ \hline
\end{tabular}

}

\caption{Computation primitives needed by different ML/DL models and statistical testing.}
\label{tbl_primitives}
\end{table*}

The one-dimension iteration in a vector operation uses only the \textbf{reg\_length}. The value of \textbf{reg\_length} is initialized by the~\textbf{length} field of the instruction. During execution, \textbf{reg\_length} is decreased every cycle by~\textbf{N(track)}, which is the number of parallel tracks, until \textbf{reg\_length} is no larger than~\textbf{N(track)} and the FSM lets the module fetch the next instruction.

The matrix-vector iteration, is used in a tiling manner. It involves all three state registers.
\gyreffig{img_fsm_diagram_mvmul} shows how the FSM gets updated every cycle in the matrix-vector multiplication mode. When an instruction for matrix-vector operation is fetched, the \textbf{length} field initializes \textbf{reg\_le\-ngth} and the \textbf{wid\-th} field initializes both \textbf{reg\_wid\-th} and \textbf{r\-eg\_wid\-th\_copy}.
~\textbf{reg\_width}, which gets decremented every cycle,
controls the downward iteration along a matrix column. When one inner iteration using \textbf{reg\_wid\-th} is finished, \textbf{r\-eg\_wid\-th\_copy} refills \textbf{reg\_width} and \textbf{reg\_le\-ngth} gets decremented by \textbf{N(track)}, which is the number of parallel tracks as defined above.
This corresponds to moving to the next slice of matrix columns. When the last slice of columns in the matrix is computed, the next instruction can be fetched.
\begin{figure}[ht]
    \centering
    \includegraphics[width=\linewidth]{./figures/fsm_diagram_mvmul.pdf}
    \caption{FSM for matrix-vector iteration, e.g. Mvmul. N(track) is the number of parallel tracks.}
    \label{img_fsm_diagram_mvmul}
\end{figure}

\iffalse
\begin{lstlisting}[language=C,
xleftmargin=.5em, framexleftmargin=.5em, caption={Vector Operation FSM Control}, label={alg_fsm_vec}]
  //N_track: Number of computation tracks
  FSM of vector-vector iteration
    Init:
      reg_length = new_inst[123:110]
    Execution:
      if reg_length > N
        reg_length -= N
      else
        fetch_new_inst
\end{lstlisting}
\begin{lstlisting}[language=C, xleftmargin=.5em, framexleftmargin=.5em, caption={Matrix-vector Operation FSM Control}, label={alg_fsm_mvmul}]
    //N_track: Number of computation tracks
    Init:
      reg_length = new_inst[123:110]
      reg_width  = new_inst[109:96]
      reg_width_copy  = new_inst[109:96]
    Execution:
      if reg_width > 1
        reg_width -= 1
      else
        if reg_length > N_track
          reg_length -= N_track
          reg_width  =  reg_width_copy
        else
          fetch_new_inst
\end{lstlisting}
\fi

This encoding-control mechanism has the advantages of both scalability and performance. The design is scalable since the hardware is aware of the number of parallel data tracks that are implemented and can perform automatic control of the loop(s). For performance, the control by FSM avoids using branch instructions for frequent jump-backs as needed in general-purpose processors, which can take up a large portion of processor throughput for simple loop bodies.

\subsection{Parallel Operations}
\subsection{Functional Operations Supported}
\label{sec_parallel_operation}

Our basic hardware module, SID, supports all the operations from \textbf{Vadd} through \textbf{Vsqnorm} in~\gyreftbl{tbl_primitives}.
The instructions from \textbf{Vargmax} to \textbf{Vsqrt} (at the bottom of~\gyreftbl{tbl_primitives}) are needed only for extracting the features, e.g. computing minimum, maximum, square-root, Fourier transform frequency components, for a KRR algorithm~\cite{lee2017implicit}. We decide not to implement these since they introduce significant hardware complexity while achieving lower accuracy than the other algorithms (see \gyreftbl{tbl_accuracy_binary}).

For a detection algorithm, programming SID~is directly writing a sequence of instructions corresponding to the operations in the equations for model inference, and specifying the sizes of the matrices and vectors, i.e. model parameters, in the instruction.

We describe briefly the vector and matrix-vector operation modes supported by going down the rows of~\gyreftbl{tbl_primitives}. Key design optimizations are reusing functional units to reduce the hardware cost, automatic FSM operation, tiling and adding the local scratchpad memory to reduce memory accesses.

\bheading{Basic vector-vector operations.}
The addition (\textbf{Vadd}), subtraction (\textbf{Vsub}) and multiplication (\textbf{Vmul}) are straightforward. Vector comparison (\textbf{Vsgt}) is implemented as a set-greater-than (SGT) operation where

\begin{equation}
c[i] = \begin{cases}
    1, & a[i] \geq b[i] \\
    0, & a[i] < b[i]
\end{cases}
, i = 0, ..., len(c)-1
\end{equation}
}
In the~SID~module,~\textbf{Vadd},~\textbf{Vsub} and~\textbf{Vsgt} modes are computed in parallel by the 4 parallel tracks shown in~\gyreffig{img_blockdiagram}~in the adder stage (EXE2). The~\textbf{Vmul} mode only uses the multiplier stage (EXE1) for parallel multiplication.

\bheading{Reuse of functional units in vector non-linear functions.}
When computing non-linear functions like sigmoid (\textbf{Vsig}), tanh (\textbf{Vtanh}) and exponential (\textbf{Vexp}),
we use the look-up table (LUT) implementation to compute the linear approximation. This avoids needing to implement complex non-linear functions, while providing the flexibility of implementing arbitrary non-linear functions by table lookup of the slope and intercept for linear approximation.
An added benefit of our approach over prior work \cite{diannao:Chen:2014:DSH:2541940.2541967} is that we place the LUTs before the multipliers and adders in the three consecutive execution stages so that no extra multipliers or adders are needed.
The ELE0 (LUT) stage of SID~outputs a slope, $k(i)$, and an intercept, $b(i)$, for each input value. The interpolation is then computed in the later two stages as $Z[i] = k[i] \times X[i] + b[i]$ with Z[i] being the value of the non-linear function for input X[i].
\iffalse
\begin{equation}
\label{eq_vact}
Z[i] = k[i] \times X[i] + b[i]
\end{equation}
\fi

\iffalse
\begin{figure}[ht]
    \centering
    \includegraphics[width=\linewidth]{./figures/vact.pdf}
    \caption{Active function units in Vact Mode}
    \label{img_vact}
\end{figure}
\fi

\iffalse
For a weight matrix $W$ and an input vector $X$, a matrix-vector multiplication computes
\begin{equation}
  Y[i] = \sum_{j=0}^{n-1} W[i,j] \times X[j], i = 0, ..., m-1
\end{equation}
\fi

\bheading{Local scratchpad with tiling and reuse of adders in matrix-vector multiplication.}
\textbf{MVmul} mode computes the product of a weight matrix and an input vector.
We have two optimizations for \textbf{MVmul} mode.

First, instead of having another adder tree stage to sum the products computed in the EXE1 stage, we put multiplexers before the adders in EXE2 stage to reuse the adders, which saves on hardware cost.
In the 4-track example, the first three adders sum the four products. The fourth one adds the sum to the partial sum read from the local scratchpad and writes the new partial sum back to the local scratchpad if the computation is not finished.

\iffalse
\begin{figure}[ht]
    \centering
    \includegraphics[width=\linewidth]{./figures/mvmul_fig.pdf}
    \caption{Active function units in MVmul Mode}
    \label{img_mvmul}
\end{figure}
\fi

\begin{figure}[ht]
    \centering
    \includegraphics[width=\linewidth]{./figures/matrix_fig.pdf}
    \caption{The macro instruction (top) and tiling (bottom) of the matrix-vector multiplication in the \textbf{MVmul} Mode}
    \label{img_matrix}
\end{figure}

Second, we use a local scratchpad memory and loop tiling to save the latency of storing and accessing partial sums from memory and also reduce the memory traffic. A specification of a matrix-vector multiplication is shown in \gyreffig{img_matrix}~followed by the pseudo-code description of tiling. \textbf{N\_track} and \textbf{N\_local} in the pseudo-code are implementation parameters of SID~that are provided to the instruction generator. When an output entry of the vector \textbf{O} cannot be computed in one cycle, its partial sum is stored in the local scratchpad memory in the EXE2 stage (see \gyreffig{img_blockdiagram}) so that it does not need to be fetched from memory every cycle. Also, every cycle, \textbf{N\_track} weights (the inner loop of \textbf{jj}) in the same row and their corresponding input elements are multiplied and added to one partial sum. The weights at the same location in the rows below (the loop of \textbf{ii}) will be used in the following cycles, so the input data is reused. The same reuse of the input data is exploited for all columns (the loop of \textbf{j}). A single \textbf{MvMul} instruction can perform all the loops except the first loop of \textbf{i}. When the local scratchpad memory cannot hold all the partial sums, the loop of \textbf{i} is performed using multiple \textbf{MvMul} instructions, each of which corresponds to an FSM in \gyreffig{img_fsm_diagram_mvmul}.
\iffalse
\begin{lstlisting}[language=C,
xleftmargin=.5em, framexleftmargin=.5em, caption={Tiling for Matrix-vector Multiplication}, label={alg_tiling}]
  // N_local: number of entries in local scratchpad
  // N_track: number of parallel computing tracks
  for (i = 0; i < l; i+= N_local)
    for (j = 0; j < w; j += N_track)
      for (ii = 0; ii < N_local; ii ++)
        // Computed every cycle
        for (jj = 0; jj < N_track; jj++)
          O[i*N_local+ii] += W[i*N_local+ii][j*N_track+jj] * I[j*N_track+jj]
\end{lstlisting}
\fi

\bheading{Adding vector-scalar comparison for statistical test.} The vector-scalar comparison mode,~\textbf{VSsgt}, is required by the statistical test support (see~\gyrefsec{sec_op_kstest}). It is computed by setting an output element to one if the corresponding element in the input vector is larger than the other scalar operand, otherwise zero. During execution, the scalar is kept in the pipeline, and the vector elements are streamed in for comparison.

\bheading{Using local scratchpad for fast accesses to intermediate results.}
Besides \textbf{Mvmul}, other operation modes, which need to store intermediate results, may also use the local scratchpad to avoid always reading them from memory outside the SID~module. The~\textbf{Vmaxabs} mode compares the absolute values of input elements in EXE2 stage by doing subtraction with the adders. A temporary maximum is stored in the local scratchpad and gets updated every cycle.
The~\textbf{Vsqnorm} mode computes the squ\-ared L2-\-norm of a vector.
The scratchpad is also used in this mode to store the partial sum of $V[i]^2$ (V is the input vector)
which are computed by the multipliers and adders in EXE1 and EXE2 stages.
\iffalse
\begin{equation}
sqnorm(V) = \sum V[i]^2
\label{eq_vsqnorm}
\end{equation}
\fi

\subsection{Nested Loop Support}
\subsection{Software-Hardware Nested Loop Support}
\label{sec_loop_support}

Although a macro instruction can perform the innermost loop operations on vector and/or matrix operands, often there are multiple nested loops that must be performed, e.g., in computing the kernel function in an SVM model for each support vector (\gyrefapd{apd_class_alg}). To avoid increasing the code size linearly with the number of iterations, we propose \textbf{SID +}, which is the SID~module with the extension to do nested loops efficiently. In \textbf{SID +}, the software program is responsible for managing the nested loop control information including the loop state and the address offsets to load/store an operand using four new instructions (\textbf{Loop}, \textbf{RegAddi}, \textbf{RegStore} and \textbf{RegLoad}).

The loop state is determined by three newly added registers, \textbf{loop\_be\-gin},~\textbf{lo\-op\_e\-nd} and~\textbf{lo\-op\_n}, which store the current loop's beginning PC, end PC and the number of unfinished iterations respectively. They are initialized when a~\textbf{Loop} instruction is executed. The hardware assumes that the loop will start after the~\textbf{Loop} instruction and sets~\textbf{loop\_end} and~\textbf{lo\-op\_n} to \textbf{inst[95:64]} and \textbf{inst[63:32]} respectively. When the program counter reaches \textbf{loop\_end}, the hardware checks~\textbf{loop\_n} and jumps back to the beginning of the loop if there are iterations that have not been finished.

We enable data accesses to different addresses by adding three address offset registers to access the operands (two input operands and one output operand) used in different iterations. The offset enable bits in the macro instruction, e.g.~\textbf{Off\_X} in~\gyreftbl{tbl_inst_fmt}, decides whether an address offset is added to the base address. To increase the address offset after each iteration, we support adding an immediate number to an address offset register using the~\textbf{RegAddi} instruction.

For a nested loop with multiple loop layers, we add two instructions,~\textbf{RegStore} and~\textbf{RegLoad}, to save and restore the loop state registers. In a nested loop, the external loop saves its loop state and address offsets before entering the inner loop and restores them after the inner loop exits.

\input{0604_loop_code_reduction.tex}
\subsection{Nested Loop Support}
\subsection{Software-Hardware Nested Loop Support}
\label{sec_loop_support}

Although a macro instruction can perform the innermost loop operations on vector and/or matrix operands, often there are multiple nested loops that must be performed, e.g., in computing the kernel function in an SVM model for each support vector (\gyrefapd{apd_class_alg}). The straightforward way to fulfill a nested loop is repeating the macro instructions and manually specifying the operands in each iteration. However, this will make the code size scale linearly with the number of iteration
. We propose \textbf{SID +}, which is the SID~module with the extension to do nested loops in a flexible and efficient way. We identify the necessary loop control information (loop states and the address offsets to load/store an operand) and provide the ability to save/restore the loop state and address offsets. In \textbf{SID +}, the software program is responsible for managing the nested loop controls, and four new instructions (\textbf{Loop}, \textbf{RegAddi}, \textbf{RegStore} and \textbf{RegLoad})

\iffalse
For instance, the inference of a Gaussian kernel based SVM requires multiple kinds of vector operations for each support vector (\gyrefsec{sec:service:alg}). The straightforward way to fulfill a nested loop is repeating the macro instructions and manually specifying the operands in each iteration
. However, this will make the code size scale linearly with the number of iteration
. We enable a flexible nested loop control by identifying the necessary loop control information (loop states and the address offsets to load/store an operand) and providing the ability to save/restore the loop state and address offsets.
\fi

The loop state is determined by three newly added registers, \textbf{loop\_be\-gin},~\textbf{lo\-op\_e\-nd} and~\textbf{lo\-op\_n}, which store the current loop's beginning PC, end PC and the number of unfinished iterations respectively. They are initialized when a~\textbf{Loop} instruction is executed. The hardware assumes that the loop will start after the~\textbf{Loop} instruction and sets~\textbf{loop\_end} and~\textbf{lo\-op\_n} to the numbers encoded in the~\textbf{Loop} instruction as shown in~\gyreflst{alg_loop_control}. The three registers then control the loop execution by deciding whether SID~needs to jump back to the beginning of the loop. When a loop reaches the end, the hardware checks~\textbf{loop\_n} and jumps back to the beginning of the loop if there are iterations that have not been finished.
\begin{lstlisting}[language=C,
xleftmargin=.5em, framexleftmargin=.5em, caption={Loop States}, label={alg_loop_control}]
    Init: Registers are set by a loop instruction
  	  // The loop starts from the next instruction
      loop_begin <- PC(loop) + 1
      loop_end   <- inst[95:64]
      loop_n     <- inst[63:32]
    Execution:
      if (PC == loop_end) && (loop_n != 0)
        // jump back to the beginning of the loop
        PC     <- loop_begin
        loop_n <- loop_n - 1
\end{lstlisting}

The address offsets are needed for incremental data access as iterations may need to access multiple operands in a row, e.g. support vectors in an SVM model. We enable this feature by adding three address offset registers to access the operands (two input operands and one output operand). The address generation logic decides whether each address offset is added to the corresponding address depending on the offset enable bits encoded in the current macro instruction, e.g.~\textbf{Off\_X} in~\gyreftbl{tbl_inst_fmt}. To increase the address offset properly after each iteration, we support adding an immediate number to an address offset register using the~\textbf{RegAddi} instruction.

For a single-layer loop, the loop control registers and address offset registers define the current loop state.
For a nested loop with multiple loop layers, we add two instructions,~\textbf{RegStore} and~\textbf{RegLoad}, to save and restore the loop state registers. In a nested loop, the external loop saves its loop state and address offsets before entering the inner loop and restores them after the inner loop exits.~\gyreflst{alg_nested_loop} shows the pseudo-code of an example two-layer loop.
\begin{lstlisting}[language=C,
xleftmargin=.5em, framexleftmargin=.5em, caption={Save/restore loop state registers in a nested loop.}, label={alg_nested_loop}]
  A two-layer loop
    loop out_begin, out_end, out_n
      some_operations()
      // Save the outer loop state before inner loop
      RegStore loop_regs,   addr_out_loop_reg
      RegStore offset_regs, addr_out_offset_reg
      loop in_begin, in_end, in_n
        some_operations()
    // Restore the outer loop state after the inner loop finishes
    RegLoad loop_regs,   addr_out_loop_reg
    RegLoad offset_regs, addr_out_offset_reg
\end{lstlisting}

\input{0604_loop_code_reduction.tex}
\subsection{Design Goals}

We want to design a detection module that can give a yes-or-no answer for impostor detection without any subsequent processing on other devices. This greatly reduces the attack surface where the detection service itself can be attacked.
\subsection{Primitives for Statistic Test}
\subsection{Support for Distribution Representation and Comparison}
\label{sec_op_kstest}

A novel contribution of this work is to show that the statistical test for comparing empirical probability distributions can be done efficiently using the multipliers and adders already needed for the ML/DL algorithms. To the best of our knowledge, we are the first to describe the following simple and efficient hardware support for comparing PEDs and the KS test.

We implement the KS test using a five-step workflow and show an example in \gyreffig{fig:ks_test}. The grey dotted boxes represent inputs, including the reference prediction error distribution (PED) and the observed PED and the output. The reference PED is collected in the training phase and is represented by reference bin boundaries and a cumulative histogram.
The observed RED is collected online and represented by a series of observed errors.

Step \ding{172}: compare an observed error with bin boundaries. The output of this step is a vector of ``0''s and ``1''s. ``1'' represents that the corresponding bin boundary is greater than the observed error, and ``0'' otherwise. This step requires a new operation for vector-scalar comparison (\textbf{VSsgt} described in~\gyrefsec{sec_parallel_operation}). Step \ding{173}: accumulate all binary vectors from \ding{172}. The accumulated vector, namely ``observed histogram'', conceptually represents the cumulative histogram of the observed errors using the reference bins. Step \ding{174} and step \ding{175}: find the largest difference in the reference and observed histograms, i.e. the KS statistic $D_{n,m}$ in \gyrefeq{equ:KSstatistic}. Step \ding{174} is a vector subtraction and step \ding{175} is a new operation, \textbf{Vmaxabs} (described in~\gyrefsec{sec_parallel_operation}), to find the maximum absolute value in a vector. Step \ding{176}: compare $D_{n,m}$ with a threshold, which is treated as a single-entry vector,
to determine normality.
Hence, we only need two extra operation modes, \textbf{VSsgt} and \textbf{Vmaxabs}, to support KS testing.

\begin{figure}[ht]
    \centering
    \includegraphics[width=\linewidth]{./figures/ks_test.pdf}
    \caption{An example of five-step KS test. Only two new operation primitives, \textbf{VSsgt} and \textbf{Vmaxabs}, are implemented in step \ding{173} and \ding{175} to support KS test.}
    \label{fig:ks_test}
\end{figure}

\section{Models for Impostor Detection}
\label{sec_sw_model}

In this section, we present the models that can be used for impostor detection. We first present the accuracy of representative multi-class models, such as MLP and SVM, and the kernel ridge regression (KRR) model proposed as the most effective detection model. We then study the accuracy when using one-class models and show that it can be compensated with a statistical test.

\subsection{Dataset and Window Generation}

\begin{figure*}
    \centering
    \includegraphics[width=\textwidth]{./figures/180722_diffstep.eps}
    \caption{Detection accuracy when different window steps are used. A sampling step of 4 leads to a larger overlap between consecutive windows and has better classification accuracy of legitimate versus non-legitimate users.}
    \label{img_diffstep}
\end{figure*}

We test the accuracy of diffehrent ML/DL models for impostor detection on the smartphone-based recognition of human activities and postural transitions (HAPT) dataset~\cite{reyes2016transition} from UCI machine learning repository~\cite{Dua:2017}. The HAPT dataset contains the sensor readings from smartphone sensors. The smartphone is worn on the waist by a group of 30 participants of various ages from 19 to 48. The participants are asked to perform six kinds of activities including standing, sitting, walking, etc. and the transitions between them. The readings are collected in the meantime and later labeled with both the activity and user ID.

The HAPT dataset is initially designed to recognize human activities but we extend it to study the feasibility to distinguish users given that the activity is already recognized. We use the WALK subset in HAPT whose data has the same activity ID (WALK) to perform user classification. The raw inputs of each user are several long \textbf{sequences} of readings collected when the participants walked for several times. Each reading contains the 3-axial measurements of both the linear acceleration and angular velocity so it could be treated as a 6-element vector.

As some ML or DL models like MLP and SVM require an input of fixed size, we sample \textbf{windows} of the same length from the \textbf{sequences}. There are two considerations when we choose the window size. It is firstly desirable to recognize a user based on a window covering one or two full steps of a typical person. Also, we obtain the frequency domain information by doing FFT so choosing a power-of-2 window is a straightforward method. Hence, knowing that sensors are sampled at a rate of 50Hz, we choose a window size of 64 readings.

However, consecutively sampling the windows with little-overlap can result in low recognition accuracy.~\gyreffig{img_diffstep} gives an intuitive heat-map of detection accuracy when the training and testing windows are sampled at a step of 4 readings and 32 readings. The model used is a single hidden layer MLP with 100 hidden neurons. Each block in the heat-map with a coordinate of (RID, TID) is the recognition accuracy when the model trained for the user labeled RID is used to recognize the testing data produced by the user labeled TID. As described by the color-bar, a darker block in the heat-map represents a lower recognition accuracy while a pure white block means all samples are classified correctly. The low-overlap sampling step of 32 is shown to generally have a worse accuracy. This is partly because a large step when collecting training windows can let the model only learn the features of a window which starts at the same phase within a step. If there is a misalignment between a testing window and windows in the training set with respect to the period of a step, a testing window is unlikely to be classified correctly.

To eliminate the problem, we produce fine-grained windows by using a smaller step when sampling the windows from the long sequences. The step is finally chosen to be 4 readings, which corresponds to an offset of 0.08s between consecutive windows and two consecutive windows have an overlap of 93.75\%.

We then need to separate the windows into a training set and a testing set. One thing to note here is that it is not reasonable to randomly pick windows from all the sequences. Due to the high overlap between consecutive windows from the same sequence, a random pick can result in high similarity between some training windows and testing windows, which makes the classification accuracy not representative. Instead, we claim that the windows in the training set and the testing set must be from different sequences. Therefore, the sequences are first labeled as training sequences or testing sequences and then the windows are divided according to the sequence they are from.

\subsection{Models Trained with Multi-class Data}

We investigate the ability of multi-class models trained to detect whether the current (test) user is not the correct user. We consider the case in real life when we have no knowledge of potential attackers. To emulate this situation, 25 out of the 30 users in the HAPT dataset are selected as registered users while the other 5 users act as unregistered users. For each registered user, models are trained using his/her data and some randomly picked sensor data of the other 24 registered users. In the training process, all the data from other 24 registered users are labeled as 'not being the correct user'. This is a method of labeling data in real application without losing the privacy of other registered users. The data of unregistered users is only used in testing to examine whether previously unseen attackers can be detected.

\begin{table}[ht]
\centering
\resizebox{0.48\textwidth}{!}{
\begin{tabular}{|l|l|l|l|l|l|l|}
\hline
\textbf{}                                                           & \multicolumn{6}{c|}{\textbf{All Users}}                                                             \\ \hline
\textbf{}                                                           & \textbf{TNR}   & \textbf{TPR}   & \textbf{Accuracy} & \textbf{R}    & \textbf{P}    & \textbf{F1}   \\ \hline
\textbf{\begin{tabular}[c]{@{}l@{}}Ridge\\ Regression\end{tabular}} & \textbf{80.77} & \textbf{66.05} & \textbf{73.41}    & \textbf{0.66} & \textbf{0.77} & \textbf{0.69} \\ \hline
\textbf{MLP-20}                                                     & \textbf{97.42} & \textbf{88.79} & \textbf{93.10}    & \textbf{0.89} & \textbf{0.96} & \textbf{0.91} \\ \hline
\textbf{MLP-50}                                                     & \textbf{98.31} & \textbf{92.70} & \textbf{95.51}    & \textbf{0.93} & \textbf{0.98} & \textbf{0.94} \\ \hline
\textbf{MLP-100}                                                    & \textbf{98.60} & \textbf{94.65} & \textbf{96.63}    & \textbf{0.95} & \textbf{0.98} & \textbf{0.96} \\ \hline
\textbf{MLP-200}                                                    & \textbf{98.41} & \textbf{95.72} & \textbf{97.06}    & \textbf{0.96} & \textbf{0.98} & \textbf{0.97} \\ \hline
\textbf{MLP-500}                                                    & \textbf{98.68} & \textbf{95.47} & \textbf{97.07}    & \textbf{0.95} & \textbf{0.99} & \textbf{0.96} \\ \hline
\textbf{MLP-50-25}                                                  & \textbf{98.13} & \textbf{94.49} & \textbf{96.31}    & \textbf{0.94} & \textbf{0.98} & \textbf{0.96} \\ \hline
\textbf{MLP-100-50}                                                 & \textbf{98.44} & \textbf{95.45} & \textbf{96.95}    & \textbf{0.95} & \textbf{0.98} & \textbf{0.96} \\ \hline
\textbf{MLP-200-100}                                                & \textbf{98.47} & \textbf{95.72} & \textbf{97.10}    & \textbf{0.96} & \textbf{0.98} & \textbf{0.97} \\ \hline
\textbf{SVM}                                                        & \textbf{99.26} & \textbf{97.57} & \textbf{98.42}    & \textbf{0.98} & \textbf{0.99} & \textbf{0.98} \\ \hline
\textbf{KRR}                                                        & \textbf{88.91} & \textbf{82.66} & \textbf{85.78}    & \textbf{0.83} & \textbf{0.87} & \textbf{0.83} \\ \hline
\end{tabular}
}
\caption{Recognition accuracy of binary classification models}
\label{tbl_accuracy_binary}
\end{table}

We present the accuracy of binary classification models in~\gyreftbl{tbl_accuracy_binary} where each row corresponds to one model. The results shows that SVM performs best for all the metrics. Among the single-layer MLP models, the single hidden layer with 500 hidden neurons has the best average accuracy and increasing the number of hidden layers can slightly improve the average accuracy. The KRR method requiring manual feature extraction performs much better than the ridge regression model on raw data but has a lower accuracy than MLP and SVM based models.

\iffalse
\bheading{Regression}

\bheading{Multilayer Perceptron}

\bheading{Support Vector Machine}
\fi

\subsection{One-class Models for Impostor Detection}

While the previous models for classification always need other users' data for training, it is desirable for a user to have a one-class model trained with only his/her own data.
To train multi-class models, the detection provider needs users' consent to use their data to train the models for other people. This requirement raises privacy concerns for users and has a high probability to be rejected. The one-class models can avoid asking for the data of other users.
From the security perspective, the training with data from certain users can be unreliable. It is non-trivial to predict how an unregistered anomalous user will differ from registered users. It is hence hard to guarantee that a multi-class model can learn all the patterns of anomalous users using only registered users' data in the training.

We test two one-class models, the one-class SVM and a LSTM-based model for time-series modeling. The one-class SVM has an average accuracy of 69.22\% which is about thirty percent worse than the best multi-class models. The LSTM-based model makes predictions for every sensor reading in the sample window. The classification result is given by comparing the average prediction error with a threshold obtain from the training set. The LSTM-based models have an accuracy about 70 percent for both real user and wrong users, which is similar to the one-class SVM model. It is also found that the accuracy doesn't improve much when the number of hidden nodes is increased.

\begin{table}[ht]
\centering
\resizebox{0.48\textwidth}{!}{
\begin{tabular}{|l|l|l|l|l|l|l|}
\hline
\textbf{}                                                             & \multicolumn{6}{c|}{\textbf{All Users}}                                                             \\ \hline
\textbf{}                                                             & \textbf{TNR}   & \textbf{TPR}   & \textbf{Accuracy} & \textbf{R}    & \textbf{P}    & \textbf{F1}   \\ \hline
\textbf{OCSVM}                                                        & \textbf{64.24} & \textbf{74.19} & \textbf{69.22}    & \textbf{0.74} & \textbf{0.59} & \textbf{0.65} \\ \hline
\textbf{LSTM-20}                                                      & \textbf{70.53} & \textbf{64.96} & \textbf{67.75}    & \textbf{0.65} & \textbf{0.54} & \textbf{0.57} \\ \hline
\textbf{LSTM-50}                                                      & \textbf{72.43} & \textbf{67.04} & \textbf{69.74}    & \textbf{0.67} & \textbf{0.57} & \textbf{0.60} \\ \hline
\textbf{LSTM-100}                                                     & \textbf{72.20} & \textbf{69.27} & \textbf{70.73}    & \textbf{0.69} & \textbf{0.58} & \textbf{0.62} \\ \hline
\textbf{LSTM-200}                                                     & \textbf{67.88} & \textbf{71.60} & \textbf{69.74}    & \textbf{0.72} & \textbf{0.58} & \textbf{0.62} \\ \hline
\textbf{LSTM-500}                                                     & \textbf{67.57} & \textbf{74.42} & \textbf{70.99}    & \textbf{0.74} & \textbf{0.60} & \textbf{0.65} \\ \hline \hline
\textbf{\begin{tabular}[c]{@{}l@{}}LSTM-200-\\ KS-OCSVM\end{tabular}} & \textbf{74.82} & \textbf{92.25} & \textbf{83.53}    & \textbf{0.92} & \textbf{0.80} & \textbf{0.84} \\ \hline
\textbf{\begin{tabular}[c]{@{}l@{}}LSTM-200-\\ KS-Vote\end{tabular}}  & \textbf{94.62} & \textbf{83.31} & \textbf{88.97}    & \textbf{0.83} & \textbf{0.91} & \textbf{0.83} \\ \hline
\end{tabular}
}
\caption{Recognition accuracy of one-class models}
\label{tbl_accuracy_oneclass}
\end{table}

\iffalse
\begin{table}[]
\begin{tabular}{|l|l|l|l|}
\hline
\multicolumn{1}{|c|}{\multirow{2}{*}{\begin{tabular}[c]{@{}c@{}}Real\\ User ID\end{tabular}}} & \multicolumn{3}{c|}{OC-SVM}                                                         \\ \cline{2-4}
\multicolumn{1}{|c|}{}                                                                        & \multicolumn{1}{c|}{Avg} & \multicolumn{1}{c|}{Min} & \multicolumn{1}{c|}{Min User} \\ \hline
1                                                                                & 53.58                    & 0                        & 3                             \\ \hline
2                                                                                & 96.65                    & 22.22                    & 2                             \\ \hline
3                                                                                & 97                       & 30.91                    & 18                            \\ \hline
4                                                                                & 98.6                     & 58.85                    & 4                             \\ \hline
5                                                                                & 86.31                    & 0                        & 4                             \\ \hline
6                                                                                & 72.83                    & 0                        & 4                             \\ \hline
7                                                                                & 50.16                    & 0                        & 3                             \\ \hline
8                                                                                & 29.55                    & 0                        & 3                             \\ \hline
9                                                                                & 83.54                    & 0                        & 16                            \\ \hline
10                                                                                & 65.4                     & 0                        & 4                             \\ \hline
11                                                                                & 96.79                    & 40.08                    & 11                            \\ \hline
12                                                                                & 86.07                    & 9.19                     & 25                            \\ \hline
13                                                                                & 45.75                    & 0                        & 3                             \\ \hline
14                                                                                & 95.62                    & 1.36                     & 18                            \\ \hline
15                                                                                & 88.54                    & 0                        & 4                             \\ \hline
16                                                                                & 98.28                    & 70.45                    & 18                            \\ \hline
17                                                                                & 84.34                    & 0                        & 18                            \\ \hline
18                                                                                & 98.93                    & 71.36                    & 18                            \\ \hline
19                                                                                & 17.43                    & 0                        & 1                             \\ \hline
20                                                                                & 25.66                    & 0                        & 1                             \\ \hline
21                                                                                & 84.1                     & 0                        & 3                             \\ \hline
22                                                                                & 49.66                    & 0                        & 3                             \\ \hline
23                                                                                & 45.37                    & 0                        & 3                             \\ \hline
24                                                                                & 98.45                    & 60.85                    & 24                            \\ \hline
25                                                                                & 97.91                    & 40.28                    & 25                            \\ \hline
Avg                                                                                & 68.39                    & 15.02                    &                               \\ \hline
\end{tabular}
\end{table}

\fi
\subsection{Introduction of Frequency Information}

Previous work~\cite{lee2017implicit} used statistical methods to find the most valuable features in both time and frequency domains and demonstrated an improvement on recognition accuracy.
We study if the frequency domain information can be introduced to improve the accuracy of one-class models
The frequency domain features can be computed through a discrete Fourier transform (DFT) as~\gyrefeq{eq_freq_amp}. The complex frequency components are presented as two arrays of numbers which account for the real part and the imaginary part respectively. The squared amplitudes of all the frequency components could be computed by our processing module and used as additional feature afterwards. Different from~\cite{lee2017implicit} which further manually extracts the frequency domain features such as the most significant components, we expect the DL models can automatically learn useful features from the frequency domain amplitudes.

\bheading{Binary Classification Models} We compare the accuracy to recognize a given window using only time domain features and using both time and frequency domain features. The time domain features are the sums of the squared amplitudes of accelerometer and gyroscope readings as expressed in~\gyrefeq{eq_time_amp}. In a 64-point window, we could get 64 amplitude features from the accelerometer and another 64 features from the gyroscope.
\begin{equation}
\label{eq_time_amp}
\begin{split}
acc_{amp}[i]& = acc_x[i]^2 + acc_y[i]^2 + acc_z[i]^2 \\
gyr_{amp}[i]& = gyr_x[i]^2 + gyr_y[i]^2 + gyr_z[i]^2 \\
&i = 0, 1,..., window\_len-1
\end{split}
\end{equation}
The frequency features are the amplitudes of frequency components of the current window.~\gyrefeq{eq_freq_amp} describes how the features are obtained. $e^{\frac{-i2\pi kj}{window\_len}}$ terms are constants that can be computed before the inference. $acc/gyr_f[k]$ are complex numbers produced by the DFT procedure. We only captures their amplitude information by computing their norms.

There are originally 64 such features from both the accelerometer and the gyroscope for a 64-point window. However, since ${acc/gyr}_{amp}[j]$ are all real numbers, ${acc/gyr}_{f}[k], k = 1,...,31$ can be proved~\cite{wiki:dft} to be the complex conjugate of $acc/gyr_{f}[64-k]$ and their norms are the same. Therefore, only the first half of $acc/gyr_{famp}[k], k = 0,1,...,32$ (33 components in total for each sensor) needs to be saved as input features.
\begin{equation}
\label{eq_freq_amp}
\begin{split}
acc_{f}[k]   & = \textstyle \sum_{j=0}^{window\_len-1} acc_{amp}[j] \times e^{\frac{-i2\pi kj}{window\_len}} \\
&(e^{\frac{-i2\pi kj}{window\_len}} = \cos \tfrac{-2\pi kj}{window\_len} + i \sin \tfrac{-2\pi kj}{window\_len}) \\
acc_{ampf}[k]& = \norm{acc_{f}[k]} = \textstyle \sqrt{Re(acc_{f}[k])^2 + Im(acc_{f}[k])^2} \\
gyr_{f}[k]   & = \textstyle \sum_{j=0}^{window\_len-1} gyr_{amp}[j] \times e^{\frac{-i2\pi kj}{window\_len}} \\
gyr_{ampf}[k]& = \norm{gyr_{f}[k]} = \textstyle \sqrt{Re(gyr_{f}[k])^2 + Im(gyr_{f}[k])^2} \\
&k = 0,1,..., window\_len-1
\end{split}
\end{equation}
\gyreftbl{tbl_freq_binary} shows the impact of adding frequency-domain features on the recognition accuracy. The features increase the best average accuracy of LR, MLP and SVM models by 21.17\%, 3.78\% and 4.90\% respectively. A more significant improvement can be found for the users having minimum recognition accuracy. The accuracy of the worst recognized user is increased by 30.03\%, 6.63\% and 14.71\% for the three models.
\begin{table}[h]
\centering
\resizebox{\columnwidth}{!}{
\begin{tabular}{|l|l|l|l|l|l|l|l|}
\hline
Model                                                      & Avg   & Min   & Min User & Model                                                          & Avg   & Min   & Min User \\ \hline
\begin{tabular}[c]{@{}l@{}}Amp-\\ LR\end{tabular}          & 66.4  & 46.52 & 6        & \begin{tabular}[c]{@{}l@{}}Amp-FFT-\\ LR\end{tabular}          & 87.57 & 76.55 & 10       \\ \hline
\begin{tabular}[c]{@{}l@{}}Amp-\\ MLP-10\end{tabular}      & 75.15 & 50.9  & 6        & \begin{tabular}[c]{@{}l@{}}Amp-FFT-\\ MLP-10\end{tabular}      & 91.52 & 79.69 & 6        \\ \hline
\begin{tabular}[c]{@{}l@{}}Amp-\\ MLP-20\end{tabular}      & 79.56 & 60.28 & 13       & \begin{tabular}[c]{@{}l@{}}Amp-FFT-\\ MLP-20\end{tabular}      & 92.4  & 81.47 & 6        \\ \hline
\begin{tabular}[c]{@{}l@{}}Amp-\\ MLP-50\end{tabular}      & 84.07 & 61.79 & 13       & \begin{tabular}[c]{@{}l@{}}Amp-FFT-\\ MLP-50\end{tabular}      & 93.16 & 82.21 & 6        \\ \hline
\begin{tabular}[c]{@{}l@{}}Amp-\\ MLP-100\end{tabular}     & 86.89 & 68.54 & 13       & \begin{tabular}[c]{@{}l@{}}Amp-FFT-\\ MLP-100\end{tabular}     & 93.29 & 83.04 & 6        \\ \hline
\begin{tabular}[c]{@{}l@{}}Amp-\\ MLP-200\end{tabular}     & 87.87 & 66.67 & 6        & \begin{tabular}[c]{@{}l@{}}Amp-FFT-\\ MLP-200\end{tabular}     & 93.16 & 82    & 6        \\ \hline
\begin{tabular}[c]{@{}l@{}}Amp-\\ MLP-500\end{tabular}     & 88.45 & 69.31 & 13       & \begin{tabular}[c]{@{}l@{}}Amp-FFT-\\ MLP-500\end{tabular}     & 92.92 & 80.84 & 6        \\ \hline
\begin{tabular}[c]{@{}l@{}}Amp-\\ MLP-1000\end{tabular}    & 87.99 & 71.07 & 1        & \begin{tabular}[c]{@{}l@{}}Amp-FFT-\\ MLP-1000\end{tabular}    & 92.25 & 80.25 & 6        \\ \hline
\begin{tabular}[c]{@{}l@{}}Amp-\\ MLP-50-25\end{tabular}   & 86.5  & 66.3  & 5        & \begin{tabular}[c]{@{}l@{}}Amp-FFT-\\ MLP-50-25\end{tabular}   & 94.1  & 85.96 & 6        \\ \hline
\begin{tabular}[c]{@{}l@{}}Amp-\\ MLP-100-50\end{tabular}  & 89.3  & 72.3  & 13       & \begin{tabular}[c]{@{}l@{}}Amp-FFT-\\ MLP-100-50\end{tabular}  & 94.38 & 84.17 & 6        \\ \hline
\begin{tabular}[c]{@{}l@{}}Amp-\\ MLP-200-100\end{tabular} & 90.6  & 79.33 & 6        & \begin{tabular}[c]{@{}l@{}}Amp-FFT-\\ MLP-200-100\end{tabular} & 94.37 & 85.54 & 1        \\ \hline
\begin{tabular}[c]{@{}l@{}}Amp-\\ SVM\end{tabular}         & 90.44 & 71.5  & 5        & \begin{tabular}[c]{@{}l@{}}Amp-FFT-\\ SVM\end{tabular}         & 95.34 & 86.21 & 5        \\ \hline
\end{tabular}
}
\caption{Improving binary classification models with frequency-domain features}
\label{tbl_freq_binary}
\end{table}

\bheading{One-class models}
We study whether frequency-domain information will improve the accuracy of one-class models. The one-class models using only time-domain amplitude features in the left part of~\gyreftbl{tbl_freq_oc} have low average recognition accuracy for all the users. Moreover, for the user being worst recognized, the accuracy stays around 10 percent even if the size of LSTM model is increased. Hence one-class models using only time-domain features are not acceptable security solutions to protect the smartphone.

On the other hand, including frequency-domain features obtained is shown to make one-class models usable. For the one-class SVM (OCSVM) model, we include the frequency features of a window which is computed according to~\gyrefeq{eq_freq_amp}. The frequency domain information increases the OCSVM model's average accuracy by 29.81\% and worst-case accuracy from 7.42\% to 53.65\%. While the worst-case accuracy of OCSVM is still not satisfactory for user 6, the LSTM-based DL model benefits more from the extra frequency features. For the LSTM-based model, we regenerate the data so that each reading contains not only the amplitude features but also the 64-point DFT result of the latest 64 readings. As shown in the right part of~\gyreftbl{tbl_freq_oc}, the average accuracy of LSTM-based model can be increased to 94.25\%. Even in the worst case, the user can still be protected with an accuracy of more than 80 percent.
\begin{table}[h]
\centering
\resizebox{\columnwidth}{!}{
\begin{tabular}{|l|l|l|l|l|l|l|l|}
\hline
Model                                                   & Avg   & Min   & Min User & Model                                                       & Avg   & Min   & Min User \\ \hline
\begin{tabular}[c]{@{}l@{}}Amp-\\ OCSVM\end{tabular}    & 55.87 & 7.42  & 19       & \begin{tabular}[c]{@{}l@{}}Amp-FFT-\\ OCSVM\end{tabular}    & 85.68 & 53.65 & 6        \\ \hline
\begin{tabular}[c]{@{}l@{}}Amp-\\ LSTM-10\end{tabular}  & 61.67 & 4.28  & 19       & \begin{tabular}[c]{@{}l@{}}Amp-FFT-\\ LSTM-10\end{tabular}  & 65.68 & 12.36 & 19       \\ \hline
\begin{tabular}[c]{@{}l@{}}Amp-\\ LSTM-20\end{tabular}  & 61.08 & 4.58  & 19       & \begin{tabular}[c]{@{}l@{}}Amp-FFT-\\ LSTM-20\end{tabular}  & 70.63 & 18.96 & 19       \\ \hline
\begin{tabular}[c]{@{}l@{}}Amp-\\ LSTM-50\end{tabular}  & 61.58 & 7.61  & 20       & \begin{tabular}[c]{@{}l@{}}Amp-FFT-\\ LSTM-50\end{tabular}  & 77.63 & 28.62 & 23       \\ \hline
\begin{tabular}[c]{@{}l@{}}Amp-\\ LSTM-100\end{tabular} & 67.52 & 10.59 & 19       & \begin{tabular}[c]{@{}l@{}}Amp-FFT-\\ LSTM-100\end{tabular} & 86.54 & 52.34 & 1        \\ \hline
\begin{tabular}[c]{@{}l@{}}Amp-\\ LSTM-200\end{tabular} & 67.74 & 8.4   & 19       & \begin{tabular}[c]{@{}l@{}}Amp-FFT-\\ LSTM-200\end{tabular} & 94.25 & 80.25 & 7        \\ \hline
\begin{tabular}[c]{@{}l@{}}Amp-\\ LSTM-500\end{tabular} & 68.35 & 7.45  & 19       & \begin{tabular}[c]{@{}l@{}}Amp-FFT-\\ LSTM-500\end{tabular} & 92.96 & 80.49 & 7        \\ \hline
\end{tabular}
}
\caption{Improving one-class models with frequency-domain features}
\label{tbl_freq_oc}
\end{table}
\subsection{Enhance one-class models with KS Test}
\label{sec_ks_test}

~\gyreftbl{tbl_accuracy_oneclass} indicates that characterizing the user behaviors with the average of sensor prediction errors is not enough. We opt to assume that normal sensor samples can feature one of several reference patterns of the real user. To be more specific, a sample is considered to be from a certain user if the distribution of its sensor prediction errors is similar to at least one such distribution in that user's training set.

We measure the difference between two error distributions with Kolmogorov-Smirnov test (KS test). KS test finds the maximum difference (also termed KS-test-statistic) between two empirical distribution function, $F_1(x)$ and $F_2(x)$, as shown in~\gyrefeq{eq_kstest_d}.

\begin{equation}
\centering
  D(F_1, F_2) = \sup_x \norm{F_1(x) - F_2(x)}
\label{eq_kstest_d}
\end{equation}

We classify a testing sample by doing a majority vote using the KS-test-statistic. We first randomly choose 20 samples of prediction errors from the training set and use them to represent the reference distributions. In the testing phase, the error distribution of each testing sample is compared to all the reference distributions. If half of the KS-test-statistic's are larger than a commonly used threshold, the current sample is considered from an attacker. This method corresponds to the row of \textbf{LSTM-200-KS-Vote} in~\gyreftbl{tbl_accuracy_oneclass} and gives an average accuracy of 88.97\%, which gets much closer to the accuracy of multi-class models. This voting-based method achieves a high TNR of 94.62\% but a lower TPR of 83.31\%, which means it is a loose model providing good usability for the real user but limited security against impostors.

We also characterize the cluster of reference distributions and then recognize outliers as attackers. A one-class SVM model is applied on top of the KS-test-statistic's. While still using 20 reference samples, we now represent each sample as $(D_1, D_2,..., D_{20})$, which are its KS-test-statistic's with 20 reference samples. We train a one-class SVM model using the features of training samples. The features of testing sample are fed into the one-class SVM model to obtain a binary classification result. This method corresponds to the row of \textbf{LSTM-200-KS-OCSVM} in~\gyreftbl{tbl_accuracy_oneclass} and gives an average accuracy of 83.53\%. Using one-class SVM achieves a high TPR of 92.25\% but a lower TPR of 74.82\%, which means it is a strict model providing good security against impostors but can erroneously reject the real user.
\section{Evaluation}
\label{sec_evaluation}

\input{0602_hw_stats.tex}

\input{0603_hw_util.tex}

\input{0605_energy_savings.tex}
\subsection{Security against Unregistered Impostors}
\label{sec_security_analysis}

\begin{table}[ht]
\centering
\resizebox{\columnwidth}{!}{
\begin{tabular}{|l|l|l|l|}
\hline
\textbf{}                                                             & \textbf{TPR(Registered)} & \textbf{TPR(Unregistered)} & \textbf{TPR Difference} \\ \hline
\textbf{MLP-50}                                                       & \textbf{92.41}           & \textbf{94.09}             & \textbf{1.68}           \\ \hline
\textbf{MLP-500}                                                      & \textbf{95.64}           & \textbf{94.65}             & \textbf{-0.99}          \\ \hline
\textbf{MLP-50-25}                                                    & \textbf{91.00}           & \textbf{92.52}             & \textbf{1.53}           \\ \hline
\textbf{MLP-200-100}                                                  & \textbf{92.84}           & \textbf{93.62}             & \textbf{0.78}           \\ \hline
\textbf{SVM}                                                          & \textbf{97.53}           & \textbf{97.73}             & \textbf{0.20}           \\ \hline
\textbf{KRR}                                                          & \textbf{82.60}           & \textbf{82.92}             & \textbf{0.32}           \\ \hline
\textbf{\begin{tabular}[c]{@{}l@{}}LSTM-200-\\ KS-OCSVM\end{tabular}} & \textbf{92.21}           & \textbf{92.43}             & \textbf{0.22}           \\ \hline
\textbf{\begin{tabular}[c]{@{}l@{}}LSTM-200-\\ KS-Vote\end{tabular}}  & \textbf{83.39}           & \textbf{82.94}             & \textbf{-0.45}          \\ \hline
\end{tabular}
}
\caption{Recognition accuracy of registered and unregistered users}
\label{tbl_reg_unreg}
\end{table}

We verify the applicability of models when potential impostors are likely to be unregistered users. For each registered user, the other 24 registered users are used as impostors whose data are available in the training dataset. The 5 users whose data is not used during training are considered unforeseen impostor.

We compare the average accuracy to recognize registered users and unregistered users in~\gyreftbl{tbl_reg_unreg}. The accuracy is the TPR in these experiments as it is evaluated only on other users' data. The result shows that the models trained with only registered users' data can detect unregistered users at a similar accuracy. The difference of TPR between registered users and unregistered users given in the last column show that most models can even detect unregistered users slightly more accurately than registered users.

\iffalse
\begin{table*}[h!]
\centering
\begin{tabular}{|l|l|l|l|l|l|l|l|l|l|l|l|}
\hline
\textbf{}                                                             & \multicolumn{1}{c|}{\textbf{Real User}} & \multicolumn{5}{l|}{\textbf{Other Registered Users}}                               & \multicolumn{5}{l|}{\textbf{Unregistered Users}}                                   \\ \hline
\textbf{}                                                             & \textbf{TNR}                            & \textbf{TPR}   & \textbf{Accuracy} & \textbf{R}    & \textbf{P}    & \textbf{F1}   & \textbf{TPR}   & \textbf{Accuracy} & \textbf{R}    & \textbf{P}    & \textbf{F1}   \\ \hline
\textbf{MLP-50}                                                       & \textbf{98.31}                          & \textbf{92.41} & \textbf{95.36}    & \textbf{0.92} & \textbf{0.98} & \textbf{0.94} & \textbf{94.09} & \textbf{96.20}    & \textbf{0.94} & \textbf{0.98} & \textbf{0.95} \\ \hline
\textbf{MLP-500}                                                      & \textbf{98.68}                          & \textbf{95.64} & \textbf{97.16}    & \textbf{0.96} & \textbf{0.99} & \textbf{0.97} & \textbf{94.65} & \textbf{96.66}    & \textbf{0.95} & \textbf{0.99} & \textbf{0.96} \\ \hline
\textbf{MLP-50-25}                                                    & \textbf{96.05}                          & \textbf{91.00} & \textbf{93.52}    & \textbf{0.91} & \textbf{0.95} & \textbf{0.92} & \textbf{92.52} & \textbf{94.29}    & \textbf{0.93} & \textbf{0.96} & \textbf{0.94} \\ \hline
\textbf{MLP-200-100}                                                  & \textbf{97.90}                          & \textbf{92.84} & \textbf{95.37}    & \textbf{0.93} & \textbf{0.98} & \textbf{0.94} & \textbf{93.62} & \textbf{95.76}    & \textbf{0.94} & \textbf{0.98} & \textbf{0.95} \\ \hline
\textbf{SVM}                                                          & \textbf{99.26}                          & \textbf{97.53} & \textbf{98.40}    & \textbf{0.98} & \textbf{0.99} & \textbf{0.98} & \textbf{97.73} & \textbf{98.50}    & \textbf{0.98} & \textbf{0.99} & \textbf{0.98} \\ \hline
\textbf{KRR}                                                          & \textbf{88.91}                          & \textbf{82.60} & \textbf{85.75}    & \textbf{0.83} & \textbf{0.87} & \textbf{0.83} & \textbf{82.92} & \textbf{85.91}    & \textbf{0.83} & \textbf{0.87} & \textbf{0.83} \\ \hline \hline
\textbf{\begin{tabular}[c]{@{}l@{}}LSTM-200-\\ KS-OCSVM\end{tabular}} & \textbf{74.82}                          & \textbf{92.21} & \textbf{83.51}    & \textbf{0.92} & \textbf{0.80} & \textbf{0.84} & \textbf{92.43} & \textbf{83.63}    & \textbf{0.92} & \textbf{0.81} & \textbf{0.84} \\ \hline
\textbf{\begin{tabular}[c]{@{}l@{}}LSTM-200-\\ KS-Vote\end{tabular}}  & \textbf{94.62}                          & \textbf{83.39} & \textbf{89.00}    & \textbf{0.83} & \textbf{0.92} & \textbf{0.83} & \textbf{82.94} & \textbf{88.78}    & \textbf{0.83} & \textbf{0.88} & \textbf{0.83} \\ \hline
\end{tabular}
\caption{Recognition accuracy of registered and unregistered users}
\label{tbl_reg_unreg}
\end{table*}
\fi
\subsection{Combined Metrics}
\label{sec_combined_metrics}

In this section, we propose to use a single metric for users to choose their models for detection with the tradeoff between accuracy, performance and memory usage. In most use cases, users don't need to be aware of the details of the model that can be applied but they can specify the key factor they care about. For instance, some users may want to get the highest accuracy without caring about the memory usage or performance. Some other users may want to save the memory usage while maintain a reasonable accuracy.

We provide a generalized decision method in~\gyrefeq{eq_combined_metric} to consider all the metrics. Each model in a certain scope has a penalty for a higher error rate, a longer execution time or a larger memory usage. To make fair comparison, we identify the minimum error rate, the minimum execution time and the minimal memory usage in the set of available models. The penalty in each dimension is normalized to its corresponding minimum in the model set. The combined metric is computed as a weighted sum of three normalize penalties. The weights can be controlled by the user based on the user's preference.
\begin{equation}
\begin{split}

    &Score(Model_i) = \\
    &\frac{2}{\begin{split}
                            &a1 \times NormFNR_i +
                             a2 \times NormFPR_i + \\
                            &b1 \times NormTime_i +
                              b2 \times NormMem_i\end{split}}
                              \\
                              \\
    &(a1 + a2 + b1 + b2 = 2) \\

        &~NormFNR_i =  \frac{FNR_i}{Min(FNR)} \\
        &~NormFPR_i =  \frac{FNR_i}{Min(FPR)} \\
        &NormTime_i =  \frac{Time_i}{Min(Time)} \\
        &~NormMem_i =  \frac{MemUsage_i}{Min(MemUsage)}
\end{split}

\label{eq_combined_metric}
\end{equation}

\begin{table*}[h]
\centering
\begin{tabular}{|l|l|l|l|l|l|l|}
\hline
\multicolumn{1}{|c|}{\begin{tabular}[c]{@{}c@{}}Weights\\ (a1,a2,b1,b2)\end{tabular}}                                    & \multicolumn{1}{c|}{Model Name} & \multicolumn{1}{c|}{\begin{tabular}[c]{@{}c@{}}Normalized\\ FNR\\ (security)\end{tabular}} & \multicolumn{1}{c|}{\begin{tabular}[c]{@{}c@{}}Normalized\\ FPR\\ (usability)\end{tabular}} & \multicolumn{1}{c|}{\begin{tabular}[c]{@{}c@{}}Normalized\\ Execution\\ Time\end{tabular}} & \multicolumn{1}{c|}{\begin{tabular}[c]{@{}c@{}}Normalized\\ Memory\\ Usage\end{tabular}} & \multicolumn{1}{c|}{\begin{tabular}[c]{@{}c@{}}Combined\\ Metric\end{tabular}} \\ \hline
\multirow{5}{*}{\begin{tabular}[c]{@{}l@{}}(0.5, 0.5, 0.5, 0.5)\\ Equal weights\end{tabular}}                            & MLP-50                          & 3.00                                                                                & 2.29                                                                                & 1.00                                                                                & 1.00                                                                                & 0.55                                                                           \\ \cline{2-7}
                                                                                & MLP-500                         & 1.86                                                                                & 1.80                                                                                & 9.98                                                                                & 10.00                                                                                & 0.17                                                                           \\ \cline{2-7}
                                                                                & MLP-50-25                       & 3.59                                                                                & 5.38                                                                                & 1.07                                                                                & 1.07                                                                                & 0.36                                                                           \\ \cline{2-7}
                                                                                & MLP-200-100                     & 2.89                                                                                & 2.86                                                                                & 5.03                                                                                & 5.04                                                                                & 0.25                                                                           \\ \cline{2-7}
                                                                                & SVM                             & 1.00                                                                                & 1.00                                                                                & 37.05                                                                                & 17.34                                                                                & 0.07                                                                           \\ \hline
\multirow{5}{*}{\begin{tabular}[c]{@{}l@{}}(0.9, 0.1, 0.5, 0.5)\\ FNR critical\end{tabular}}                             & MLP-50                          & 3.00                                                                                & 2.29                                                                                & 1.00                                                                                & 1.00                                                                                & 0.51                                                                           \\ \cline{2-7}
                                                                                & MLP-500                         & 1.86                                                                                & 1.80                                                                                & 9.98                                                                                & 10.00                                                                                & 0.17                                                                           \\ \cline{2-7}
                                                                                & MLP-50-25                       & 3.59                                                                                & 5.38                                                                                & 1.07                                                                                & 1.07                                                                                & 0.41                                                                           \\ \cline{2-7}
                                                                                & MLP-200-100                     & 2.89                                                                                & 2.86                                                                                & 5.03                                                                                & 5.04                                                                                & 0.25                                                                           \\ \cline{2-7}
                                                                                & SVM                             & 1.00                                                                                & 1.00                                                                                & 37.05                                                                                & 17.34                                                                                & 0.07                                                                           \\ \hline
\multirow{5}{*}{\begin{tabular}[c]{@{}l@{}}(0.9, 0.1, 0.9, 0.1)\\ FNR critical\\ Execution time\\ critical\end{tabular}} & MLP-50                          & 3.00                                                                                & 2.29                                                                                & 1.00                                                                                & 1.00                                                                                & 0.51                                                                           \\ \cline{2-7}
                                                                                & MLP-500                         & 1.86                                                                                & 1.80                                                                                & 9.98                                                                                & 10.00                                                                                & 0.17                                                                           \\ \cline{2-7}
                                                                                & MLP-50-25                       & 3.59                                                                                & 5.38                                                                                & 1.07                                                                                & 1.07                                                                                & 0.41                                                                           \\ \cline{2-7}
                                                                                & MLP-200-100                     & 2.89                                                                                & 2.86                                                                                & 5.03                                                                                & 5.04                                                                                & 0.25                                                                           \\ \cline{2-7}
                                                                                & SVM                             & 1.00                                                                                & 1.00                                                                                & 37.05                                                                                & 17.34                                                                                & 0.06                                                                           \\ \hline
\multirow{5}{*}{\begin{tabular}[c]{@{}l@{}}(0.1, 0.9, 0.5, 0.5)\\ FPR critical\end{tabular}}                             & MLP-50                          & 3.00                                                                                & 2.29                                                                                & 1.00                                                                                & 1.00                                                                                & 0.59                                                                           \\ \cline{2-7}
                                                                                & MLP-500                         & 1.86                                                                                & 1.80                                                                                & 9.98                                                                                & 10.00                                                                                & 0.17                                                                           \\ \cline{2-7}
                                                                                & MLP-50-25                       & 3.59                                                                                & 5.38                                                                                & 1.07                                                                                & 1.07                                                                                & 0.32                                                                           \\ \cline{2-7}
                                                                                & MLP-200-100                     & 2.89                                                                                & 2.86                                                                                & 5.03                                                                                & 5.04                                                                                & 0.25                                                                           \\ \cline{2-7}
                                                                                & SVM                             & 1.00                                                                                & 1.00                                                                                & 37.05                                                                                & 17.34                                                                                & 0.07                                                                           \\ \hline
\end{tabular}
\caption{The combined metric of models used in scenario 2 when users care most about different factors.}
\label{tbl_combined_metric_2}
\end{table*}

\begin{table*}[h]
\centering
\begin{tabular}{|l|l|l|l|l|l|l|}
\hline
\multicolumn{1}{|c|}{\begin{tabular}[c]{@{}c@{}}Weights\\ (a1,a2,b1,b2)\end{tabular}}                                    & \multicolumn{1}{c|}{Model Name} & \multicolumn{1}{c|}{\begin{tabular}[c]{@{}c@{}}Normalized\\ FNR\\ (security)\end{tabular}} & \multicolumn{1}{c|}{\begin{tabular}[c]{@{}c@{}}Normalized\\ FPR\\ (usability)\end{tabular}} & \multicolumn{1}{c|}{\begin{tabular}[c]{@{}c@{}}Normalized\\ Execution\\ Time\end{tabular}} & \multicolumn{1}{c|}{\begin{tabular}[c]{@{}c@{}}Normalized\\ Memory\\ Usage\end{tabular}} & \multicolumn{1}{c|}{\begin{tabular}[c]{@{}c@{}}Combined\\ Metric\end{tabular}} \\ \hline
\multirow{4}{*}{\begin{tabular}[c]{@{}l@{}}(0.5, 0.5, 0.5, 0.5)\\ Equal weights\end{tabular}}                            & OCSVM                           & 3.06                                                                                & 6.03                                                                                & 1.00                                                                                & 1.00                                                                                & 0.36                                                                           \\ \cline{2-7}
                                                                                & LSTM-200-th                     & 3.37                                                                                & 5.54                                                                                & 1.32                                                                                & 2.78                                                                                & 0.31                                                                           \\ \cline{2-7}
                                                                                & LSTM-200-KS-OCSVM               & 1.00                                                                                & 4.36                                                                                & 1.87                                                                                & 2.82                                                                                & 0.40                                                                           \\ \cline{2-7}
                                                                                & LSTM-200-KS-Vote                & 1.98                                                                                & 1.00                                                                                & 1.84                                                                                & 2.81                                                                                & 0.52                                                                           \\ \hline
\multirow{4}{*}{\begin{tabular}[c]{@{}l@{}}(0.9, 0.1, 0.5, 0.5)\\ FNR critical\end{tabular}}                             & OCSVM                           & 3.06                                                                                & 6.03                                                                                & 1.00                                                                                & 1.00                                                                                & 0.46                                                                           \\ \cline{2-7}
                                                                                & LSTM-200-th                     & 3.37                                                                                & 5.54                                                                                & 1.32                                                                                & 2.78                                                                                & 0.35                                                                           \\ \cline{2-7}
                                                                                & LSTM-200-KS-OCSVM               & 1.00                                                                                & 4.36                                                                                & 1.87                                                                                & 2.82                                                                                & 0.54                                                                           \\ \cline{2-7}
                                                                                & LSTM-200-KS-Vote                & 1.98                                                                                & 1.00                                                                                & 1.84                                                                                & 2.81                                                                                & 0.48                                                                           \\ \hline
\multirow{4}{*}{\begin{tabular}[c]{@{}l@{}}(0.9, 0.1, 0.9, 0.1)\\ FNR critical\\ Execution time\\ critical\end{tabular}} & OCSVM                           & 3.06                                                                                & 6.03                                                                                & 1.00                                                                                & 1.00                                                                                & 0.46                                                                           \\ \cline{2-7}
                                                                                & LSTM-200-th                     & 3.37                                                                                & 5.54                                                                                & 1.32                                                                                & 2.78                                                                                & 0.40                                                                           \\ \cline{2-7}
                                                                                & LSTM-200-KS-OCSVM               & 1.00                                                                                & 4.36                                                                                & 1.87                                                                                & 2.82                                                                                & 0.61                                                                           \\ \cline{2-7}
                                                                                & LSTM-200-KS-Vote                & 1.98                                                                                & 1.00                                                                                & 1.84                                                                                & 2.81                                                                                & 0.52                                                                           \\ \hline
\multirow{4}{*}{\begin{tabular}[c]{@{}l@{}}(0.1, 0.9, 0.5, 0.5)\\ FPR critical\end{tabular}}                             & OCSVM                           & 3.06                                                                                & 6.03                                                                                & 1.00                                                                                & 1.00                                                                                & 0.30                                                                           \\ \cline{2-7}
                                                                                & LSTM-200-th                     & 3.37                                                                                & 5.54                                                                                & 1.32                                                                                & 2.78                                                                                & 0.27                                                                           \\ \cline{2-7}
                                                                                & LSTM-200-KS-OCSVM               & 1.00                                                                                & 4.36                                                                                & 1.87                                                                                & 2.82                                                                                & 0.31                                                                           \\ \cline{2-7}
                                                                                & LSTM-200-KS-Vote                & 1.98                                                                                & 1.00                                                                                & 1.84                                                                                & 2.81                                                                                & 0.58                                                                           \\ \hline
\end{tabular}
\caption{The combined metric of models used in scenario 2 when users care most about different factors.}
\label{tbl_combined_metric_2}
\end{table*}

\iffalse
\gyreffig{img_pnlty1} shows the penalties when the user cares most about the accuracy and uses $(w_a, w_t, w_m) = (0.9, 0.05, 0.05)$ as the weights. In this case, the basic LSTM model which suffers from the the low accuracy of 71.60\% has the highest penalty. The LSTM models with KS support improves the accuracy but has more penalties than the multi-class models. The multi-class models all have small penalties as their accuracy is larger than 90\%.

\begin{figure}[ht]
    \centering
    \includegraphics[width=\linewidth]{./figures/pnlty1.pdf}
    \caption{The combined penalty emphasizing the accuracy}
    \label{img_pnlty1}
\end{figure}

\gyreffig{img_pnlty2} shows a different user preference when the memory storage is the critical factor. A weight vector as $(w_a, w_t, w_m) = (0.05, 0.05, 0.9)$ is used. The result shows that the SVM model, even though having the highest detection accuracy, becomes the least preferred model due to its large model size. The small MLP models like MLP-50 and MLP-50-25 are shown to be the best choices as they use much less memory but stall achieve a high accuracy.

\begin{figure}[ht]
    \centering
    \includegraphics[width=\linewidth]{./figures/pnlty2.pdf}
    \caption{The combined penalty emphasizing the memory usage}
    \label{img_pnlty2}
\end{figure}
\fi
\subsection{Model Evaluation}

\subsection{Model Performance and Trade-offs}
\label{sec_model_perf}

\iffalse
\begin{figure}[ht]
    \centering
    \includegraphics[width=\linewidth]{./figures/plot_perf_mem_0.pdf}
    \caption{Evaluation of models used in Scenario 1}
    \label{img_eval_scene1}
\end{figure}
\fi

\iffalse
\begin{figure}[ht]
    \centering
    \includegraphics[width=\linewidth]{./figures/0605_cmp_time.pdf}
    \caption{Execution time of models on~SID~compared to CPU/GPU platforms}

\end{figure}
\fi

\gyreffig{img_hw_perf} compares different machine learning and deep learning algorithms for their trade-offs between execution time (orange bars with crosshatch) and accuracy (red line) on the~SID~module and the CPU-GPU platform. The models to the left of the black dashed line are used in the IDaaS scenario when sensor data from other users are available. The models trained without the other users' data in the LAD scenario are to the right of the dashed line.
Although the SVM algorithm achieves slightly higher accuracy than MLP-200-100 (98.4\% versus 97.1\%), it needs significantly longer execution time.

\begin{figure}[ht]
    \centering
    \includegraphics[width=\linewidth]{./figures/0602_cmp_time.pdf}
    \caption{Execution time and accuracy of models}
    \label{img_hw_perf}
\end{figure}

For the one-class models in the LAD scenario, we find that the KS test technique increases the detection accuracy, but also needs additional execution time. Compared to the \textbf{LSTM-200} model which involves only LSTM and error computation and has low accuracy, \textbf{PED-LSTM-200-OCSVM} increases the execution time on SID~by 44.9\%, to which KS test contributes 41.7\% and the following one-class SVM contributes the remaining 3.2\%. \textbf{PED-LSTM-200-Vote} increases the execution time by 41.8\%, compared to the basic LSTM model, and the execution time is mostly spent in computing the KS test. The majority vote part only contributes less than 0.1\%.

To achieve real-time detection of an impostor, the total execution time should be less than the period of sensor reading (PoS). We also tried a CPU-only platform with 32 Intel Xeon E5-2667 cores, it fails to meet this requirement. Running a single prediction on one sensor sample with the \textbf{LSTM-200} model takes more than 20 ms, i.e. the PoS of our dataset for the smartphone sensor sampling rate of 50Hz. The SID~module and our \textbf{CPU-GPU} platform which has a NVIDIA GTX 1080Ti GPU can meet the requirement to provide enough performance, however, the latter consumes much more energy as we will show in~\gyrefsec{sec_eval_energy}.

\iffalse
Therefore, we only use the performance (the blue bars with slash hatch) of our \textbf{CPUGPU} platform which has a NVIDIA GTX 1080Ti GPU for comparison.

The~SID~module can compute the models used in Scenario 2 which include~\textbf{LSTM-200} faster than the~\textbf{CPUGPU} platform but performs no better for larger MLP models in Scenario 1 and the SVM models. However, as long as the execution time of all the models on the~SID~module is significantly (at least 12.5X) less than the PoS of 20ms, the~SID~module meets the requirement to provide enough performance for real-time impostor detection. The execution time stills matters in terms of power consumption as we will discuss in~\gyrefsec{sec_eval_energy}
\fi

\subsection{Model Memory Usage}

\iffalse
\begin{figure}[ht]
    \centering
    \includegraphics[width=\linewidth]{./figures/plot_perf_mem_1.pdf}
    \caption{Evaluation of models used in Scenario 2}
    \label{img_eval_scene2}
\end{figure}
\fi

\gyreffig{img_hw_mem}
compares models used in the IDaaS and LAD scenarios, in terms of their accuracy and the size of the model parameters. The red line stands for the average accuracy. In the IDaaS scenario, we see that a 2-layer MLP-200-100 can achieve a slightly higher detection accuracy with a smaller model size than MLP-500.
The SVM model has little improvement on the accuracy over MLP-200-100 but incurs the highest cost in terms of memory usage as it has to store many support vectors. Hence, MLP-200-100 appears to be the best for the cost (execution time + memory usage) versus accuracy trade-off.

In the LAD scenario which preserves the sensor data privacy, the standard LSTM-200 solution requires more space than the OCSVM model but gives a similar accuracy of 69\%. However, The LSTM-based models enhanced with KS-test, PED-LSTM-200-OCSVM and PED-LSTM-200-Vote, take 1.8\% and 0.6\% more space, respectively, but can significantly improve the overall accuracy. They are better choices in the cost-versus-accuracy trade-off as the improvement in accuracy is significant, and a user can choose one or the other based on his/her need for better security or usability (\gyrefsec{sec_alg_eval}).

\iffalse
The LSTM-based model with 200 hidden nodes uses a similar amount of memory as the single-layer MLP model with 500 hidden neurons. This is because it has weights to compute the three gates and the candidate state. The KS support is found to have a small additional memory usage of 0.6\% compared to the basic LSTM model. The difference in memory usage between whether OCSVM or majority vote is used after the KS test is small, as neither exceeds 1.2\%.
\fi

\begin{figure}[ht]
    \centering
    \includegraphics[width=\linewidth]{./figures/mem.pdf}
    \caption{Number of parameters and accuracy of models}
    \label{img_hw_mem}
\end{figure}
\subsection{Hardware Design Complexity}
\label{sec_hardware_design_cost}

\iffalse
\begin{table*}[ht]
\centering
\resizebox{0.9\textwidth}{!}{
\begin{tabular}{|l|l|l|l|l|}
\hline
\multirow{2}{*}{} & \multicolumn{1}{c|}{\multirow{2}{*}{DeltaRNN}}                                   & \multicolumn{1}{c|}{\multirow{2}{*}{C-LSTM}}                   & \multicolumn{2}{c|}{SID}                                                                                \\ \cline{4-5}
                  & \multicolumn{1}{c|}{}                                                            & \multicolumn{1}{c|}{}                                          & \multicolumn{1}{c|}{Basic}                                                & \multicolumn{1}{c|}{Nested Loop}                                          \\ \hline
Functionality     & \begin{tabular}[c]{@{}l@{}}Supports Gated \\ Recurrent Unit only\end{tabular}    & Supports LSTM only                                             & \multicolumn{2}{l|}{\begin{tabular}[c]{@{}l@{}}Supports LSTM and other RNNs,\\ ML models and statistical tests\end{tabular}}                          \\ \hline
Platform          & \begin{tabular}[c]{@{}l@{}}Xilinx Zynq-7000 All \\ Programmable SoC\end{tabular} & \begin{tabular}[c]{@{}l@{}}Alpha Data's\\ ADM-7V3\end{tabular} & \multicolumn{2}{l|}{\begin{tabular}[c]{@{}l@{}}Xilinx Zynq-7000 SoC\\ ZC706 Evaluation Kit\end{tabular}}                                              \\ \hline
FPGA              & Kintex-7 XC7Z100                                                                 & Xilinx Virtex-7 (690t)                                         & \multicolumn{2}{l|}{XC7Z045 FFG900}                                                                                \\ \hline
Design Tool       & Xilinx Vivado 2017.2                                                             & Xilinx SDx 2017.1                                              & \multicolumn{2}{l|}{Xilinx Vivado 2016.2}                                                                                \\ \hline
Quantization      & 16-bit fixed point                                                               & 16-bit fixed point                                             & \multicolumn{2}{l|}{32-bit fixed point}                                                                                \\ \hline
Slice LUT         & 261,357 \textbf{(31.52X)}                                                                 & 406,861 \textbf{(49.07X)}                                               & 8,292 \textbf{(1X)}                                                                & 8,714 \textbf{(1.05X)}                                                             \\ \hline
Slice Flip-flop   & 119,260 \textbf{(31.40X)}                                                                 & 402,876 \textbf{(106.07X)}                                              & 3,798 \textbf{(1X)}                                                                & 4,215 \textbf{(1.11X)}                                                             \\ \hline
DSP               & 768 \textbf{(48X)}                                                                        & 2675 \textbf{(167.19X)}                                                 & 16 \textbf{(1X)}                                                                   & 16 \textbf{(1X)}                                                                   \\ \hline
BRAM              & 457.5 \textbf{(0.94X)}                                                                    & 966 \textbf{(1.98X)}                                                    & 489 \textbf{(1X)}                                                                  & 489 \textbf{(1X)}                                                                  \\ \hline
Clock Frequency   & 125MHz                                                                           & 200MHz                                                         & 115MHz                                                                    & 115MHz                                                                    \\ \hline
Power (W)         & \begin{tabular}[c]{@{}l@{}}Static: 7.9\\ FPGA Running: 15.2\end{tabular}         & FPGA Running: 22                                               & \begin{tabular}[c]{@{}l@{}}Static: 0.12\\ FPGA Running: 0.62\end{tabular} & \begin{tabular}[c]{@{}l@{}}Static: 0.12\\ FPGA Running: 0.62\end{tabular} \\ \hline
\end{tabular}
}
\caption{Significantly lower hardware utilization and power of SID~compared to performance-oriented RNN accelerators}
\label{tbl_hw_util}
\end{table*}
\fi

\begin{table}[ht]
\centering
\resizebox{0.48\textwidth}{!}{
\begin{tabular}{|l|l|l|l|}
\hline
                & \multicolumn{1}{c|}{DeltaRNN}                                                     & \multicolumn{1}{c|}{C-LSTM}                                    & \multicolumn{1}{c|}{SID}                                                                       \\ \hline
Functionality   & \begin{tabular}[c]{@{}l@{}}Supports Gated \\ Recurrent Unit\\ only\end{tabular}   & \begin{tabular}[c]{@{}l@{}}Supports LSTM\\ only\end{tabular}   & \begin{tabular}[c]{@{}l@{}}Supports LSTM\\ and other RNNs,\\ ML models and\\ statistical tests\end{tabular} \\ \hline
Platform        & \begin{tabular}[c]{@{}l@{}}Xilinx Zynq-7000\\ All Programmable\\ SoC\end{tabular} & \begin{tabular}[c]{@{}l@{}}Alpha Data's\\ ADM-7V3\end{tabular} & \begin{tabular}[c]{@{}l@{}}Xilinx Zynq-\\ 7000 SoC ZC706\\ Evaluation Kit\end{tabular}                      \\ \hline
FPGA            & Kintex-7 XC7Z100                                                                  & Xilinx Virtex-7                                                & \begin{tabular}[c]{@{}l@{}}XC7Z045\\ FFG900\end{tabular}                                                    \\ \hline
Design Tool     & Vivado 2017.2                                                                     & Xilinx SDx 2017.1                                              & Vivado 2016.2                                                                                \\ \hline
Quantization    & 16-bit fixed point                                                                & 16-bit fixed point                                             & 32-bit fixed point                                                                                \\ \hline
Slice LUT       & 261,357 (31.52X)                                                                  & 406,861 (49.07X)                                               & 8,292 (1X)                                                                                \\ \hline
Slice Flip-flop & 119,260 (31.40X)                                                                  & 402,876 (106.07X)                                              & 3,798 (1X)                                                                                \\ \hline
DSP             & 768 (48X)                                                                         & 2675 (167.19X)                                                 & 16 (1X)                                                                                \\ \hline
BRAM            & 457.5 (0.94X)                                                                     & 966 (1.98X)                                                    & 489 (1X)                                                                                \\ \hline
Clock Freq      & 125MHz                                                                            & 200MHz                                                         & 115MHz                                                                                \\ \hline
Power (W)       & \begin{tabular}[c]{@{}l@{}}Static: 7.9\\ Running: 15.2\end{tabular}               & Running: 22                                                    & \begin{tabular}[c]{@{}l@{}}Static: 0.12\\ Running: 0.62\end{tabular}                                        \\ \hline
\end{tabular}
}
\caption{Hardware utilization and power compared to performance-oriented RNN accelerators}
\label{tbl_hw_util}
\end{table}

We implement an FPGA prototype of the~SID~module. The implementation has four parallel tracks and a 256-byte scratchpad. The size of the data RAM is 1.75MB ($448 \times 4kB$ BRAM blocks) and the size of the instruction RAM is 128KB ($32 \times 4kB$ BRAM blocks). We use 32-bit fixed-point numbers, since prior work~\cite{pudiannao:Liu:2015:PPM:2694344.2694358} has shown significant accuracy degradation with 16-bit numbers. The platform board is Xilinx Zynq-7000 SoC ZC706 evaluation kit. The hardware implementation is generated with Vivado 2016.2.

In~\gyreftbl{tbl_hw_util}, we compare SID~to two FPGA implementations of RNN accelerators, DeltaRNN~\cite{gao2018deltarnn} and C-LSTM~\cite{wang2018clstm}, as the LSTM or GRU RNN computation is used in the LAD scenario and accounts for most of its execution time and memory usage. DeltaRNN ignores the minor change in the input of gated recurrent unit (GRU) RNN to reuse the old computation result and thus reduces the computation workload. C-LSTM shows that, some matrices in LSTM can be represented as multiple block-circulant matrices and reduces the storage and computation complexity. These two accelerators are capable of inferring the RNN but lack the support for generating and comparing empirical PED's, which we have shown is indispensable to achieve acceptable accuracy.

\iffalse
From the functionality perspective, the~SID~module supports the whole detection by providing all the necessary computation primitives when the accelerators require subsequent processing. Also, for fair comparison without saving energy by losing accuracy, we evaluate the impact of using the RNN algorithms supported by DeltaRNN and C-LSTM. C-LSTM proves to resemble the original LSTM after structural compression, so no accuracy loss is expected to happen. DeltaRNN accelerates the variant of GRU RNN. We compare the accuracy of using GRU and using LSTM in~\gyreftbl{tbl_gru_lstm_accuracy}. The result shows the GRU achieves a 1.24\% increase in TNR and a 5.08\% increase in TPR but still performs no better than the LSTM model with KS test support. Moreover, the GRU is still within the scope of models the SID~module can support so using SIDE will not sacrifice the accuracy.

\begin{table}[ht]
\resizebox{0.48\textwidth}{!}{
\begin{tabular}{|l|l|l|l|l|l|l|}
\hline
\textbf{}                                                             & \textbf{TNR}   & \textbf{TPR}   & \textbf{Accuracy} & \textbf{R}    & \textbf{P}    & \textbf{F1}   \\ \hline
\textbf{LSTM-200}                                                     & \textbf{67.88} & \textbf{71.60} & \textbf{69.74}    & \textbf{0.72} & \textbf{0.58} & \textbf{0.62} \\ \hline
\textbf{GRU-200}                                                      & \textbf{69.12} & \textbf{76.68} & \textbf{72.90}    & \textbf{0.77} & \textbf{0.63} & \textbf{0.68} \\ \hline
\textbf{\begin{tabular}[c]{@{}l@{}}LSTM-200-\\ KS-OCSVM\end{tabular}} & \textbf{74.82} & \textbf{92.25} & \textbf{83.53}    & \textbf{0.92} & \textbf{0.80} & \textbf{0.84} \\ \hline
\textbf{\begin{tabular}[c]{@{}l@{}}LSTM-200-\\ KS-Vote\end{tabular}}  & \textbf{94.62} & \textbf{83.31} & \textbf{88.97}    & \textbf{0.83} & \textbf{0.91} & \textbf{0.83} \\ \hline
\end{tabular}
}
\caption{Accuracy comparison of LSTM and Gated Recurrent Unit (GRU).}
\label{tbl_gru_lstm_accuracy}
\end{table}

\fi

The FPGA resource usage of Slice LUTs, Slice Flip-flops and DSPs of SID~are one or two orders of magnitude less than the other two, which shows a major difference between the minimalist SID~module and the performance-oriented accelerators. We measure the FPGA power consumption using TI Fusion Digital Power Designer tool. The power consumption is an order of magnitude less, making it more suitable for a smartphone.
We use an FPGA implementation as a prototype of SID~to compare with existing FPGA accelerators. Further power reduction is achievable using an ASIC implementation in real smartphone products.

\subsection{Program Code Size}

\gyreftbl{tbl_code_size} shows the code size to run algorithms, which provide best detection accuracy in both scenarios, in the SID module. Note that the two complex models, \textbf{LSTM-200-KS-OCSVM} and \textbf{LSTM-200-KS-Vote}, in Scenario 2 both consist of three stages. Specifically, \textbf{LSTM-200-KS-OCSVM}~can be divided into \textbf{LSTM-200}, \textbf{KS-40} and \textbf{OCSVM (after KS-40)} and \textbf{LSTM-200-KS-Vote} can be divided into \textbf{LSTM-200}, \textbf{KS-40} and \textbf{Vote-KS}. These stages benefit differently from the nested loop support so we discuss the stages separately in \gyreftbl{tbl_code_size}.

The sizes of MLP models used in Scenario 1 grow with the model size. This is because we divide a matrix into multiple blocks when it is too big to put all the output nodes in the local scratchpad.

The loop support shows a 182.2X and 33.7X reduction in code size for SVM and OCSVM models which have the biggest code size when the loop support is not implemented.
The small OCSVM model in~\textbf{LSTM-200-KS-OCSVM} also benefits from the loop support to reduce the code size by 5.5X. The KS test has two outer loops for each prediction error and each reference distribution so the nested loop support can also be applied and this reduces the code size by 147.7X. The small code size shows that with the loop support, it is feasible to put the code in a dedicated memory for fast access

\begin{table}[h]
\centering
\resizebox{0.9\columnwidth}{!}{
\begin{tabular}{|c|l|l|l|l|}
\hline
\multicolumn{1}{|l|}{}     &                           & SIDE  & SIDE+ & \\ \hline
\multirow{5}{*}{\begin{tabular}[c]{@{}c@{}}Scenario\\ 1\end{tabular}} & MLP-50                                                            & 112                                                                             & N/A                                                                            & N/A                                                            \\ \cline{2-5}
                                                                      & MLP-500                                                           & 224                                                                             & N/A                                                                            & N/A                                                            \\ \cline{2-5}
                                                                      & MLP-50-2
                                                                      & 208                                                                             & N/A                                                                            & N/A                                                            \\ \cline{2-5}
                                                                      & MLP-200-10
                                                                      & 224                                                                             & N/A                                                                            & N/A                                                            \\ \cline{2-5}
                                                                      & SVM                                                               & 78.7k                                                                           & 432                                                                            & 182.2X                                                         \\ \hline
\multirow{5}{*}{\begin{tabular}[c]{@{}c@{}}Scenario\\ 2\end{tabular}} & OCSVM                                                             & 14,6k                                                                           & 432                                                                            & 33.7X                                                          \\ \cline{2-5}
                                                                      & LSTM-200                                                          & 560                                                                             & N/A                                                                            & N/A                                                            \\ \cline{2-5}
                                                                      & KS-40 (20 Refs)
                                                                      & 52k
                                                                      & 352                                                                            & 147.7X                                                         \\ \cline{2-5}
                                                                      & OCSVM-KS
                                                                      & 2.4k                                                                            & 432                                                                            & 5.5X                                                           \\ \cline{2-5}
                                                                      & Vote-KS                                                           & 32                                                                              & N/A                                                                            & N/A                                                            \\ \hline
\end{tabular}
}
\caption{Code size of algorithms (in bytes) for impostor detection. KS-40 means a KS test of two PED's with 40 prediction errors.}
\label{tbl_code_size}
\end{table}
\subsection{Energy Consumption versus CPUGPU}
\label{sec_eval_energy}

We evaluate the energy consumption of platforms supporting the impostor detection algorithms within one period of sensor reading (PoS), which is also the period of real-time detection. The energy consumption includes that consumed during execution when the devices are actively running and also the energy when they finish execution and become idle. Specifically, the equations to model the energy consumption of the~\textbf{CPUGPU} platform and the SID~platform are given in~\gyrefeq{eq_detection_energy_platforms}. $P_{run}$ and $P_{idle}$ are the power consumption when a device is running the detection algorithms or in the idle state. $t_*$ is the time when the corresponding device is executing the algorithm. $T$ is the PoS, so $T-t_*$ is the time within a PoS when the corresponding device is not executing anything.
\begin{equation}
\resizebox{0.9\columnwidth}{!}{
$
\begin{split}
    E(CPUGPU)&=P_{run}(CPU) \times t_{CPU} + P_{idle}(CPU) \times (T-t_{CPU}) +\\
             &~P_{run}(GPU) \times t_{GPU} + P_{idle}(GPU) \times (T-t_{GPU}) \\
    E(SID) = &P_{run}(SID) \times t_{SID} + P_{idle}(SID) \times (T-t_{SID})
\end{split}
$
}
\label{eq_detection_energy_platforms}
\end{equation}
We then apply our model to compare the energy consumption used by the two platforms within a single period of sensor reading. We present a conservative comparison by not including the CPU energy in the equation of~\textbf{E(CPUGPU)} as the CPU typically runs many processes and the statistical computations it does for impostor detection may not add much to the total energy consumption.

\begin{table}[h]
\centering
\begin{tabular}{|l|c|c|}
\hline
     & Idle Power (W
     & Running Power (W
     \\ \hline
GPU  & 8                                                        & 56                                                          \\ \hline
SID & 0.12                                                     & 0.62                                                        \\ \hline
\end{tabular}
\caption{Idle and running power of different devices}
\label{tbl_power_platforms}
\end{table}

The idle and running power of GPU and~SID~are given in~\gyreftbl{tbl_power_platforms}.
The GPU power is measured using the nvidia-smi tool when the GPU is idle or is running the detection algorithm. The power consumption of our SID~module is as given in~\gyreftbl{tbl_hw_util}.
\gyreffig{img_0605_energy} shows the energy consumption normalized to the SID~platform when running the models on the two platforms.
The average consumption for all models on the CPU-GPU platform is 64.41X
higher than that of the SID~platform. Interestingly, the ratio of energy consumption is close to the ratio of idle powers of the two platforms ($\frac{8}{0.12}=66.66$). The takeaway is that when the execution time is short compared to the PoS (\gyrefsec{sec_model_perf}), the energy consumption is determined by that consumed in the idle state, which complies with our effort to reduce hardware cost and static power. SID~also frees up the GPU for its original graphics purposes, rather than tying it up for impostor detection.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.95\linewidth]{./figures/0605_energy.pdf}
    \caption{Normalized energy consumed by the impostor detection algorithms within one sensor reading period}
    \label{img_0605_energy}
\end{figure}
\section{Related Work}
\label{sec_related_work}

Dedicated accelerators for a single machine learning (ML) algorithm have been built by researchers,
e.g. for SVM \cite{fpga:svm:cascade}\cite{svm:training:fpga:papadonikolakis2010heterogeneous}, for k-th neatest neighbors~\cite{knn:acceleration:schumacher2008hardware} and for k-means\cite{accelerator:k-means}.

Accelerators are also built for deep learning (DL) algorithms, e.g. for deep neural networks (DNNs).
Many of these accelerators are designed with insights identified in software execution. EIE~\cite{han2016eie} and Minerva~\cite{reagen2016minerva} exploit data sparsity of weights and activations during inference to improve performance and energy efficiency. The sparsity in training is exploited in~\cite{rhu2018compressingdma} to improve performance. Minerva~\cite{reagen2016minerva} presents a framework to reduce power consumption by finding the optimal data quantization in the accelerator with software exploration.
Weight sharing in CNN is identified by~\cite{song2018insituai} in early software exploration before designing a dedicated accelerator. These accelerators benefit from different properties of DNN models to improve performance or energy efficiency, but the versatility for implementing other ML/DL algorithms is not considered, as we do for SID.

Some hardware accelerators also target supporting multiple machine learning models.~\cite{multi:cnn:kmeans:accleration:majumdar2011energy} evaluates the acceleration of four models in an embedded CPU-GPU-Acce\-lerator system. MAPLE~\cite{multi:maple:majumdar2012massively} accelerates the vector and matrix operations found in five classification workloads. PuDianNao~\cite{pudiannao:Liu:2015:PPM:2694344.2694358} highlights the non-vector operations and data locality in seven ML techniques. While these work support their chosen ML/DL algorithms, they do not support other needed processing such as statistical KS tests as we do.

For minimalist hardware design, we incorporate conventional energy-saving techniques, e.g. the tiling method~\cite{diannao:Chen:2014:DSH:2541940.2541967}, that can benefit multiple ML/DL algorithms. We do not implement hardware modules for a specific ML/DL algorithm. To the best of our knowledge, we are the first to explore the direction of delivering versatility and sufficient hardware performance to solve a critical security problem, like smartphone impostor detection, with reduced energy consumption, rather than shooting for maximum performance or minimum power efficiency.

\section{Modeling Energy Consumption of Impostor Detection}
\label{sec_detection_energy}

We provide a generalized model of power consumption when the impostor detection is deployed on one/multiple hardware devices which are able to support the computation. The devices can include CPUs, GPUs and accelerators. For a system using a given set of devices ($SoD$),~\gyrefeq{eq_detection_energy} can be used to compute $E(SoD)$, which is the energy consumed within one period of sensor reading. The energy consumption of each device is computed as the sum of execution (running) energy, the idle energy and the energy to turn on ($E_{wakeup}$) and off ($E_{sleep}$) the devices.

\begin{equation}
\begin{split}
    E(SoD) = \sum_{i \in SoD} &(P_{running}(i) \times t_i
                              + P_{idle}(i) \times (T-t_i) \\
                              &+ E_{wakeup} + E_{sleep}) \\
       SoD :~&Set~of~devices \\
         T :~&Period~of~sensor~reading
\end{split}
\label{eq_detection_energy}
\end{equation}

We then propose three principles to deploy impostor detection on a hardware system:

\bheading{Make the execution time of detection less than the period of sensor reading.} To achieve shortest latency of detection, the detection analysis should be done for each incoming reading from sensors. Hence, the computation time should be less than the period of sensor reading. It means each $t_i$ in~\gyrefeq{eq_detection_energy} should be less than the period $T$. Furthermore, if the processing on different devices are sequential, the sum of $t_i$ should be less than $T$.

\bheading{Use as less devices as possible.} Additional devices can introduce energy cost even if they are in the idle state. Moreover,~\gyrefeq{eq_detection_energy}~includes the possible energy cost to wake up devices (from idle state to running state), which can add more to the $E(SoD)$. Therefore, it is desirable to have a detection module which is versatile enough to support the full detection without subsequent processing.

\bheading{Reduce the idle power for high-performance accelerators} We consider the case when the existing high-performance accelerators are used for ML-based impostor detection. Here we ignore the functionality issue and assume the detection process can be fully supported by the accelerator (which is not true for most specialized accelerators and becomes one of the design goals of our impostor detection module). The performance-oriented accelerators, with large throughput, is possible to accomplish the computation within a short period of time.

~\gyrefeq{eq_detection_energy}~for a single accelerator can then be reduced to~\gyrefeq{eq_detection_energy_accelerator}~if $P_{running}$ and $P_{idle}$ are of the same magnitude, which is verified in~\gyrefsec{sec_hardware_design_cost}. Given that $T$, $E_{wakeup}$ and $E_{sleep}$ can be considered a constant for a detection system, $P_{idle}$ becomes the key factor of energy consumption. Our third principle is hence to design a hardware module with enough performance and the least idle power.

\begin{equation}
\begin{split}
    E(SoD) = &P_{running} \times t_i + P_{idle} \times (T-t_i)
              + E_{wakeup} + E_{sleep} \\
     \approx &P_{idle} \times T
              + E_{wakeup} + E_{sleep} \\
             &(when~P_{running} \sim P_{idle}~and~t_i\ll T)
\end{split}
\label{eq_detection_energy_accelerator}
\end{equation}

We investigate the performance of detection algorithms on baseline platforms containing the general-purpose computing devices of CPU and GPU. The execution time of models giving best accuracy in~\gyrefsec{sec_sw_model} is shown in~\gyreffig{img_0800_cpugpu_time}. Two results about execution time, one for CPU platform and one for CPU-GPU platform are included for each model. The CPU platform runs the algorithms on a server with 32 Intel Xeon E5-2667 cores. We use the SciPy~\cite{scipy:jones2001scipy}~implementation of SVM, one-class SVM (OCSVM) and KS test and the TensorFlow~\cite{tensorflow:abadi2016tensorflow} CPU library for the computation of MLP and LSTM. The CPU-GPU platform uses the same server as the CPU platform and also has a NVIDIA GTX 1080Ti GPU. While the computation of MLP and LSTM can be moved to GPU by calling TensorFlow GPU library, the CPU is still needed to compute the algorithms of SVM, OCSVM and KS test.

\begin{figure}[ht]
    \centering
    \includegraphics[width=\linewidth]{./figures/0800_cpugpu_time.pdf}
    \caption{Exection time of models on platforms with only CPU and both CPU and GPU}
    \label{img_0800_cpugpu_time}
\end{figure}

While the CPU-GPU platform shows the benefit to deliver better performance as expected, there are a few critical things that should be noticed in~\gyreffig{img_0800_cpugpu_time}. First, the CPU platform fails to meet the first principle to provide enough performance for real-time detection as the execution time of LSTM-based algorithms exceeds the period of sensor reading (20ms), which is shown as the solid horizontal line (black). Second, the execution time of MLP models on GPU hardly increases with regard to the model size. This means the model size is not the bottleneck of performance but it can be inferred that the initialization of TensorFlow execution contributes to a large portion of the total execution time, which can involve the loading of program and is hard to reduce. Third, the execution time of KS test is making up more than half of the whole execution time of LSTM-200-KS-OCSVM and LSTM-200-KS-Vote when the time taken by LSTM inference is greatly reduced by GPU.

These observations adds more design goals for an ideal hardware module for impostor detection.
\begin{itemize}
    \item Provide enough performance for real-time detection (in accordance with Principle 1: Make the execution time of detection less than the period of sensor reading.)
    \item Reduce the code size for fast initialization
    \item Be versatile to support the statistical test, e.g. KS test
\end{itemize}
\section{Conclusions}
\label{sec_conclusion}

We show how sensors in a smartphone can be used to detect smartphone impostors and theft by identifying the current user using Deep Learning models like MLP and LSTM or GRU and the empirical PED's. We explore the algorithms that are needed in two scenarios, IDaaS and LAD, when users can or cannot use other people's data for training.

We design a hardware module, SID~, to support the best impostor detection algorithms we found in both scenarios. It is also versatile enough to support other ML/DL algorithms as well as collecting and comparing empirical probability distributions that we use to represent user behavior. This enables users of SID~to tradeoff security with data privacy in choosing one of the two scenarios, as well as choosing trade-offs in security with usability and accuracy with execution time and memory storage. Our SID~macro instructions can also map the same code to SID~hardware substrates with different amounts of parallel computation tracks. For efficiency, SID~reduces hardware resource usage by maximally reusing functional units.
Our evaluation shows that our FPGA implementation of SID~has a major difference with accelerators targeting similar ML/DL models: SID~provides sufficient performance with minimal hardware and energy costs,
which are one to two orders of magnitude less than performance-oriented FPGA accelerators. Using a general energy consumption model, we also show that the consumption of the FPGA implementation of a~SID~module is 64.41X smaller than the CPU-GPU platform.

We hope to have shown a new direction for computer architecture in the ML/DL domain: consider broader goals and trade-offs for real security (or other domain) needs rather than focus on optimizing just performance or power. Also, we hope to have shown the importance of a design methodology that considers both algorithm and architecture optimizations in solving critical problems like impostor detection, preventing subsequent confidentiality and integrity breaches.

\iffalse
\section{Conclusions}
\label{sec_conclusion}

In this paper, we show how sensors in a smartphone can be used to detect smartphone impostors and theft by identifying the current user using ML/DL models and the empirical distribution of prediction errors. We explore the models that are needed in two scenarios when users can or cannot use other people's data for training.

The hardware module called SID~is designed to support impostor detection in both scenarios. The SID~module has the versatility to support the self-contained detection by providing necessary operation primitives. Specifically, we propose a method to do efficient comparison of empirical distribution using statistical tests in the SID~module. It also has the programmability and scalability for the developers to flexibly change model configuration and map the same code to hardware substrates with different parallelism. For efficiency, SID~reduces hardware resource usage by maximally reusing functional units and the code size by providing control support for nested loops.
The evaluation shows that an FPGA implementation of SID~has a major difference with accelerators targeting similar models to provide enough performance with minimal design/energy cost,
which are one to two orders of magnitude less than performance-oriented FPGA accelerators. We finally use a general energy consumption model to show that the consumption of the FPGA implementation of~SID~module is 64.41X smaller than the CPU-GPU platform.

\fi
\section{Methodology for impostor detection}

Our proposed impostor detection methodology jointly optimizes the security enhancement and the cost of implementation. We consider the security enhancement part in this section, the cost of implementation in~\gyrefsec{sec_hw} and trade-offs in \gyrefsec{sec_evaluation}. Within security enhancement, our methodology explicitly takes three important factors into consideration: attack detection capability (security), usability and user data privacy of the solution.

\subsection{Security and Usability Trade-off}

A good implicit impostor detection solution needs to both be able to detect suspicious impostors and not affect the legitimate user's utilization. At the center of this trade-off is the selection of an appropriate model for impostor detection. One of our key intuitions and takeaways of our methodology is that choosing the right model is more important for achieving security and performance goals than increasing the model size or adding hardware complexity to accelerate a model.

Previous work on implicit smartphone user authentication mostly leverage the (user, non-user) binary classification techniques \cite{frank2012touchalytics, zheng2014you, lee2017implicit}. This scenario requires training models using both data from the real user and potential attackers and we call it the scenario of impostor detection-as-a-service (IDaaS) or Scenario 1 in this paper. The service provider is a centralized party. All end-users (customers) register their phones to the service provider by providing their patterns on motion recorded by the smartphone. The service provider collects data from all customers, creating a model for each customer using the entire data from all customers. For a specific customer, the data from him/herself are labeled as benign (the user's), while all data from other customers are labeled as malicious (non-user's).

We follow the literature and start with selecting certain classification-based machine learning and deep learning models. The goal is to find the algorithms which give the best accuracy for impostor detection (security) and legitimate user recognition (usability).

We first investigate three representative ML algorithms: logistic regression (LR), support vector machine (SVM) and kernel ridge regression (KRR). Logistic regression intrinsically provides the probability of a class and thus can be naturally leveraged as the warning score. We evaluate SVM because it is a powerful and commonly used linear model, which can establish a non-linear boundary using the kernel method (\gyrefapd{apd_class_alg}). KRR alleviates over-fitting by penalizing large parameters. It is suitable for learning with limited data and achieves the highest detection rate in the literature \cite{lee2017implicit}. Since these machine learning algorithms require heuristic feature selection, we also try the simplest deep learning model, i.e. multi-layer perceptron (MLP). MLP is a non-linear classifier and can automatically extract intricate patterns from raw data, without heuristic and tedious hand-crafting. Details of these algorithms and their equations are given in~\gyrefapd{apd_class_alg}.

\bheading{Metrics.} In order to quantify both factors, i.e. security and usability, of a given binary classification technique for impostor detection, we consider the metrics: true negative rate (TNR) and true positive rate (TPR). Note that a "positive" outcome is the detection of an impostor, while a "negative" outcome implies the legitimate user is detected.

Usability is represented by TNR, which is the percentage of samples when the real user is correctly recognized.
Security can be measured using TPR, which is the percentage of samples when a wrong user gets detected and rejected.
\gyrefeq{eq_sw_metric} gives the formula for TNR and TPR, as well as for the metrics commonly used in comparing the ML/DL models: accuracy, recall, precision and F1.

\begin{equation}
\begin{split}
  TNR
  = \frac{TN}{TN+FP} \\
  TPR &=
  = \frac{TP}{TP+FN} \\
  Accuracy &= \frac{TN+TP}{TN+FP+TP+FN}\\

  Recall (R) &= TPR \\
  Precision (P) &= \frac{TP}{TP+FP} \\

  F1 score &= \frac{2 \times Recall \times Precision}{Recall+Precision}
\end{split}
\label{eq_sw_metric}
\end{equation}

\subsection{Privacy and usability trade-off}

The aforementioned binary classification machine learning approaches can only be applied to the IDaaS scenarios where the data from other users are available. Unfortunately, many smartphone users do not want to share their sensor data for privacy reasons and for fear of being attacked in the network or the cloud. For data privacy concerns \cite{fbleak, googleleak} about users' behavioral biometric data, we need to consider another important scenario where the smartphone user only has his/her own data for training. Therefore, the impostor detection in this scenario can be formalized as a one-class classification, i.e. outlier detection problem. We call this local anomaly detection (LAD) or Scenario 2 in this paper.

Although most of the previous work of continuous smartphone authentication concentrate on binary classifications, there exists some preliminary pioneering work on one-class classification \cite{kumar2018continuous, kazachuk2016one, liatsis2017one}. However, none of these work leverage deep learning techniques on one-class smartphone authentication.

We consider three representative algorithms to deal with the lack of positive (non-user) training data, i.e. One-Class SVM (OCSVM), Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU). We propose enhancing the deep learning models, LSTM and GRU, with the comparison of reference and actual Prediction Error Distributions (PED's). We show in \gyrefsec{sec_alg_eval} that generating and comparing the prediction error distributions is the key to a successful detection for this Scenario 2. The ML/DL prediction algorithm appears to be only of secondary importance, and does not even have to be very accurate.

\bheading{OCSVM.} OCSVM is an extension of normal SVM, by separating all the data points from the origin in the feature space and maximizing the distance from this hyperplane to the origin. Intuitively, the OCSVM looks for the minimum support of the normal data, and recognizes points outside this support as anomalies. The kernel method (\gyrefapd{apd_class_alg}) can also be applied to establish a non-linear boundary.

\bheading{LSTM.} Different from the above discussed stateless models (SVM, KRR, OCSVM, etc.), the LSTM model has two hidden states ($h_t$ and $c_t$) which can remember the previous input information.
When being used to model temporal sequences, an LSTM cell updates its hidden states ($h_{t}$, $c_{t}$) for each input time-frame using the previous states ($h_{t-1}$, $c_{t-1}$) and the current input $x_t$ as described in~\gyrefeq{eq_lstm}. Three control gates, the forget gate $f_t$, the input gate $i_t$ and the output gate $o_t$, are used to determine how much of the old states are preserved. This significantly prevents the gradient vanishing problem in training deep neural networks. In~\gyrefeq{eq_lstm}, the W's and U's are weight matrices, and the b's are bias vectors.

{\baselineskip=0pt
\begin{equation}
\label{eq_lstm}
\begin{split}
  &cand_t = tanh(W_c \times x_{t} + U_c \times h_{t-1} + b_c) \\
  &f_t = \sigma (W_f \times x_{t} + U_f \times h_{t-1} + b_f) \\
  &i_t = \sigma (W_i \times x_{t} + U_i \times h_{t-1} + b_i) \\
  &o_t = \sigma (W_o \times x_{t} + U_o \times h_{t-1} + b_o) \\
  &c_t = f_t \odot c_{t-1} + i_t \odot cand_t \\%, where f(z) is the activation function
  &h_t = o_t \odot tanh(c_t)
\end{split}
\end{equation}
}
\bheading{GRU.} We replace the cell model with a more light-weight Gated Recurrent Unit (GRU) model, described in \gyrefeq{eq_gru}.
{\baselineskip=0pt
\begin{equation}
\label{eq_gru}
\begin{split}
  &z_t = \sigma (W_z \times x_{t} + U_z \times h_{t-1} + b_z) \\
  &r_t = \sigma (W_r \times x_{t} + U_r \times h_{t-1} + b_r) \\
  &h_t = (1-z_t) \odot h_{t-1} + z_t \odot \sigma (W_h x_t + U_r (r_t \odot h_{t-1}) + b_h) \\
\end{split}
\end{equation}
}

\iffalse
\begin{figure}
    \centering
    \includegraphics[width=0.7\linewidth]{./figures/lstm_fig.pdf}
    \caption{The computations within an LSTM cell}
    \label{img_lstm}
\end{figure}
\fi

We use an LSTM or GRU model as an outlier detector \cite{malhotra2015long}, by training it to predict the next sensor reading, and investigate the prediction errors. The intuition is, an LSTM model trained on only the normal user's data predicts better for his/her behavior than the other users' (potentially an impostor) behavior. The deviation of the real monitored behavior from the predicted behavior indicates the anomalies. Typically, a threshold value is used to decide if the prediction error is normal or not. A key contribution we make is to show that the choice of deep learning models, e.g. LSTM or GRU, is not the most important factor for good performance, if Prediction Error Distributions are leveraged.

\bheading{LSTM/GRU + Prediction Error Distribution (PED).} Our intuition is that a single prediction error may significantly vary, but the probability distribution of the errors is stable. Therefore, comparing the difference between the observed PED and a reference PED from the real user's validation data is more stable than comparing the average prediction error.

As we do not need to assume the prior distribution of PED, non-parametric tests are powerful tools to determine if two distributions are the same. The Kolmogorov-Smirnov (KS) test is a statistical test that determines whether two i.i.d sets of samples are from the same distribution. The KS statistic for two sets with $n$ and $m$ samples is:
\begin{align}\label{equ:KSstatistic}
D_{n,m} = sup_{x} |F_{n}(x)-F_{m}(x)|
\end{align}

where $F_{n}$ and $F_{m}$ are the empirical distribution functions of two sets of samples respectively, i.e. $F_{n}(t)=\frac{1}{n}\sum_{i=1}^{n}1_{x_i\leq t}$, and $sup$ is the supremum function. The null hypothesis that the two sets of samples are i.i.d. sampled from the same distribution, is rejected at level $\alpha$ if:
{\small
\begin{align}\label{equ:KStestinequ}
D_{n,m}>c(\alpha)\sqrt{\frac{n+m}{nm}}
\end{align}
}%
where $c(\alpha)$ is a pre-calculated value and can be found in the standard KS test lookup table.

\subsection{Experimental Settings}

We evaluate the models for impostor detection using the UCI \cite{Dua:2017} Human Activities and Postural Transitions (HAPT) dataset \cite{reyes2016transition}. The HAPT dataset contains smartphone sensor readings. The smartphone is worn on the waist of a group of 30 participants of various ages from 19 to 48. The participants were asked to perform six activities including standing, sitting, walking, etc. and the transitions between them. The sensor readings are collected and labeled with both the activity and user ID. Each reading consists of the 3-axial measurements of both the linear acceleration and angular velocity, so it could be treated as a 6-element vector. The sensors are sampled at 50Hz. We use the WALK dataset in HAPT to determine the feasibility of user versus impostor classification using the two most common built-in motion sensors and just one common type of motion.

We select 25 out of the 30 users in the HAPT dataset as the registered users while the other 5 users act as unregistered users. For each registered user, models are trained using his/her data and randomly picked sensor data of the other 24 registered users. In the training process, all the data from the other 24 registered users are labeled as positive for impostor detection and not the correct user. The unregistered users are used in testing to examine whether unseen attackers can be successfully detected.

As \cite{lee2017implicit} reports very high accuracy for recognizing users using sensor data, we implement the same KRR model which first computes 14 features: for each of the 2 sensors (accelerometer and gyroscope), 4 are common statistics like min, max, mean and standard deviation, in the time domain and 3 are frequency domain features.

\subsection{Security vs Usability Evaluation}
\label{sec_alg_eval}

We show the results of two-class classification in Table \ref{tbl_accuracy_binary}. We choose a window size of 64 \footnote{It is easier to extract FFT features if the window size is a power of 2.} sensor readings because this window size covers 1-2 full human steps given the sensors are sampled at 50 Hz. We observe that the SVM model performs the best for all the metrics. Surprisingly, it also performs better than KRR with manually selected features.
However, the simple deep learning model, MLP, performs almost as well, and costs less; SVM requires more storage and execution time than MLP for a marginal 1-2\% of accuracy increase (see left half of \gyreffig{img_hw_perf} and \gyreffig{img_hw_mem} in Section \ref{sec_evaluation}). These two figures will also show that, MLP with 2 hidden levels (MLP-200-100) is better, from the memory and execution-time performance perspective, than MLP with one hidden layer but more nodes (MLP-500), while both have 97\% accuracy.

\begin{table}[ht]
\centering
\resizebox{\columnwidth}{!}{
\begin{tabular}{|l|c|c|c|c|c|}
\hline
\multirow{2}{*}{Models}       & \multicolumn{5}{c|}{All Users}                                                                                \\ \cline{2-6}
                              & \begin{tabular}[c]{@{}c@{}}TNR\\ (\%)\end{tabular} & \begin{tabular}[c]{@{}c@{}}TPR/Recall\\ (\%)\end{tabular} & \begin{tabular}[c]{@{}c@{}}Accuracy\\ (\%)\end{tabular} & P                            & F1                           \\ \hline
LR                            & 80.77                                              & 66.05                                                     & 73.41                                                   & 0.77                         & 0.69                         \\ \hline
KRR                           & 88.91                                              & 82.66                                                     & 85.78                                                   & 0.87                         & 0.83                         \\ \hline
\textit{\textbf{SVM}}         & {\ul \textit{\textbf{99.26}}}                      & {\ul \textit{\textbf{97.57}}}                             & {\ul \textit{\textbf{98.42}}}                           & {\ul \textit{\textbf{0.99}}} & {\ul \textit{\textbf{0.98}}} \\ \hline
MLP-50                        & 98.31                                              & 92.70                                                     & 95.51                                                   & 0.98                         & 0.94                         \\ \hline
MLP-100                       & 98.60                                              & 94.65                                                     & 96.63                                                   & 0.98                         & 0.96                         \\ \hline
MLP-200                       & 98.41                                              & 95.72                                                     & 97.06                                                   & 0.98                         & 0.97                         \\ \hline
\textit{\textbf{MLP-500}}     & {\ul \textit{\textbf{98.68}}}                      & {\ul \textit{\textbf{95.47}}}                             & {\ul \textit{\textbf{97.07}}}                           & {\ul \textit{\textbf{0.99}}} & {\ul \textit{\textbf{0.96}}} \\ \hline
MLP-50-25                     & 98.13                                              & 94.49                                                     & 96.31                                                   & 0.98                         & 0.96                         \\ \hline
MLP-100-50                    & 98.44                                              & 95.45                                                     & 96.95                                                   & 0.98                         & 0.96                         \\ \hline
\textit{\textbf{MLP-200-100}} & {\ul \textit{\textbf{98.47}}}                      & {\ul \textit{\textbf{95.72}}}                             & {\ul \textit{\textbf{97.10}}}                           & {\ul \textit{\textbf{0.98}}} & {\ul \textit{\textbf{0.97}}} \\ \hline
\end{tabular}
}
\caption{Impostor detection in the IDaaS secnario, using binary classification ML and DL models, achieves 97\%-98\% accuracy.}
\label{tbl_accuracy_binary}
\end{table}

We use the same experimental settings on HAPT dataset for the evaluation of one-class outlier detection approaches. We train an OCSVM model with a fixed window size of 64. The latency for detection is a multiple of 20-ms (e.g.,64x, or 1.28 seconds) which is comparable to the time for an impostor to act. The LSTM or GRU model makes predictions for every sensor reading in a fixed window of size 200. The classification results of \textbf{LSTM-\#} and \textbf{GRU-\#} (rows 2 through 6 in \gyreftbl{tbl_accuracy_oneclass}, \# is the size of hidden states) are given by comparing the average prediction error with a threshold obtained from the validation set. The PED-enhanced LSTM or GRU leverages the distribution of prediction errors within the same window as LSTM or GRU. We first randomly choose 20 samples of prediction errors from the validation set and use them to represent the reference PEDs. In the testing phase, the prediction error distribution of each testing sample is compared to all the reference distributions. If half of the KS-test-statistics are larger than a commonly used p-value threshold, e.g. 0.05 in our experiment, the current sample is considered as abnormal.

We show the results of three types of one-class models, i.e. OCSVM, LSTM/GRU, and LSTM or GRU + PED, in Table \ref{tbl_accuracy_oneclass}. The one-class SVM achieves an average accuracy of 69.2\%, thirty percent worse than classification models trained with positive data involved. The standard LSTM or GRU models have an accuracy of $\sim$ 70\%, only slightly better than the one-class SVM model, regardless of the network architecture. On the contrary, there exists a significant improvement if PED and statistical KS test are leveraged, e.g. the accuracy of LSTM or GRU models + KS test reach 89.0\% or 92.3\%, respectively. We conclude that, neither the choice of deep learning models, e.g. LSTM or GRU (3.5\% gain), nor the design of network architectures (1.5\% gain) is the first essential factor for good performance. However, it is the Prediction Error Distributions and KS test that provide the significant increase in detection capability, i.e. +19.2\% for the best LSTM and +19.4\% for the best GRU. Without loss of generality, we use the more conservative numbers for LSTM in the rest of the paper (GRU will be slightly better).

Once PED and KS are used, we observe that the PED-LSTM-200-OCSVM and PED-LSTM-200-VOTE tradeoff between security and usability, respectively. The PED-LSTM-200-OCSVM achieves a high TPR of 92.25\% but a lower TNR of 74.82\%, i.e. a ``strict'' detection providing good security against impostors but can erroneously reject the real user. The voting-based method, i.e. PED-LSTM-200-VOTE, achieves a high TNR of 94.62\% but a lower TPR of 83.31\%, which means it is a ``lenient'' detection mechanism providing good usability for the correct user but limited security against impostors. Both approaches significantly outperform the non-PED (non-KS) approaches.

Given that adding the KS test is critical to improving the detection accuracy in this scenario, we provide the hardware support for generating empirical PEDs and computing the KS statistic in~\gyrefsec{sec_op_kstest}.

\begin{table}[ht]
\centering
\resizebox{0.48\textwidth}{!}{
\begin{tabular}{|l|c|c|c|c|c|}
\hline
\multirow{2}{*}{Models}                                       & \multicolumn{5}{c|}{All Users}                                                                                \\ \cline{2-6}
                                                              & \begin{tabular}[c]{@{}c@{}}TNR\\ (\%)\end{tabular} & \begin{tabular}[c]{@{}c@{}}TPR/Recall\\ (\%)\end{tabular} & \begin{tabular}[c]{@{}c@{}}Accuracy\\ (\%)\end{tabular} & P    & F1   \\ \hline
OCSVM                                                         & 64.24                                              & 74.19                                                     & 69.22                                                   & 0.59 & 0.65 \\ \hline
LSTM-50                                                       & 72.43                                              & 67.04                                                     & 69.74                                                   & 0.57 & 0.60 \\ \hline
LSTM-100                                                      & 72.20                                              & 69.27                                                     & 70.73                                                   & 0.58 & 0.62 \\ \hline
LSTM-200                                                      & 67.88                                              & 71.60                                                     & 69.74                                                   & 0.58 & 0.62 \\ \hline
LSTM-500                                                      & 67.57                                              & 74.42                                                     & 70.99                                                   & 0.60 & 0.65 \\ \hline
GRU-200                                                       & 69.12                                              & 76.68                                                     & 72.9                                                    & 0.63 & 0.68 \\ \hline
\begin{tabular}[c]{@{}l@{}}PED-LSTM-\\ 200-OCSVM\end{tabular} & 74.82                                              & {\ul \textit{\textbf{92.25}}}                             & 83.53                                                   & 0.80 & 0.84 \\ \hline
\begin{tabular}[c]{@{}l@{}}PED-GRU-\\ 200-OCSVM\end{tabular}  & 78.47                                              & {\ul \textit{\textbf{93.58}}}                             & 86.03                                                   & 0.82 & 0.86 \\ \hline
\begin{tabular}[c]{@{}l@{}}PED-LSTM-\\ 200-Vote\end{tabular}  & {\ul \textit{\textbf{94.62}}}                      & 83.31                                                     & 88.97                                                   & 0.91 & 0.83 \\ \hline
\begin{tabular}[c]{@{}l@{}}PED-GRU-\\ 200-Vote\end{tabular}   & {\ul \textit{\textbf{96.86}}}                      & 87.79                                                     & {\ul \textit{\textbf{92.33}}}                           & 0.94 & 0.88 \\ \hline
\end{tabular}
}
\caption{Impostor detection accuracy in the LAD secnario, using one-class models}
\label{tbl_accuracy_oneclass}
\end{table}

\iffalse
\begin{table}[]
\begin{tabular}{|l|l|l|l|}
\hline
\multicolumn{1}{|c|}{\multirow{2}{*}{\begin{tabular}[c]{@{}c@{}}Real\\ User ID\end{tabular}}} & \multicolumn{3}{c|}{OC-SVM}                                                         \\ \cline{2-4}
\multicolumn{1}{|c|}{}                                                                        & \multicolumn{1}{c|}{Avg} & \multicolumn{1}{c|}{Min} & \multicolumn{1}{c|}{Min User} \\ \hline
1                                                                                & 53.58                    & 0                        & 3                             \\ \hline
2                                                                                & 96.65                    & 22.22                    & 2                             \\ \hline
3                                                                                & 97                       & 30.91                    & 18                            \\ \hline
4                                                                                & 98.6                     & 58.85                    & 4                             \\ \hline
5                                                                                & 86.31                    & 0                        & 4                             \\ \hline
6                                                                                & 72.83                    & 0                        & 4                             \\ \hline
7                                                                                & 50.16                    & 0                        & 3                             \\ \hline
8                                                                                & 29.55                    & 0                        & 3                             \\ \hline
9                                                                                & 83.54                    & 0                        & 16                            \\ \hline
10                                                                                & 65.4                     & 0                        & 4                             \\ \hline
11                                                                                & 96.79                    & 40.08                    & 11                            \\ \hline
12                                                                                & 86.07                    & 9.19                     & 25                            \\ \hline
13                                                                                & 45.75                    & 0                        & 3                             \\ \hline
14                                                                                & 95.62                    & 1.36                     & 18                            \\ \hline
15                                                                                & 88.54                    & 0                        & 4                             \\ \hline
16                                                                                & 98.28                    & 70.45                    & 18                            \\ \hline
17                                                                                & 84.34                    & 0                        & 18                            \\ \hline
18                                                                                & 98.93                    & 71.36                    & 18                            \\ \hline
19                                                                                & 17.43                    & 0                        & 1                             \\ \hline
20                                                                                & 25.66                    & 0                        & 1                             \\ \hline
21                                                                                & 84.1                     & 0                        & 3                             \\ \hline
22                                                                                & 49.66                    & 0                        & 3                             \\ \hline
23                                                                                & 45.37                    & 0                        & 3                             \\ \hline
24                                                                                & 98.45                    & 60.85                    & 24                            \\ \hline
25                                                                                & 97.91                    & 40.28                    & 25                            \\ \hline
Avg                                                                                & 68.39                    & 15.02                    &                               \\ \hline
\end{tabular}
\end{table}

\fi

Note that our impostor detection methodology is an extra layer in a defense-in-depth approach that does not conflict with explicit authentication using PIN, voice or fingerprint. Very high sensitivity levels (95\%-99\%) are achieved for accuracy, security (TPR) and usability (TNR) when SVM or MLP models are used in the IDaaS scenario. If sensor data privacy is required, the accuracy is lower but still acceptable: security (92\%-93\%) or usability (94\%-96\%) can be achieved using our enhanced LSTM or GRU models, depending on one's needs. Although the accuracy is not perfect, it is actually comparable to the state-of-the-art one class smartphone authentication using various handcrafted features and complex model fusion \cite{kumar2018continuous} in the literature.

\section{Classification Algorithms}
\label{apd_class_alg}

\bheading{Logistic Regression} uses a logistic function to model a binary dependent variable (impostor or not), i.e.:
\begin{equation}
\label{eq_linearreg}
  \hat y = f(w^T x + b)
\end{equation}
where $x$ denotes the input, $w^T$ represents the transpose of $w$, and $f(z) = \frac{1}{1+e^{-z}}$ is the logistic function. The model prediction $\hat y \in (0,1)$ can be interpreted as the probability of $x$ being an impostor. The model parameters $w$ and $b$ are trained to fit the training data by minimizing the cross-entropy loss.

\bheading{Support Vector Machine} is a popular linear model in machine learning. To perform a binary classification, it identifies a hyperplane ($w^T x + b$ in \gyrefeq{eq:SVM:inner}) as the decision boundary
\iffalse
\begin{equation}
\label{eq_linearsvm}
  \hat y = sign(w^T x + b)
\end{equation}
\fi
where $w$ is the normal vector of the hyperplane. $Sign(\cdot)$ is the sign function. Unlike the logistic regression, $w$ and $b$ are trained by maximizing the hinge loss.
A property of the SVM is $w$ can be represented by a weighted average of support vectors (a subset of data points in the training set).
\begin{align}
  \hat y = sign(w^T x + b)
   = sign(\sum^{N}_{i=1} a_i \langle v_i, x \rangle + b)
\label{eq:SVM:inner}
\end{align}
where $N$ is the number of support vectors, $v_i$'s are the support vectors, $a_i$'s are the weights in representing $w$ of support vectors and $< \cdot >$ is the inner product of vectors.

If the data points are not linearly separable in the original space, a typical way is to map the data into a high-dimensional space where they can be linearly separated.
A kernel function can be leveraged
to obtain an equivalent classifier in the high-dimensional space, by replacing the inner product in \gyrefeq{eq:SVM:inner}.
The kernel function used in this paper is the Gaussian Kernel, i.e. $ \kappa(u,v)=e^{-\gamma \norm{u-v}^2}$.
\iffalse
\begin{align}
    \hat y = sign(\sum^{N}_{i=1} a_i \kappa(v_i, x) + b) \label{eq:SVM:kerneltrick}
\end{align}
\fi

\bheading{Kernel Ridge Regression.} Similar to SVM, ridge regression can be used as a linear classifier for binary classification:
\begin{align}
    \hat y = sign(w^Tx + b)
\end{align}

However, different from SVM and logistic regression, ridge regression is a technique specifically designed to deal with ill-posed problems with very limited data, by introducing an extra $l_2$ norm of $w$ in the loss function during training. Kernel ridge regression (KRR) is when the same kernel trick as for SVM can be applied to ridge regression.

\bheading{Multi-layer Perceptron (MLP)} is a family of feed-forward neural network models, consisting of an input layer, an output layer and one or more hidden layers in between. Each layer
of neurons linearly transforms $all$ signals from the previous layer, applies a non-linear activation function and outputs to the next layer. Finally, a softmax function \footnote{Softmax($z$) = $ \frac{e^{z_i}}{\sum_i e^{z_i}}$} is applied to output the probabilities of classes. Formally,
\begin{align*}
    h_{1} &= f(W_1^T x+b_1) \\

    &... \\
    h_{n} &= f(W_{n}^T h_{n-1}+b_{n}) \\
    \hat y &= softmax(h_n)
\end{align*}

where $h_i$ denotes the output of layer $i$, $f$ denotes a non-linear activation function, e.g. sigmoid or ReLU \footnote{ReLU($z$) = $z$ if $z\ge 0$, else 0}.
Similar to logistic regression, the weight-matrix $W$'s and the bias $b$'s are trained by minimizing the cross-entropy loss.

\section{Metrics of Accuracy}
\label{apd_sw_metric}

The true negative rate (TNR) is defined as $TNR = \frac{TN}{\vert N\vert} = \frac{TN}{TN+FP}$
where $\vert N\vert$ is the number of samples from the real user, which can be further divided into the true negatives (TN) and the false positives (FP).
A low TNR means that very few users will tend to enable the detection mechanism (low usability).
\iffalse
\begin{equation}
\begin{split}
  &TNR = \frac{TN}{\vert N\vert} = \frac{TN}{TN+FP} \\
\end{split}
\label{eq_tnr}
\end{equation}
\fi

The true positive rate (TPR) is defined as $TPR = \frac{TP}{\vert P\vert} = \frac{TP}{TP+FN}$
where $\vert P\vert$ is the total number of samples from an impostor which is the sum of the true positive number (TP) and the false negative number (FN).
A detection mechanism has little security if its TPR is low.
\iffalse
\begin{equation}
\begin{split}
  &TPR = \frac{TP}{\vert P\vert} = \frac{TP}{TP+FN} \\
\end{split}
\label{eq_tpr}
\end{equation}
\fi

The average accuracy is computed for all the testing samples including the real user and an impostor. When the dataset is balanced, the accuracy is equal to the average of TNR and TPR (see ~\gyrefeq{eq_rpf}). Other three metrics, recall (R), precision (P) and F1 score, are also defined in~\gyrefeq{eq_rpf}.
\iffalse
\begin{equation}
\begin{split}

\end{split}
\end{equation}
\fi
\begin{equation}
\begin{split}
  Accuracy &= \frac{TN+TP}{\vert N\vert+\vert P\vert}

  \approx \frac{TNR+TPR}{2} (if~\vert N\vert \approx \vert P\vert)\\
  Recall &= TPR \\
  Precision &= \frac{TP}{TP+FP} = \frac{\vert P\vert\times TPR}{\vert P\vert\times TPR+\vert N\vert \times FPR}\\

  F1 score &= \frac{2 \times Recall \times Precision}{Recall+Precision}
\end{split}
\label{eq_rpf}
\end{equation}
\section{Pseudo-codes in SID}
\label{apd_pseudo_code}

Cryptography and Security (cs.CR); Signal Processing (eess.SP)