RAPPER: Ransomware Prevention via Performance Counters

Abstract: Ransomware can produce direct and controllable economic loss, which makes it one of the most prominent threats in cyber security. As per the latest statistics, more than half of malwares reported in Q1 of 2017 are ransomware and there is a potent threat of a novice cybercriminals accessing rasomware-as-a-service. The concept of public-key based data kidnapping and subsequent extortion was introduced in 1996. Since then, variants of ransomware emerged with different cryptosystems and larger key sizes though, the underlying techniques remained same. Though there are works in literature which proposes a generic framework to detect the crypto ransomwares, we present a two step unsupervised detection tool which when suspects a process activity to be malicious, issues an alarm for further analysis to be carried in the second step and detects it with minimal traces. The two step detection framework- RAPPER uses Artificial Neural Network and Fast Fourier Transformation to develop a highly accurate, fast and reliable solution to ransomware detection using minimal trace points.

\section{Introduction}
If your organization has not been hit by ransomware yet, there are chances that it will soon be. The number of medium to large-scale business falling prey to ransom payment and extortion of their private databases have increased manifold.
These malicious executables infect the victim machine and demands a ransom amount after encrypting the files and documents of the machine. In May 2017, WannaCry ransomware has affected approx. 200000 business across 150 countries.
Identification, blocking of these ransomwares at the earliest along with recovering the contents of the already encrypted files is already an open challenge.

Hardware Performance Counters (HPCs) were first introduced for checking the static and dynamic integrity of programs, for the purpose of detecting any malicious modifications to them as discussed in~\cite{malone2011hardware}.
While in~\cite{demme2013feasibility} performance counters are used to build a malware detector in hardware.
Detecting malware which modifies the kernel control flow has been targeted in ~\cite{karri:13, wang2016reusing}. The paper uses performance counters to monitor the system calls to detect the vulnerability.
However, detection of ransomwares through the HPCs, to the best of our knowledge, has not been attempted so far. Though the underlying technique is similar~\cite{simha:14}, ransomware detection requires far more accuracy and faster response time to limit the damage.

A range of ransomwares were studied in~\cite{kharraz_2015_dimva}, which identified 15 different ransomware families. It is suggested that despite advancing encryption systems, the prominent ransomwares leave a trait in the access of IO and file-systems. Accordingly, Kharraz $et al.$~\cite{kharraz_2016_usenix_automate} proposed a technique of correlating high file system activity with the intrusion of ransomware, which, however, is susceptible to false positives and also can be defeated with a slow encryption process. Moreover, the technique requires modification in OS kernel, which may not be practical in many real scenarios.
In a recent work, Kiraz $et al.$ presented a technique, where large integer multiplication blocks are identified within an execution~\cite{kiraz_2017_eprint_arith}. Since public-key cryptosystems rely on large integer multiplications, it can detect the threat at an early stage. Similar approaches, for detection of symmetric-key cryptographic primitives via data flow graph isomorphism~\cite{Lestringant_2015_asiaccs_isomorph} or by identifying characteristics of a cipher in a binary code~\cite{grobert_2011_book_binary}, are also presented.
In this paper, neither we target a specific family of ransomwares nor the properties corresponding to a particular cipher implementation. Instead we develop a generic anomaly based approach based on the HPC statistics.
\subsection*{Motivation and Contribution}
The primary contributions of this paper are listed below:
\begin{itemize}{\small\vspace*{-0.1cm}
\item The main objective of RAPPER is to learn the behavior of the system under observation with performance event statistics obtained from HPCs. Unlike other works in literature, which save the templates of malicious processes and matches it on its occurrence, here we allow our tool to learn the normal operating behavior of the system. The time-series data as observed from a selected cluster of HPC events is fed to an Artificial Neural Network to learn the specific characteristics of the data.
\item Any deviation from this normal behavior as learned by the Autoencoder is considered as a suspect to RAPPER. Thus we observe that the performance statistics of the system in presence of ransomware are significantly dissimilar from the normal system behavior because of repeated encryption process.
\item In the course of our study, we identified a benign benchmark application which can raise a false alarm. This is typically a benign process, but because of its high computational overhead the system behavior differ significantly from its normal behavior thus raising a false alarm.
\item Thus in the final step, we transform the time series to frequency domain using
Fast Fourier Transformation (FFT) and understand the repeatability of data with help of a second autoencoder.

}
\end{itemize}
RAPPER is a lightweight tool, which neither requires any root privilege, nor requires any hardware and kernel modification, thereby making it practical to use in almost every environment.

\section{Anomaly Detection by Analysing the System Behavior}
In this section, we first analyze the normal behavior of a system by monitoring some
appropriately selected hardware performance counters in parallel. We then present a
notion of anomalous activity in the system and demonstrate a detailed methodology
for detecting those anomalies by using an \texttt{Autoencoder}.

\begin{figure*}[!t]
	\centering
	\small
	\subfigure[\textbf{\small \# Branch Instructions}]{
		\includegraphics[width=0.28\textwidth]{figs/branch.png}
		\label{fig:observe_branch}}
	~~
	\subfigure[\textbf{\small \# Branch Mispredictions observed}]{
		\includegraphics[width=0.28\textwidth]{figs/branch-mis.png}
		\label{fig:observe_br_miss}
	~~
	\subfigure[\textbf{\small \# Cache Misses encountered}]{
		\includegraphics[width=0.28\textwidth]{figs/cache-mis.png}
		\label{fig:observe_ca_miss}
	\quad
	\subfigure[\textbf{\small \# Cache References observed }]{
		\includegraphics[width=0.28\textwidth]{figs/cache-ref.png}
		\label{fig:observe_cache}
	~~
	\subfigure[\textbf{\small \# Instructions observed}]{
		\includegraphics[width=0.28\textwidth]{figs/instruction.png}
		\label{fig:observe_ins}}\vspace*{-0.5cm}
	\caption{\small Variation of Performance Event Counters from HPCs
		in presence of Wannacry Ransomware\label{fig:perf_dev}
\end{figure*}
\subsection{Observing the system behavior using HPCs}\label{sec:hpc}
The Hardware Performance Counters (HPCs) are a set of special purpose registers built
into modern processors to dynamically observe the hardware related activities in a computer
system. There are some recent works~\cite{karri:13} which use these HPCs to detect
malicious programs targeted for a particular system. The HPCs can be monitored dynamically
with a user privilege using the well-known \texttt{perf} tool, available in Linux kernels 2.6.31+.
One interesting property of the perf tool is that a user can observe the performance counters
associated with a system with some time interval, thereby giving the benefit of observing the
system behavior continuously in a succession of time. The command to monitor a particular
HPC event in such way is as follows:

\begin{center}
	\footnotesize \texttt{perf stat -e <event\_name> -I <time\_interval> <executable\_name>}
\end{center}

Since our objective is to detect the presence of ransomware, which mainly contains an encryption program, typically involving both symmetric and asymmetric key encryptions, we selected the hardware events which are more likely to change because of the
encryptions. The hardware events selected for our study are \texttt{instruction},
\texttt{cache-references}, \texttt{cache-misses}, \texttt{branches}, and \texttt{branch-misses}. The events are self-explanatory by their names.
Generally the symmetric encryption affects the cache based events while the asymmetric encryptions affect the instruction and branching events.

To represent the prototype of a typical system behavior, we designed a watchdog program, and
collected the \texttt{perf stat} values with \texttt{10ms} time interval for that executable.
The effects of all the other processes including the ransomware running in the system will have an impact on the performance
counters. We collected these values at the different point of time in the target system and created
a dataset of regular observation. We articulate that any behavior which is not close to this dataset
is an unusual activity, but may not be a malicious one.

We show the effect of a Ransomware Program (for example, a WannaCry) on the HPCs in Figure~\ref{fig:perf_dev}.
The \textcolor{blue}{blue} lines in Figure represent the effect of normal system programs on the
watchdog executable for different HPCs, whereas the \textcolor{orange}{orange} lines show the effect
of WannaCry ransomware.

An important point to be observed from Figure~\ref{fig:observe_br_miss} is that for a particular time interval the behavior
of WannaCry does not change much from the typical system behavior, for example around time
interval of 100 the effect of WannaCry on the hardware event branch misses is same as normal
system behavior. So, instead of considering individual points for decision making, we select a
window of observations considering each of the five events collectively. Thus, we transform
the problem into anomaly detection in multivariate time-series data.

\subsection{Learning a Time-Series data using an Autoencoder}\label{sec:learn_auto}
To present a generalized ransomware detection strategy we do not model
the behavior of the ransomwares as there can be potential new ransomware
whose behavior is unknown and cannot be modeled. Instead, we model the
normal system behavior, as we can get a majority of such instances. Another
advantage of detecting anomalies by modeling normal behavior is that we do
not need the necessity of labeled dataset as any activity with unusual behavior
crossing a certain threshold can be detected as an anomaly. Thus, we propose
an unsupervised approach to detect these anomalies. The HPC event counts observed over the watchdog application can be considered as the time-series data which is system dependent.
An LSTM (Long-Short-Term-Memory)
based autoencoder can efficiently implement the unsupervised anomaly
detection for time-series, which we discuss below.

Autoencoder is an Artificial Neural Network used for efficient coding of the
input space by unsupervised learning. The primary goal of an autoencoder
is to induce a representation for a set of data by learning an approximate
identity function, i.e., if the input data is $\mathcal{X}$, the goal of the
autoencoder is to learn the function $f$, given by - \hspace{1cm}
	$f : \mathcal{X} \rightarrow \mathcal{X}$

An autoencoder always consists of two mapping, encoding and decoding,
which are given as $\phi$ and $\psi$ respectively.
\begin{gather*}
	\phi : \mathcal{X} \rightarrow \mathcal{F}, \hspace{1cm}
	\psi : \mathcal{F} \rightarrow \mathcal{X}
\end{gather*}
where $\mathcal{F}$ is a vector referring to the decisive intermediate representation learned by the autoencoder,
which is used to regenerate the original input data. The error incurred by the autoencoder
to regenerate the input from vector $\mathcal{F}$ is termed as \texttt{Reconstruction Error},
which is given as below:
\begin{equation}
\mathcal{L} = \| \mathcal{X} - (\psi \circ \phi)\mathcal{X}\|^2
\end{equation}
The learning goal of the autoencoder is to minimize these reconstruction cost for all the input
samples, i.e., to find the mappings $\phi$ and $\psi$ such that $\mathcal{L}$ is minimum.
\begin{equation}
	\label{eq:rec_err}
	\argmin_{\phi, \psi} \mathcal{L} = \argmin_{\phi, \psi} \| \mathcal{X} - (\psi \circ \phi)\mathcal{X}\|^2
\end{equation}
In our case, the input $\mathcal{X}$ is a multivariate time-series sequence, and the
objective is to learn the structure of the sequence. LSTM networks, belonging to a
class of Recurrent Neural Network Model, are typically
used for modeling sequence data, which efficiently handles the dependencies within
the sequence. Hence, we use the LSTM based autoencoder for our detection purpose.
The anomaly detector model first takes a multivariate input sequence ($\mathcal{X}$),
generates an intermediate feature vector ($\mathcal{F}$) related to the sequence,
and then reconstructs the same sequence from the intermediate feature vector. The
autoencoder is trained using all the input sequences by following the objective function
mentioned in Equation~(\ref{eq:rec_err}).

The training dataset is constructed from the observed data for normal system behavior
by taking a window of 100 trace points (i.e., a window trace points collected over 1 second, since each interval
data is collected after 10ms).
Without loss of generality, we have chosen 100 trace points for our experiments.
We shift the window by one time-interval (i.e., 10ms) repeatedly to consider consecutive 100 sample point for learning.
Once the learning of intermediate vector $\mathcal{F}$ is completed, for an anomalous sequence, the autoencoder makes an attempt to reconstruct the original input sequence. Thus, the autoencoder maps it to the normal sequence, based on the intermediate vector $\mathcal{F}$. There is an inherent information loss in this process and hence will incur a substantial \texttt{reconstruction error}. Next, we quantify the amount of error to be incurred by
a process to be termed as an anomaly.
\subsection{Determining Threshold for Decision}\label{sec:threshold}
To quantify the threshold for detecting anomalous activities, we calculate the reconstruction
error distribution ($\mathcal{R}$) for all the training samples. According to the $3\sigma$ rule of thumb,
all the values are considered within three standard deviations of the mean.
Hence, we set the threshold for reconstruction error ($\mathcal{R}_t$) as below.
\begin{equation}
\label{eq:three_sigma}
\mathcal{R}_{t} = \mu_{\mathcal{R}} + 3 * \sigma_{\mathcal{R}}
\end{equation}
where $\mu_{\mathcal{R}}$ and $\sigma_{\mathcal{R}}$ are the mean and standard deviation of distribution $\mathcal{R}$.
In our experimental setup $\mathcal{R}_{t}$ came out to be $0.114$.
\vspace{-0.2cm}
\subsubsection{Anomalous Behaviors of Ransomwares}
	In our study, we considered two ransomware programs - namely \emph{WannaCry} and \emph{Vipasana}
	to show the impact of selecting the threshold $\mathcal{R}_t$ in detecting them as anomalies.
	Figure~\ref{fig:seq_rec_error} shows the sequence of reconstruction errors for both the ransomwares.
	The first point on both the plot appears after observing the first window of 100 time-interval (equivalently
	1 second after the start of execution of the ransomwares). The successive points come after each
	time-interval of 10ms since we are sliding the window by one time-interval for calculating the next
	reconstruction error. The \textcolor{blue}{blue} line indicates the reconstruction errors of each window
	whereas the \textcolor{red}{red} line signifies the threshold $\mathcal{R}_t$ as calculated before.

	\begin{figure}[!t]
		\centering
		\small
		\subfigure[\textbf{\small WannaCry}]{
			\includegraphics[width=0.46\linewidth]{figs/wannacry.png}
			\label{fig:rec_wannacry}}
		\subfigure[\textbf{\small Vipasana}]{
			\includegraphics[width=0.46\linewidth]{figs/vipasana.png}
			\label{fig:rec_vipasana}}\vspace*{-0.5cm
		\caption{\small Sequence of Reconstruction Errors for Ransomware\label{fig:seq_rec_error}}\vspace*{-0.6cm}
	\end{figure}

	We can observe from Figure~\ref{fig:rec_wannacry}, the execution of WannaCry starts behaving like a
	regular program (since the reconstruction error lies well below the threshold value), but the reconstruction
	error shoots over the threshold at $245^{th}$ observation. Thus, the WannaCry
	is detected as anomaly $(1000 + 244*10) = 3440$ ms or $3.44$ seconds after the start of execution.
	Whereas, from Figure~\ref{fig:rec_vipasana}, we can observe that the Vipasana is detected as an anomaly
	at the first window itself, i.e., $1$ second after the start of execution. In both the cases there is an extra
    overhead of time due to the testing time of Autoencoder, which we discuss in Section~\ref{sec:results}.

\section{How good is Reconstruction error as a decider?}
In the previous section, we suggest that a threshold as high as $\mathcal{R}_t$
can be used to determine whether a particular application
behavior deviates from the normal system behavior significantly.
In this section, we explain why a single decision step is not enough
to claim that the anomaly observed is from a malicious process.
	\subsection{Understanding the Ambiguity}
	In order to test the robustness of our detection scheme, we incorporate
	an analysis in presence of SPEC2006 server and multimedia benchmarks.
	We consider the Gshare predictor implementation
	as provided in ($https://www.jilp.org/jwac-2/cbp3\_framework\_\\instructions.html$) and observe the
	HPC sampling counts from a background process exactly like our previous setting. Figure~\ref{fig:perf_dev_com}
    presents the variation of different hardware events in presence of both SPEC benchmarks and WannaCry
    ransomware. We can observe that the execution behavior for both the programs are significantly different
    from the normal observations. Thus, the sequences of data for the SPEC programs may also create considerable
    reconstruction errors. In Figure~\ref{fig:rec_spec}, it clearly shows that the reconstruction error for the sequences
	in presence of the SPEC benchmark programs is above the predetermined threshold at the first
	window itself. Though the error is very close to the threshold, this essentially raises an alarm to
	RAPPER that this benchmark program is a potential malicious	program which deviates in an extent
	from the normal system behavior.
	But surely in this case, it is a false alarm, since the benchmark is composed of
	server and multimedia benchmarks and can be considered as the representative of the
	high computational processes which may deviate highly to the normally running processes in a system.
\begin{figure}[!t]
	\centering
	\small
	\subfigure[\textbf{\small \# Branch Instructions}]{
		\includegraphics[width=0.2\textwidth]{figs/Branches.png}
		\label{fig:com_branch}}
	\quad
	\subfigure[\textbf{\small \# Branch Mispredictions}]{
		\includegraphics[width=0.2\textwidth]{figs/Branch-Misses.png}
		\label{fig:com_br_miss}
	\quad
	\subfigure[\textbf{\small \# cache misses}]{
		\includegraphics[width=0.2\textwidth]{figs/Cache-Misses.png}
		\label{fig:com_ca_miss}
	\quad
	\subfigure[\textbf{\small \# Cache References}]{
		\includegraphics[width=0.2\textwidth]{figs/Cache-References.png}
		\label{fig:com_cache}
	\quad
	\subfigure[\textbf{\small \# Instructions}]{
		\includegraphics[width=0.2\textwidth]{figs/Instructions.png}
		\label{fig:com_ins}}\vspace*{-0.5cm
	\caption{\small Comparison of the Effects on Performance Event Counters from HPCs
		in presence of Wannacry Ransomware and SPEC Benchmark Programs\label{fig:perf_dev_com}}\vspace*{-0.6cm}
\end{figure}

	\begin{figure}[!b]
		\centering
		\includegraphics[width=0.46\linewidth]{figs/spec.png}\vspace*{-0.5cm}
		\caption{\small Sequence of Reconstruction Errors for SPEC Benchmark Programs}\vspace*{-0.4cm}
		\label{fig:rec_spec}
	\end{figure}

\begin{figure*}[!h]
	\centering
	\small
	\subfigure[\textbf{\small FFT of branch Instructions}]{
		\includegraphics[width=0.26\textwidth]{figs/Branches_fft.png}
		\label{fig:fft_branch}}
	\quad
	\subfigure[\textbf{\small FFT of branch mispredictions}]{
		\includegraphics[width=0.26\textwidth]{figs/Branch-Misses_fft.png}
		\label{fig:fft_br_miss}
	\quad
	\subfigure[\textbf{\small FFT of cache misses}]{
		\includegraphics[width=0.26\textwidth]{figs/Cache-Misses_fft.png}
		\label{fig:fft_ca_miss}
	\quad
	\subfigure[\textbf{\small FFT of Cache references}]{
		\includegraphics[width=0.26\textwidth]{figs/Cache-References_fft.png}
		\label{fig:fft_cache}
	\quad
	\subfigure[\textbf{\small FFT of Instructions}]{
		\includegraphics[width=0.26\textwidth]{figs/Instructions_fft.png}
		\label{fig:fft_ins}}\vspace*{-0.5cm
	\caption{\small Variation of Amplitude in frequency domain of the performance counters from HPCs in the presence of SPEC observation and Wannacry Ransomware\label{fig:fft_dev}}\vspace*{-0.4cm}
\end{figure*}
In the next subsection, we perform a transformation from the time domain to the frequency domain to differentiate actual malicious processes from false positives.

\subsection{Introducing Fast Fourier Transformation}
In the second phase of detection using RAPPER, we transform the traces from the
time domain to the frequency domain using the Fast Fourier Transformation(FFT).
FFT is the most efficient way to implement the Discrete Fourier Transformation.
The primary reason to convert the analysis from the time domain to the frequency domain is to understand the repetitive pattern of the traces. The ransomware executable runs encryption repeatedly on multiple files thus it repeats a same set of operations of opening a file, encrypting and closing the file followed by deleting it for multiple files one after another.
The transformation is illustrated in Figure~\ref{fig:fft_dev}, which typically indicates that the amplitude for each frequency bins are constantly higher for the ransomware in contrary to the SPEC benchmark.

We have applied FFT on the time domain values for different hardware events as mentioned in Section~\ref{sec:hpc},
to obtain the frequency domain values. Figure~\ref{fig:fft_dev} presents the FFT plots for the normal system measurements in \textcolor{blue}{blue} lines, along with the SPEC Observations
in \textcolor{green}{green} lines and
WannaCry Ransomware in \textcolor{orange}{orange} lines for different hardware events.
Figure~\ref{fig:fft_dev} shows that for most of the hardware events (apart from the cache misses), the FFT plot behavior of the SPEC benchmark overlaps exactly with the FFT of the normal system behavior.
Also, it is quite clear from the Figure~\ref{fig:fft_branch}, Figure~\ref{fig:fft_br_miss},
Figure~\ref{fig:fft_cache}, and Figure~\ref{fig:fft_ins} that the amplitude of almost all the frequency bins are
higher for WannaCry than the SPEC observation, which is eminent as the WannaCry program repeatedly encrypt
multiple files.

The detection of these variations of amplitudes for different frequency bins can again be considered
as a time-series data, and an LSTM based autoencoder, as discussed before, can be used to detect the
anomaly. The amplitudes for SPEC benchmark programs are very close to that of regular observations
for most of the hardware events. Thus, modeling the FFT data for regular sequences using an autoencoder
will result in reconstruction errors close to the threshold (say $\mathcal{R}^{\prime}_t$) for SPEC benchmarks, and
the error will be much higher in case of ransomwares because of the repeated encryptions.
We modeled an autoencoder as mentioned before and the calculated the threshold $\mathcal{R}^{\prime}_t$
to be $0.033$. Fig~\ref{fig:seq_rec_error_2} presents the sequence of reconstruction errors for both SPEC and WannaCry programs
and we can verify that the reconstruction errors of the SPEC programs always lies below the threshold and thus discarded from
false positives, whereas the reconstruction error of the WannaCry program always remains higher to the threshold.

\begin{figure}[!h]
		\centering
		\small
		\subfigure[\textbf{\small SPEC Benchmark}]{
			\includegraphics[width=0.45\linewidth]{figs/spec_fft.png}
			\label{fig:rec_wannacry_2}}
		\subfigure[\textbf{\small WannaCry}]{
			\includegraphics[width=0.45\linewidth]{figs/wannacry_fft.png}
			\label{fig:rec_vipasana_2}}\vspace*{-0.5cm
		\caption{\small Sequence of Reconstruction Errors for SPEC Benchmark (a) and WannaCry (b)\label{fig:seq_rec_error_2}}\vspace*{-0.4cm}
	\end{figure}
\section{Architecture of RAPPER}
In this section, we present an overview of the architecture of proposed detection methodology - RAPPER. The basic
diagram of the system is shown in Figure~\ref{fig:architecture}. All the experimentation for this study have been performed in a sandbox environment, such that the ransomwares do not affect the actual filesystem.

The architecture contains four modules (\emph{Watchdog Program}, \emph{Autoencoder\_1}, \emph{FFT Converter}, and \emph{Autoencoder\_2}). The detection methodology works in two phases, namely \emph{Offline Phase} and \emph{Online Phase}.
The functioning of each of the module in both the phases are decribed below:
\vspace{-0.2cm}
\subsection{Offline Phase}
In the offline phase, the detection methodology is trained with the normal behavior of the sandbox environment, such that any unusual activity of a ransomware is properly detected in real-time scenario. The functioning of each of the modules in this phase are described below.
\begin{enumerate}{\small
\item \emph{Watchdog Program}: Monitors the HPCs of the Sandbox Environment continuously and forwards a window of data (calculated as described before) to the Autoencoder\_1 and the FFT Converter in parallel.
\item \emph{Autoencoder\_1}: Collects all the data forwarded by the watchdog program and an autoencoder is trained with the dataset as mentioned in Section~\ref{sec:learn_auto}.
\item \emph{FFT Converter}: Converts the Fast Fourier Transformation of each window forwarded by the watchdog program and passes the results to the Autoencoder\_2.
\item \emph{Autoencoder\_2}: Collects all the data passed by the FFT Converter and trains another autoencoder based on the FFT dataset.}
\end{enumerate}

\subsection{Online Phase}
In the online phase, the detection module is deployed in the sandbox system for real-time monitoring to detect
ransomwares. The functioning of each modules in this phase are discussed below.
\begin{enumerate}{\small
\item \emph{Watchdog Program}: Monitors the sandbox system as performed in training phase, and forwards the data
to the Autoencoder\_1 module. In this phase, watchdog program does not forward data to the FFT converter. This
helps us to monitor the system with lower computational cost.
\item \emph{Autoencoder\_1}: Receives sequence window at each time interval from the watchdog program and calculates
the reconstruction error of the sequence. If the error is higher than the predefined threshold $\mathcal{R}_t$, it
sends a signal to the watchdog program to transmit the same window to the FFT Converter.
\item \emph{FFT Converter}: Receives a sequence window from the watchdog module, converts the data into frequency domain,
and forwards the transformed data to the Autoencoder\_2 as done in training phase, but with a condition imposed by the
Autoencoder\_1 module.
\item \emph{Autoencoder\_2}: Calculates the reconstruction error of the received FFT data, and based on the predefined
threshold, $\mathcal{R}^{\prime}_t$, sends a warning whether the sequence belongs to a ransomware or not.}
\end{enumerate}

\begin{figure}[!t]
  \centering
   \includegraphics[width=\linewidth, height = 6cm]{figs/arch.png}\vspace*{-0.4cm}
  \caption{Detection Methodology of RAPPER\label{fig:architecture}}\vspace*{-0.4cm}
\end{figure}

\subsection{Evaluating the performance of RAPPER}\label{sec:results}
We performed all the experiments in a sandbox system having specification \texttt{Intel Core i3 M350} running Ubuntu 16.04 with 4.10.0-38-generic kernel.
We used popular open source python based neural network library \texttt{Keras}~\cite{keras} for the implementation
of both the autoencoder. The architecture used to model the autoencoders are mentioned in Table~\ref{table:architec}. The Dropout Layer is added to
regularize the neural network and prevent from overfitting (Dropout
is a technique where a set of randomly selected neurons are ignored during training).

The distribution of reconstruction errors for regular observations produced by both the
autoencoders are shown in Figure~\ref{fig:threshold}. The thresholds thus calculated using
Equation~\ref{eq:three_sigma} are $0.114$ and $0.033$ respectively.

\begin{figure}[!t]
	\centering
	\small
	\subfigure[]{
		\includegraphics[width=0.45\linewidth]{figs/threshold_brand_new.png}
		\label{fig:threshold1}}
	\subfigure[]{
		\includegraphics[width=0.45\linewidth]{figs/threshold_brand_new_fft.png}
		\label{fig:threshold2}}\vspace*{-0.5cm
	\caption{Distribution of reconstruction errors for (a) Autoencoder\_1 (b) Autoencoder\_2\label{fig:threshold}}\vspace*{-0.4cm}
\end{figure}

The FFT converter usually takes $0.0003$ milliseconds to convert a sequence within a window into frequency domain.
The model building times for Autoencoder\_1 and Autoencoder\_2 are on average $10$ and $14$ minutes respectively. Testing time to calculate whether a single window is an anomaly or not
is $1.321$ milliseconds for Autoencoder\_1 and $1.699$ milliseconds for Autoencoder\_2 respectively. As shown in the Architecture of RAPPER in Figure~\ref{fig:architecture}, the testing of a regular observation only passes through
the Autoencoder\_1, thereby taking only $1.699$ milliseconds, and an anomalous observation passes through all the three modules: Autoencoder\_1, FFT Converter, and Autoencoder\_2, thereby taking $1.321+0.0003+1.699 = 3.0203$ milliseconds to be detected. However, in both the cases, the detection time is less that the sampling interval, which is $10$ milliseconds. Hence, the
detection is performed seamlessly, without the need of any storage buffer, as a new window of data will be created after 10 milliseconds.

Without loss of generality, we measured the performance of RAPPER on most recent WannaCry ransomware. As shown in Section~\ref{sec:threshold}, the WannaCry is detected as an anomaly at the $245^{th}$ window and instantly detected as
ransomware at the same time because it's reconstruction error is always higher than the threshold of Autoencoder\_2. Hence,
the total time taken to detect WannaCry is equal to (Time taken to generate the first window) + $244$ * (time interval for each
sample) + (Autoencoder\_1 testing time) + (Time for single FFT Conversion) + (Autoencoder\_2 testing time)
$=1000+244*10+1.321+0.0003+1.699$ millisecond $=3443.0203$ milliseconds. Thus, WannaCry is detected by RAPPER in approximately $3.443$ seconds. As a sample run with RAPPER, out of 10000 files of approximately 21 bytes each, when the detection stops the execution, 68 files are encrypted. It maybe noted that, the size of a typical file is much larger than 21 bytes, and hence, a lesser number of files will be encrypted.

\section{File Recovery and Conclusions}
RAPPER is thus capable of detecting the presence of ransomwares
fast, as we show for the case of WannaCry within a time
of approximately 4 sec from its launch. Depending on the latency,
the malware can encrypt few files (say $n$). We conclude with a
suggested approach on data retrieval.
A practical solution would be to take backups of
the $n$-recently opened files. After
the lapse of the time quantum required to encrypt these files,
we delete the copies if no ransomware alarm is raised by RAPPER.
This minimizes the storage necessary for the backup files.
To further ensure that the backup files are not encrypted
we perform locking operation, like in linux using {\tt mlock}.

In this paper, thus we explored the effect of ransomware on normal system behaviors. We take the aid of the Artificial Neural Network to detect the presence of ransomwares using a two-step detection framework. The entire detection procedure does not need any template of the malicious process beforehand. Instead it thrives on an anomaly detection procedure to detect the infectious ransomwares in as less as 4 seconds with almost zero false positives, using a frequency analysis.

We also explored the opportunity of applying
side channel techniques to recover the secret key used to
encrypt the files from the perf statistics. Wlog. we
found for ransomwares like WannaCry, each file
is encrypted using AES-128 CBC (Cipher Block Chaining) with a randomly generated distinct key.  These keys are in turn encrypted using an infection
specific RSA public key
and stored in the memory.
It would be indeed a challenging exercise to recover the
AES key by targeting the AES CBC operation.
However, we leave that as a future scope of work.

\begin{table}[!t]
\centering
{\footnotesize
\caption{Model Architecture for Autoencoders} \vspace*{-0.2cm}
\label{table:architec}
\begin{tabular}{|c|c|c|c|}
\hline
\textbf{\begin{tabular}[c]{@{}c@{}}Layer\\ Number\end{tabular}} & \textbf{\begin{tabular}[c]{@{}c@{}}Layer\\ Type\end{tabular}} & \textbf{\begin{tabular}[c]{@{}c@{}}Input\\ Shape\end{tabular}} & \textbf{\begin{tabular}[c]{@{}c@{}}Output\\ Shape\end{tabular}} \\ \hline
\multicolumn{4}{|c|}{\textbf{Autoencoder\_1}} \\ \hline
1 & LSTM & (None, 100, 5) & (None, 100, 32) \\ \hline
2 & Dropout & (None, 100, 32) & (None, 100, 32) \\ \hline
3 & LSTM & (None, 100, 32) & (None, 100, 5) \\ \hline
\multicolumn{4}{|c|}{\textbf{Autoencoder\_2}} \\ \hline
1 & LSTM & (None, 100, 5) & (None, 100, 64) \\ \hline
2 & Dropout & (None, 100, 64) & (None, 100, 64) \\ \hline
3 & LSTM & (None, 100, 64) & (None, 100, 5) \\ \hline
\end{tabular}}\vspace*{-0.4cm}
\end{table}

\bibliographystyle{ACM-Reference-Format}


Adversarial Mask: Real-World Universal Adversarial Attack on Face Recognition Model

Abstract: Deep learning-based facial recognition (FR) models have demonstrated state-of-the-art performance in the past few years, even when wearing protective medical face masks became commonplace during the COVID-19 pandemic. Given the outstanding performance of these models, the machine learning research community has shown increasing interest in challenging their robustness. Initially, researchers presented adversarial attacks in the digital domain, and later the attacks were transferred to the physical domain. However, in many cases, attacks in the physical domain are conspicuous, and thus may raise suspicion in real-world environments (e.g., airports). In this paper, we propose Adversarial Mask, a physical universal adversarial perturbation (UAP) against state-of-the-art FR models that is applied on face masks in the form of a carefully crafted pattern. In our experiments, we examined the transferability of our adversarial mask to a wide range of FR model architectures and datasets. In addition, we validated our adversarial mask's effectiveness in real-world experiments (CCTV use case) by printing the adversarial pattern on a fabric face mask. In these experiments, the FR system was only able to identify 3.34% of the participants wearing the mask (compared to a minimum of 83.34% with other evaluated masks). A demo of our experiments can be found at:this https URL.

\section{\label{sec:intro}Introduction}

For the past two years, the coronavirus has impacted every aspect of our lives, and its impact will continue for the foreseeable future.
Since its emergence, various suggestions have been made to reduce its spread.
While the effectiveness of some actions is questionable, there is no doubt that face masks are a key factor in preventing the spread of the virus in crowded and enclosed spaces.
The widespread adoption of face masks and the ever-increasing use of deep learning-based facial recognition (FR) models in everyday systems can be leveraged to perpetrate targeted adversarial attacks that will enable attackers to evade such models and compromise their robustness, without raising an alarm.

Adversarial attacks in the computer vision domain have gained a lot of interest in recent years, and various ways of fooling image classifiers~\cite{szegedy2013intriguing,goodfellow2014explaining} and object detectors~\cite{sitawarin2018darts,thys2019fooling,zolfi2021translucent} have been proposed.
Attacks against FR systems have also been shown to be effective.
For example, research has demonstrated that face synthesis in the digital domain can be used to fool FR models~\cite{yang2020design}.
In the physical domain, some of the proposed methods involved wearing adversarial eyeglasses~\cite{sharif2016accessorize}, projecting lights on human faces~\cite{shen2019vla}, wearing a hat containing an adversarial sticker~\cite{komkov2021advhat}, and using adversarial makeup~\cite{guetta2021dodging}.
However, the proposed attacks are conspicuous and do not allow the attacker to blend in naturally in real-world scenarios, potentially triggering defense systems.

In this work, we propose a \textit{universal} adversarial attack that can be used to physically evade FR systems; in this case, an adversarial pattern is printed on a fabric face mask, as shown in Figure~\ref{fig:intro}.
To create the adversarial pattern, we use a gradient-based optimization process that aims to cause \textit{all} identities wearing the mask to be misclassified by the FR model.
We first demonstrate the attack's ability to fool state-of-the-art models (e.g., ArcFace~\cite{deng2019arcface}) in the digital domain by applying the face mask to every facial image in the dataset (dynamically) using 3D face reconstruction.
Then, we print the adversarial pattern on an actual fabric face mask and test it under real-world conditions.
The results in the digital domain show that our adversarial mask performs better than all evaluated masks and is transferable to other models.
In the physical domain, we show that 96.66\% of the participants wearing our mask evaded the detection by the FR system.

\begin{figure}[t!]
    \centering
    \includegraphics[width=0.7\linewidth]{figures/intro.png}
    \caption{Illustrating the effect of an adversarial pattern printed on a fabric mask (right), which results in the failure of the FR system to detect the person wearing it, compared to the FR system's ability to detect the same individual without a mask, as well as with a standard disposable mask.}
    \label{fig:intro}
\end{figure}

The contributions of our research can be summarized as follows:
\begin{itemize}
    \item We are the first to present a \textbf{physical universal} adversarial attack that fools FR models, i.e., we craft a single perturbation that causes the FR model to falsely classify all potential attackers as unknown identities, even under diverse conditions (angles, scales, etc.) in a real-world environment (fully-automated CCTV scenario).
    \item In the digital domain, we study the transferability of our attack across different model architectures and datasets.
    \item We present a fully differentiable novel digital masking method that can accurately place any kind of mask on any face, regardless of the position of the head.
    This method can be used for other computer-vision tasks (e.g., training masked-face detection models).
    \item We craft an inconspicuous pattern that ``continues" the contour of the face, allowing a potential attacker to easily blend in with a crowd without raising an alarm, given the variety and widespread use of face masks during the COVID-19 pandemic.
    \item We propose various countermeasures that can be used during the FR model training and inference phases.
\end{itemize}

\section{\label{sec:related}Background \& Related Work}

\subsection{Adversarial Attacks}

\noindent\textbf{Digital Attacks.}
Initially, attacks in the digital domain aimed at fooling classification models were introduced~\cite{szegedy2013intriguing,goodfellow2014explaining}.
While those earlier attacks are based on methods that generate a perturbation for a single image, Moosavi-Dezfooli \textit{et al.}~\cite{moosavi2017universal} proposed universal adversarial perturbations (UAPs), which enable any image that is blended with the UAP to fool a DNN.
Digital attacks on models that perform more complex computer vision tasks (e.g., face recognition and object detection) have also emerged.
Yang~\textit{et al.}~\cite{yang2020design} designed a digital patch which is placed on a person’s forehead to deceive face detectors.
Recent studies targeting FR models suggested various techniques.
Deb~\textit{et al.}~\cite{deb2020advfaces} proposed automated adversarial face synthesis, using a generative adversarial network (GAN) to create minimal perturbations.
Agarwal~\textit{et al.}~\cite{agarwal2018image} and Amada~\textit{et al.}~\cite{amada2021universal} proposed UAPs that can deceive FR models for multiple identities simultaneously.
However, these attacks only call attention to the potential threat inherent to such models but cannot be transferred to the physical world.

\noindent\textbf{Physical Attacks.}
Physical attacks differ from digital attacks in the way real-world constraints are considered throughout the process of generating the perturbation.
Consequently, these constraints allow the perturbations to transfer more easily to the physical world.
In recent years, physical attacks on object detectors have gained attention.
Chen~\textit{et al.}~\cite{chen2018shapeshifter} printed stop signs containing adversarial patterns that evaded detection by the object detector, and Sitawarin~\textit{et al.}~\cite{sitawarin2018darts} deceived autonomous car systems by crafting toxic traffic signs that look similar to the original traffic signs.
Methods against person detectors have also been proposed.
Thys~\textit{et al.}~\cite{thys2019fooling} suggested attaching a small adversarial cardboard plate to a person's body to evade detection.
Continuing this line of research, other studies involved printing adversarial patterns on t-shirts, which resulted in a more realistic article of clothing that blends into the environment more naturally~\cite{wu2020making,xu2020adversarial}.
A slightly different approach, in which the perturbation affects the sensor’s perception of the object by applying a translucent patch on the camera's lens, was also introduced~\cite{zolfi2021translucent}.

Numerous studies have demonstrated different ways of fooling FR systems.
For example, Shen~\textit{et al.}~\cite{shen2019vla} introduced the visible light-based attack, where lights are projected on human faces.
Other studies showed that carefully applied makeup patterns can negatively affect the performance of FR systems~\cite{yin2021adv,guetta2021dodging}.
Accessories were also shown to be effective; for example, Sharif~\textit{et al.}~\cite{sharif2016accessorize} suggested wearing adversarial eyeglass frames that were crafted using gradient-based methods.
Later, GAN methods were used to generate an enhanced version of the adversarial eyeglass frames~\cite{sharif2019general}.
Recently, Komkov~\textit{et al.}~\cite{komkov2021advhat} printed an adversarial paper sticker and placed it on a hat to fool the state-of-the-art \textit{ArcFace}~\cite{deng2019arcface} FR model.
However, when implemented on a person, these methods may call attention to the person by causing them to stand out in a crowd given their unnatural appearance.
In contrast, we propose a method in which the perturbation is placed on a face mask, a safety measure widely used in the COVID-19 era; in addition, unlike prior work in which the proposed attacks craft tailor-made perturbations (target a single image or person), our universal attack can be applied more widely without the need for an expert to train a tailor-made one.
Furthermore, we demonstrate the effectiveness of our method in a real-world use case involving a CCTV system, an aspect not addressed by previous studies.

\subsection{Face Recognition}

\textbf{Models.} FR models can be categorized by two main attributes, the model's backbone and the novel loss function, both of which are involved in the training phase.
The main architecture used as the backbone in these models is the ResNet~\cite{he2016deep} architecture, which varies in terms of the number of layers it contains, also referred to as the backbone \textit{depth}.
On top of the backbone, an additional layer (or more) is added, usually containing a novel loss function that is used to train the backbone weights~\cite{deng2019arcface,wang2018cosface,meng2021magface}.
Later, when the FR model is used for inference, only the backbone layers are used to generate the embedding vector.

\noindent \textbf{Systems.} The end-to-end procedure of a fully automated FR system consists of several main steps: (a) Record - a camera records the environment and then produces a series of frames (a video stream); (b) Detect - each frame is analyzed by a face detector to extract cropped faces; (c) Align - the cropped faces are aligned according to the FR model's alignment method; (d) Embed - the aligned facial images serve as input to an FR model $f$ that maps a facial image $I_{face}$ to a vector $f(I_{face})$, also referred to as an \textit{embedding} vector; (e) Verify - the embedding vector is compared to a list of precalculated embedding vectors (also referred to as ground-truth embedding vectors) using a similarity measure (e.g., cosine similarity). The identity with the highest similarity score is marked as a potential candidate and eventually confirmed if its similarity score surpasses a predefined verification threshold (which depends on the system's use case).
\section{\label{sec:method}Method}

The objective of our research is to generate an adversarial pattern that can be printed on a face mask and cause FR systems to classify a registered identity as an unknown identity.
Further, we aim to create an adversarial pattern that is: (a) universal - it must be effective on any identity from multiple views and angles, and at multiple scales, (b) practical - the pattern should remain adversarial when printed on a fabric mask in the real world, and (c) transferable - it must be effective on different models (backbone depths and loss functions).

\begin{figure}[h]
\centering
    \includegraphics[width=0.7\linewidth]{figures/mask_pipeline.png}
    \caption{Overview of our mask projection method pipeline.}
    \label{fig:mask_placement}
\end{figure}

\subsection{Mask Projection}

In order to digitally train our adversarial mask, we first need to simulate the mask overlay on a person's face in the real world.
Therefore, we use 3D face reconstruction to digitally apply a mask on a facial image.
Feng~\textit{et al.}~\cite{feng2018joint} introduced an end-to-end approach called \textit{UV position map} that records the 3D coordinates of a complete facial point cloud using a 2D image.
This map records the position information of a 3D face and provides dense correspondence to the semantic meaning of each point in the UV space, allowing us to achieve near-real approximation of the mask on the face, which is essential to the creation of a successful adversarial mask in the real world.

More formally, we consider our mask $M_{\text{adv}} \in \mathbb{R}^{w \times h \times 3}$ and a rendering function $\mathcal{R}_\theta$.
The rendering function (partially inspired from~\cite{wang2021facex}) takes a mask $M_{\text{adv}}$ and a facial image $x_{\text{face}}$, and applies the mask on the face, resulting in a masked face image $\mathcal{R}_\theta(M_{\text{adv}}, x_{\text{face}})$.
As shown in Figure~\ref{fig:mask_placement}, the pipeline of the mask's projection on the facial image is as follows:

\begin{enumerate}
    \item Detect the landmark points of the face - given a landmark detector, we extract the landmark points of the face.
    \item Map the mask pixels to the facial image - the landmark points of the face extracted in the previous step of the pipeline are used to map the mask pixels to the corresponding location on the facial images.
    \item Extract depth features of the face - the facial image is passed to the 3D face reconstruction model to obtain depth features.
    \item Transfer 2D facial image to the UV space - the depth features are used to remap the facial image to the UV space.
    \item Transfer 2D mask image to the UV space - the depth features are used to remap the mask image to the UV space.
    \item Augment mask - to improve the robustness of our adversarial mask, random geometric transformations and color-based augmentations (parameterized by $\theta$) are applied: (i) geometric transformations - random translation and rotation are added to simulate possible distortions in the mask's placement on the face in the real world, and (ii) color-based augmentations - random contrast, brightness, and noise are added to simulate changes in the appearance of the mask that might result from various factors (e.g., lighting, noise or blurring caused when the camera captures the image).

    \item Combine and reconstruct - the UV representations of the facial image and the mask are combined, and the combined image is reconstructed back to the regular 2D space, resulting in a masked face image.
\end{enumerate}

\noindent Usually, adversarial attacks that employ textile-like objects (e.g., wearable t-shirt~\cite{wu2020making, xu2020adversarial}) use thin plate splines (TPSs)~\cite{bookstein1989principal} to simulate fabric distortions.
In contrast to these studies, although we aim to craft a textile-based mask, in our case, the mask form on the face remains steady and is not subject to significant distortions.
In addition, our 3D approach allows us to simulate smaller distortions (e.g., caused by the nose shape) without actively using TPSs.

Above all, it is important to note that the entire process presented is completely differentiable and allows us to backpropagate and update the mask pixels.

\subsection{\label{subsec:patch_optimization}Patch Optimization}

To optimize our mask's pixels, we propose an iterative optimization process.
In each iteration, we select a random batch of facial images of multiple identities and digitally project the mask on each facial image.
We then feed the masked face images to the FR model and obtain the embedding representations.
Since our goal is to cause an attacker to be unknown to FR models, we aim to create a patch $M_{adv}$ that will decrease the similarity between the output embedding and the ground-truth embedding $e_{gt}$ (precalculated) for each identity.

More formally, an FR model $f:\mathcal{X}^{w\times h\times 3}\to\mathbb{R}^N$ receives a facial image $x\in\mathcal{X}$ (in our case, a masked face image $\mathcal{R}_\theta(M_{adv},x)$) as input and outputs the embedding representation $f(\mathcal{R}_\theta(M_{adv},x))$.
Therefore, we minimize the cosine similarity between the embedding vectors and use the following loss function:
\begin{equation}
    \ell_{sim}(M_{adv}) = \mathbb{E}_{\theta, x} [\text{cos}(f(\mathcal{R}_\theta(M_{adv},x)), e_{gt})]
    \label{eq:l_sim}
\end{equation}
\noindent Since our method is not system-dependent (i.e., does not use a fixed verification threshold determined by a specific use case), we aim to decrease the similarity to the fullest extent possible, in order to perform the most successful attack.

To improve the mask's transferability to other models, we train our patch using an ensemble of FR models, denoted as $J$.
We replace \ref{eq:l_sim} with the following:
\begin{equation}
    \ell_{sim}(M_{adv}) = \mathbb{E}_{\theta, x} \frac{1}{\vert J\vert}\sum\nolimits_j\text{cos}(f^{(j)}(\mathcal{R}_\theta(M_{adv},x)), e_{gt}^{(j)}),
    \label{eq:l_sim_ens}
\end{equation}
where $f^{(j)}$ denotes the $j^{th}$ model and $e_{gt}^{(j)}$ denotes the embedding representation calculated using the $j^{th}$ model.

We also include the \textit{total variation (TV)}~\cite{sharif2016accessorize} factor to ensure that the optimizer favors smooth color transitions between neighboring pixels and is calculated on the mask pixels as follows:
\begin{equation}
    \ell_{TV} = \sum\nolimits_{i,k}\sqrt{(p_{i,k}-p_{i+1,k})^2 + (p_{i,k}-p_{i,k+1})^2}
\end{equation}
\noindent When neighboring pixels are not similar, the penalty of this component is greater.

To be more precise, since the output of $\ell_{sim}$ is in the range of~$[-1,1]$ and the output of $\ell_{TV}$ is in the range of~$[0, 1]$, we transform $\ell_{sim}$ so it is in the same range ($[0, 1]$); thus, we replace \ref{eq:l_sim_ens} with the following:
\begin{equation}
    \ell_{sim}(M_{adv}) = \mathbb{E}_{\theta, x} \frac{1}{\vert J\vert}\sum\nolimits_j\frac{\text{cos}(f^{(j)}(\mathcal{R}_\theta(M_{adv},x)), e_{gt}^{(j)})+1}{2}
\end{equation}

\noindent Finally, the optimization problem we solve is as follows:
\begin{equation}
    \min\limits_{M_{adv}} [\ell_{sim}(M_{adv}) + \lambda * \ell_{TV}(M_{adv})],
\end{equation}
\noindent where $\lambda$ is set at a low value.

\section{\label{sec:eval}Evaluation}

In our evaluation, we first run experiments in the digital domain by applying the mask to facial images, using the rendering function $R_\theta$ (as explained in Section~\ref{sec:method}).
Then, we evaluate the performance of our adversarial pattern in the physical domain (i.e., real world) by printing it on a fabric mask.

\noindent\textbf{Models.}
We use three different types of loss functions that were originally used to train the models, which are considered state-of-the-art:  ArcFace~\cite{deng2019arcface}, CosFace~\cite{wang2018cosface}, and MagFace~\cite{meng2021magface}.
Specifically, we use pretrained models which were trained using the ArcFace and CosFace loss functions~\cite{an2020partical_fc}, with four different ResNet depths (18, 34, 50, and 100) each, and a pretrained ResNet100 backbone originally trained with the MagFace~\cite{meng2021magface} loss function, for a total of nine different models.
We examine multiple training variations, using one or more (i.e., ensemble) models to train the adversarial mask and then test it in a white-box setting to evaluate the performance.
We also evaluate the transferability of our mask to other unknown models (i.e., black-box setting).

\noindent\textbf{Datasets.}
Throughout this paper, we use three commonly used datasets in the face recognition domain: CASIA-WebFace~\cite{yi2014learning}, CelebA~\cite{liu2018large}, and MS-Celeb~\cite{guo2016ms}.

For the training phase, we randomly choose 100 different identities (50 men and 50 women) from the CASIA-WebFace dataset.
We extract five random facial images for each identity, for a total of 500 facial images

For the evaluation phase, we use 200 identities from each dataset (an equal number of men and women from each dataset), evaluating both the performance on the same distribution (different identities from the CASIA-WebFace dataset, $\sim$20K images) and the transferability to other datasets (CelebA and MS-Celeb, $\sim$6K and $\sim$24K images, respectively).

\noindent\textbf{Metrics.}
In our experiments, we quantify the performance of our attack as the ability to decrease the similarity score - specifically the cosine similarity (an approach originally presented in ~\cite{komkov2021advhat}).
The cosine similarity calculation is a step required prior to making a binary decision based on a predefined threshold.
This evaluation approach does not require a system-dependent predefined threshold and demonstrates our attack's effectiveness.
In the physical domain, we also quantify the effectiveness of our attack using two additional metrics, each of which relates to a different stage of an end-to-end FR system:
\begin{itemize}
    \item \textit{Recognition rate} (RR) = $|F_{rec}| \mathbin{/} |F_{det}|$,
    where $|F_{rec}|$ denotes the total number of frames in which the identity was correctly recognized (the cosine similarity between the ground-truth embedding and the output embedding surpasses the verification threshold), and $|F_{det}|$ denotes the total number of frames in which a face was detected and analyzed by the FR system.
    \item \textit{Persistence detection} - since the goal of our adversarial mask is to ensure that an attacker is not identified by the system, we propose a metric that indicates whether the goal was met.
    An attacker is considered as identified if, within a window of $N_{\text{sliding window}}$ frames, the attacker was recognized in $N_{\text{recognized}}$ frames (where $N_{\text{recognized}}\leq N_{\text{sliding window}})$.
\end{itemize}

\noindent\textbf{Implementation details.}
The models we work with in this research only take size ${3 \times 112 \times 112}$ facial images as input.
Therefore, We set the size of our patch to be $3 \times 60 \times 112$ to avoid significant downsampling when dynamically rendering the mask to the facial image, and we set the initial color of the mask to white.
The pixels are updated using the Adam optimizer~\cite{kingma2014adam}, where the initial learning rate is set at $10^{-2}$.
The weight factor of the TV component in the loss function $\lambda$ is manually set at $0.1$.
The source code is available online.\footnote{ \url{https://github.com/AlonZolfi/AdversarialMask}}

\noindent\textbf{Types of face masks evaluated.}
Since we are the first to present a physical universal perturbation, we compare the effectiveness of our mask with several control masks: (a) Clean - the original facial image without a mask, (b) Adv - our optimized adversarial mask, (c) Random - a mask with randomly colored pixels, and (d) Blue - a standard disposable blue mask (simple black and white masks were also tested and yielded the same results).
In addition, due to our trained mask's resemblance to a human face, the lower face area of a female and male are used as control masks and will be referred to as \textit{Female Face} and \textit{Male Face}, respectively.
The masks compared in our evaluation are shown in Figure~\ref{fig:masks_examples}.

\begin{figure}[t!]
    \centering
    \begin{subfigure}{0.15\linewidth}
        \centering
        \includegraphics[width=0.97\linewidth]{figures/mask_examples/clean1.png}
        \caption{Clean}
    \end{subfigure}
    \begin{subfigure}{0.15\linewidth}
        \centering
        \includegraphics[width=0.97\linewidth]{figures/mask_examples/blue1.png}
        \caption{Blue}
        \label{subfig:digital_blue}
    \end{subfigure}
    \begin{subfigure}{0.15\linewidth}
        \centering
        \includegraphics[width=0.97\linewidth]{figures/mask_examples/random1.png}
        \caption{Random}
    \end{subfigure}
    \begin{subfigure}{0.15\linewidth}
        \centering
        \includegraphics[width=0.97\linewidth]{figures/mask_examples/male1.png}
        \caption{Male}
    \end{subfigure}
    \begin{subfigure}{0.15\linewidth}
        \centering
        \includegraphics[width=0.97\linewidth]{figures/mask_examples/female1.png}
        \caption{Female}
    \end{subfigure}
    \begin{subfigure}{0.15\linewidth}
        \centering
        \includegraphics[width=0.97\linewidth]{figures/mask_examples/adv1.png}
        \caption{Adv}
    \end{subfigure}
    \caption{Examples of facial images w/o mask (a), and when various masks are digitally applied to them (b)-(f).}
    \label{fig:masks_examples}
\end{figure}

\noindent\textbf{Evaluation setup.}
Since the state-of-the-art models discussed above were not specifically designed to address the issue of masked faces, we first examine the model's (ResNet100@ArcFace) performance on a number of simple face masks.
For this evaluation, we use 100 identities from the CASIA-WebFace dataset, where five images of each identity are used to calculate the ground-truth embedding, and the remaining images are applied with different types of masks.

To the best of our knowledge, the scientific community has not reached a consensus on the way in which masked face images should be dealt with by FR models.
Therefore, we use two approaches for generating the ground-truth embedding: (a) the current approach for unmasked face models - averaging the embedding vectors of the original images only, and (b) an extension of the first approach - in addition to the original images, we create a masked face version for each image (the specific mask is randomly chosen from blue, black, and white masks) and average the embedding vectors of the two versions of the images.
We then calculate the cosine similarity between the masked face images' embedding vectors and the two versions of ground-truth embedding vectors generation.

In Table~\ref{tab:mask_performance} we can see that although the first approach (w/o Mask) performs better on unmasked images, its performance on masked images is unsatisfactory.
On the other hand, the cosine similarity for the second approach (w/Mask) only slightly decreases the cosine similarity on unmasked images ($\sim$0.05 decrease) and performs significantly better on masked images ($\sim$0.1-0.15 increase).
Thus, throughout this section the results we present are obtained using the second approach (the ground-truth embedding vectors used for the training procedure are generated using first approach).
It is important to note that by choosing the second approach, we increase the difficulty of deceiving these models, since the ground-truth embedding vectors encapsulate the use of a face mask.

\begin{table}[t!]
\centering
\captionof{table}{Cosine similarity comparison between two ground-truth embedding generation methods on the Resnet100@ArcFace. Bold indicates better performance.}

\begin{tabular}{c|c|cccc}
\hline
                                                                             & \textbf{Mask Type} & No Mask       & Blue          & Black         & White         \\ \hline \hline
\multirow{2}{*}{\textbf{\begin{tabular}[c]{@{}c@{}}Cosine\\ Similarity\end{tabular}}} & w/o Mask*           & \textbf{.732} & .399          & .407          & .428          \\ \cline{2-2}
                                                                             & w/Mask**           & .682          & \textbf{.547} & \textbf{.549} & \textbf{.561} \\ \hline
\end{tabular}
\caption*{
\scriptsize
*Embedding vectors created using original facial images.\\
**A masked version of the original images is added to the embedding calculation.}
\label{tab:mask_performance}
\end{table}

\subsection{Digital Attacks}

We conduct digital experiments to quantify our adversarial mask's effectiveness using the rendering function $R_\theta$ (see Section~\ref{sec:method}), which allows us to dynamically apply masks to the facial images in the test set.

\noindent\textbf{Effectiveness of the adversarial mask in a white-box setting.}
We examine the effectiveness of our attack in a \textit{white-box} setting in which our mask is optimized and tested on the ResNet100@ArcFace.
As shown in Figure~\ref{fig:simbox}, our adversarial mask has a significant impact compared to the no mask case, in which the average cosine similarity decreased from $\sim0.7$ to $\sim0.1$.
As the case of no mask images represents the upper bound of the cosine similarity, we also perform a targeted attack in which a mask is tailored to each person, to determine the lower bound.
The targeted mask results are averaged across all identities in the test set.
We can see that the universal mask performs almost as good as a tailor-made mask ($\sim0.1$ difference).
The tailor-made masks represent an attack that is more difficult to detect, since the adversarial pattern varies among different identities.
In addition, while the female and male face control masks are also able to decrease the cosine similarity to a lower level ($\sim0.45$), our mask outperforms them both for almost all tested identities.

\begin{figure}[t!]
\centering
    \includegraphics[width=0.6\linewidth]{figures/simbox.png}
    \caption{Distribution of the cosine similarity score across different masks. 'Adv Universal' represents our optimized universal mask, and 'Adv Targeted' represents a tailor-made mask for each identity.}
    \label{fig:simbox}
\end{figure}

\noindent\textbf{Transferability across backbone depth.}
We also examine whether our mask can deceive FR models it was not trained on.
Since the majority of the models use the ResNet architecture, we evaluate the performance across different depths of the ResNet@ArcFace.
The results are presented in Figure~\ref{subfig:depth_heatmap}.
In the figure, we can see that the use of our adversarial mask can cause the cosine similarity to decrease regardless of the model used for training.
It can also be seen that our attack generalizes better to unknown models whose architecture depth is closer to that of the trained model.
For example, an adversarial mask trained on a model with 100 layers performs better on the models with 34 and 50 layers (decreasing the cosine similarity to 0.182 and 0.168, respectively) than on the 18-layer model (0.282).
In addition, we see that the mask trained on an ensemble of all models does not outperform a mask trained on a single model in a white-box setting, however the ensemble's effectiveness is seen over all models combined.

\begin{figure}[t!]
    \centering
   \label{fig:transferability}
    \begin{subfigure}{0.48\linewidth}
        \centering
        \includegraphics[width=0.85\linewidth]{figures/depth_heatmap.png}
        \caption{\textbf{Transferability} across various ResNet backbone \textbf{depths} originally trained using the ArcFace loss function.}
        \label{subfig:depth_heatmap}
    \end{subfigure}
    \hspace{8pt}
    \begin{subfigure}{0.48\linewidth}
        \centering
        \includegraphics[width=0.85\linewidth]{figures/head_heatmap.png}
        \caption{\textbf{Transferability} across various ResNet100 backbones originally trained with different \textbf{loss functions}.}
        \label{subfig:head_heatmap}
    \end{subfigure}
    \caption{Transferability experiments measured in terms of cosine similarity.
    Rows are divided into three groups: control masks, masks trained using a single model, mask trained using all of the models.}
\end{figure}

\noindent\textbf{Transferability across different loss functions.}
We further demonstrate the adversarial mask's transferability across different model loss functions.
We use the ResNet100 backbone in which the weights were trained using one of the following loss functions: ArcFace, CosFace, and MagFace.
In Figure~\ref{subfig:head_heatmap}, we observe that our method is loss-agnostic, as the decrease in the cosine similarity is seen on for all tested models.
However, a mask that was trained using the MagFace model does not generalize as well as the masks trained with other models, where the cosine similarity decreased to 0.065 in the white-box setting but only decreased to 0.255 and 0.2 on the ArcFace and CosFace models, respectively.
It is interesting to examine the mask trained by each model (presented in Figure~\ref{fig:head_masks}).
Whereas there is a resemblance in the contour of the optimized masks, the mask trained using the ResNet100@MagFace backbone (Figure~\ref{subfig:magface_head}) learns completely different colors than the other two, in some way providing a possible explanation for its decreased ability to generalize to the ArcFace and CosFace models.

\begin{figure}[t!]
\centering
    \begin{subfigure}{0.18\linewidth}
        \centering
        \fbox{\includegraphics[width=0.94\linewidth,trim={0 0.8cm 0 0.8cm},clip]{figures/arcface100.png}}
        \caption{ArcFace}
        \label{subfig:arcface100_mask}
    \end{subfigure}
    \hspace{0.3cm}
    \begin{subfigure}{0.18\linewidth}
        \centering
        \fbox{\includegraphics[width=0.94\linewidth,trim={0 0.8cm 0 0.8cm},clip]{figures/cosface100.png}}
        \caption{CosFace}
        \label{subfig:cosface_head}
    \end{subfigure}
    \hspace{0.3cm}
    \begin{subfigure}{0.18\linewidth}
        \centering
        \fbox{\includegraphics[width=0.94\linewidth,trim={0 0.8cm 0 0.8cm},clip]{figures/magface100.png}}
        \caption{MagFace}
        \label{subfig:magface_head}
    \end{subfigure}
    \caption{Illustrations of our adversarial masks trained on different ResNet100 backbones, which vary in terms of the original loss function they were trained on.}
    \label{fig:head_masks}
\end{figure}

\noindent\textbf{Transferability across datasets.}
We also find our mask to be effective across different datasets.
In another experiment, we train our mask using images from one of the examined datasets (presented earlier in this section) and study its effectiveness on the other datasets (i.e., the ground-truth embedding vectors are generated using another dataset's images).
We train all of the masks using the ResNet100@ArcFace.
The results show that the impact of using a specific dataset is insignificant, since our mask generalizes over all datasets.
For example, when training the mask on the CASIA-WebFace dataset and testing it on the CelebA and MS-Celeb datasets, we respectively obtained an average cosine similarity of 0.128 and 0.114, similar to the white-box setting results (mask trained and tested on images from the CASIA-WebFace dataset, Figure~\ref{fig:simbox}).

\noindent\textbf{Effect of gender.}
Another aspect we studied is the effect of a specific gender on the trained mask.
The experiments include optimization of the adversarial mask using only female or male identities, and the final masks are presented in Figure~\ref{subfig:female} and Figure~\ref{subfig:male}, respectively.
The results show that even when training the mask on facial images of a single gender, the cosine similarity decreases to the same level as the mask trained on both genders ($\sim0.1$).
In addition, masks trained by a single gender were able to transfer very well to the other gender ($\text{male}\to\text{female}=0.097$, and $\text{female}\to\text{male}=0.145$).

\begin{figure}[t!]
    \centering
    \begin{subfigure}{0.18\linewidth}
        \fbox{\includegraphics[width=0.95\linewidth,trim={0.1cm 0.9cm 0.1cm 0.9cm},clip]{figures/female_mask.png}}
        \caption{Female IDs}
        \label{subfig:female}
    \end{subfigure}
    \hspace{0.3cm}
    \begin{subfigure}{0.18\linewidth}
        \fbox{\includegraphics[width=0.95\linewidth,trim={0.1cm 0.9cm 0.1cm 0.9cm},clip]{figures/male_mask.png}}
        \caption{Male IDs}
        \label{subfig:male}
    \end{subfigure}
    \caption{The adversarial masks trained on the ResNet100@ArcFace using single gender identities.}
    \label{fig:gender}
\end{figure}

Generally, the contour of the trained masks (including the mask trained on both genders, Figure~\ref{subfig:arcface100_mask}) is quite interesting.
Despite the fact that only facial images of female identities were used to train the mask (Figure~\ref{subfig:female}), the optimized mask has an high resemblance to a male face.
More generally, the resemblance of all the trained masks to a male face might indicate there is an underlying bias hidden in these models.

\subsection{Physical Attacks}

Finally, to evaluate the effectiveness of our attack in the real world, we print our digital pattern on two surfaces: on regular paper cut in the shape of a face mask and on a white fabric mask, as shown in Figure~\ref{fig:physical_masks}.
In addition, we create a testbed that operates an end-to-end fully automated FR system (explained in Section~\ref{sec:related}), simulating a CCTV use case.

\noindent \textbf{Setup.} The system contains: (a) a \textit{Dahua IPC-HDBW1431E} network camera which records a long corridor, (b) an MTCNN~\cite{zhang2016joint} detection model for face detection, preprocessing, and alignment, and (c) an attacked model - we perform a white-box attack in which the model used for training the adversarial mask is also the model under attack, a ResNet100@ArcFace.
In addition, we perform an ``offline" analysis in a black-box setting, in which the facial images are cropped from the original frames and compared to ground-truth embedding vectors generated using other models.

To calculate the specific verification threshold (set at 0.38), we use a subset of 1,000 identities from the CASIA-WebFace dataset and perform the following procedure.
Various face masks are applied (digitally) to each identity's original facial images.
Then, we calculate the cosine similarity between the identity's embedding vector and each masked face image.
Since we employ a semi-critical security use case (CCTV), we chose the threshold that led to a false acceptance rate (FAR) of 1\%.
Furthermore, to minimize false positive alarms, we used a persistence threshold of $N_{\text{recognized}}=7$ frames and a sliding window of $N_{\text{sliding window}}=10$ frames to designate a candidate identity as a valid one.

We recruited a group of 15 male and 15 female participants (after approval was granted by the university's ethics committee).
Each participant was asked to walk along the corridor seven times, once with each mask evaluated (clean, blue, random, male face, and female face), similar to the digital experiments, and two more times with our adversarial masks printed on paper and fabric.
The ground-truth embedding of each participant was calculated using two facial images, where a standard face mask was applied (digitally) to each image, for a total of four facial images.

\begin{figure}[t!]
\centering
    \includegraphics[width=0.7\linewidth]{figures/physical_eval.png}
    \caption{Physical experiments' averaged results on all participants across different evaluated masks and different victim models.}
    \label{fig:physical_results}
\end{figure}

\begin{figure}[t!]
    \centering
    \begin{subfigure}{0.18\linewidth}
        \fbox{\includegraphics[width=0.88\linewidth,trim={0.1cm 0.9cm 0.1cm 0.9cm},clip]{figures/arcface100.png}}
        \caption{}
        \label{subfig:surface_digital}
    \end{subfigure}
    \hspace{0.1cm}
    \begin{subfigure}{0.18\linewidth}
        \includegraphics[width=0.98\linewidth]{figures/arcface100_fabric.png}
        \caption{}
        \label{subfig:surface_fabric}
    \end{subfigure}
    \hspace{0.1cm}
    \begin{subfigure}{0.18\linewidth}
        \includegraphics[width=0.98\linewidth]{figures/arcface100_paper.png}
        \caption{}
        \label{subfig:surface_paper}
    \end{subfigure}
    \caption{An illustration of: (a) the digital adversarial mask trained on the ResNet100@ArcFace; (b) the digital pattern printed on fabric mask; and (c) the digital pattern printed on paper.}
    \label{fig:physical_masks}
\end{figure}

\noindent \textbf{Results.} The results of our experiments are shown in Figure~\ref{fig:physical_results} where we can see that our adversarial masks (paper and fabric) performed significantly better than the other masks evaluated on every metric, with a high correlation to the cosine similarity results obtained in the digital domain.

In terms of the RR, the performance of the FR model for the different masks can be divided into four groups (listed in decreasing order): (a) the unmasked version (74.83\%), (b) blue and random masks (53.04\% and 54.76\%, respectively), (c) male and female masks (30.85\% and 28.36\%, respectively), and (d) our fabric and paper adversarial masks (5.72\% and 4.61\%, respectively).

In a realistic case of CCTV use in which an attacker tries to evade the detection of the system, our adversarial fabric mask was able to conceal the identity of 29 out of 30 participants (which represents a persistence detection value of 3.34\%), as opposed to the control masks which were able to conceal 5 out of 30 participants at most (persistence detection value of 83.34\%).

We also examine the effectiveness of our masks on models they were not trained on.
The results presented in Figure~\ref{fig:physical_results} show that our masks have similar adversarial effect on FR models in a black-box setting as in a white-box setting.

Another aspect we examined in our physical evaluation is the ability to print the adversarial pattern on a real surface.
Figures~\ref{subfig:surface_fabric} and \ref{subfig:surface_paper} present the digital adversarial pattern (\ref{subfig:surface_digital}) printed on the different surfaces.
Due to the limited ability of a printer to accurately output the original colors onto the fabric, we can see that there is a slight difference in the performance of the masks.
Nonetheless, both of our adversarial masks outperformed the other masks evaluated.

\section{Countermeasures}
We propose two ways in which our digital masking method can be used to defend against adversarial masks: (a) adversarial training -- adversarial (universal and tailor-made) masked face images could be provided to the model during training to improve its robustness; and (b) mask substitution -- during the inference phase, every masked face image could be preprocessed so that the worn mask is replaced digitally with a standard one (e.g., blue mask~\ref{subfig:digital_blue}), where the models had satisfactory performance, as shown in Section~\ref{sec:eval}, eliminating the potential threat of an adversarial face mask.
An implementation of the mask substitution method on facial images of 100 identities ($\sim10K$ images) from the CASIA-WebFace dataset increased the RR from 0.4\% (the adversarial mask is applied to the facial images) to 65.5\% (the blue mask is applied to the adversarial images).
In a physical experiment, in which the blue mask was digitally placed on facial images extracted from the videos frames (videos of participants wearing the adversarial mask), the RR increased from 5.72\% to 57.3\%.
\section{\label{sec:conclusion}Conclusion}
In this paper, we presented a physical universal attack in the form of a face mask against FR systems.
Whereas other attack methods used different accessories that are more conspicuous and do not blend naturally in the environment, our mask will not raise any suspicion due to the widespread use of face masks during the COVID-19 pandemic.
We demonstrated the effectiveness of our mask in the digital domain, both under white-box and black-box settings.
In the physical domain, we showed how our mask is able to prevent the detection of multiple participants in a CCTV use case system.
Moreover, we proposed possible countermeasures to deal with such attacks.
To sum up, in this research, we highlight the potential risk FR models face from an adversary simply wearing a carefully crafted adversarial face mask in the COVID-19 era.

Computer Vision and Pattern Recognition (cs.CV); Cryptography and Security (cs.CR); Machine Learning (cs.LG)