Stochastic Actor-Executor-Critic for Image-to-Image Translation

Abstract: Training a model-free deep reinforcement learning model to solve image-to-image translation is difficult since it involves high-dimensional continuous state and action spaces. In this paper, we draw inspiration from the recent success of the maximum entropy reinforcement learning framework designed for challenging continuous control problems to develop stochastic policies over high dimensional continuous spaces including image representation, generation, and control simultaneously. Central to this method is the Stochastic Actor-Executor-Critic (SAEC) which is an off-policy actor-critic model with an additional executor to generate realistic images. Specifically, the actor focuses on the high-level representation and control policy by a stochastic latent action, as well as explicitly directs the executor to generate low-level actions to manipulate the state. Experiments on several image-to-image translation tasks have demonstrated the effectiveness and robustness of the proposed SAEC when facing high-dimensional continuous space problems.

\section{Introduction}

Many computer vision problems, such as face inpainting, semantic segmentation, and realistic photo generated from sketch, can be defined as learning image-to-image  translation (I2IT). Currently the most effective I2IT solution is based on a one-step framework that generates images in a single run of a deep learning (DL) model, such as VAE, U-Net, and conditional GANs. Directly learning I2IT with these DL models is challenging, due to the abundance of local minimums and poor generalization caused by overfitting. Although these problems could be  potentially alleviated by using multi-scale models or a multi-stage pipelines, we are still left with models that have intrinsically high complexities, for which the optimal parameters (e.g. stage number and scale factor) have to be determined in a subjective and {\em ad hoc} manner.

To address these limitations of DL-based methods, we explore solving I2IT problems by leveraging the recent advances in deep reinforcement learning (DRL). The key idea is to decompose the monolithic learning process into small steps by a lighter-weight CNN, with the aim of progressively improving the quality of the model. Although recent works have successfully applied DRL to solve several visual tasks \cite{caicedo2015active}, the action space in their tasks is usually discrete and
can not be used for I2IT that requires continuous action spaces.

Currently, a promising direction for learning continuous actions is the maximum entropy reinforcement learning (MERL), which improves both exploration and robustness by maximizing a standard RL objective with an entropy term \cite{haarnoja2018soft}. Soft actor-critic (SAC) \cite{haarnoja2018soft} is an instance of MERL and has been applied to solve continuous action tasks.
However, the main issue hindering the practical applicability of SAC on I2IT is its inability to handle high-dimensional states and actions effectively. Although recent work \cite{yarats2019improving} addresses this problem by combining SAC with a regularized autoencoder (RAE), RAE only provides an auxiliary loss for an end-to-end RL training and is incapable for I2IT tasks.

Besides, a glaring issue is that
the reward signal is so sparse that leads to an unstable training, and higher dimensional
states such as pixels worsen this problem \cite{yarats2019improving}. Thus, training an image-based RL model requires much more exploration and exploitation. One solution to stablize training is to extract a lower dimensional visual representation with a separately pre-trained DNN model, and learn the
value function and corresponding
policy in the latent spaces \cite{nair2018visual}.
However, this approach can not be trained from scratch, which could lead to inconsistent state representations with an optimal policy. Previous works like \cite{lee2020slac} have shown that it is beneficial to learn the image representations and continuous controls simultaneously.

In this paper, we propose a new DRL architecture, {\em stochastic actor-executor critic} (SAEC), to handle I2IT with very high dimensional continuous state and action spaces. SAEC is formed by three core deep neural networks: the actor, the critic, and the executor (see Fig. \ref{dl-rl}).
High level actions are generated by the actor in a low dimensional latent space, and forwarded to the executor for image translation. The critic evaluates the latent actions to guide the policy update. These three neural networks form two different branches in SAEC. More specifically,
the actor and the critic form the RL branch as an actor-critic model, while The actor and executor form the DL branch similar to autoencoder, with an important difference that skip connections are added between the actor and executor.
The RL branch is trained based on the MERL framework, while the DL branch is trained by minimizing a reconstruction objective.

Our contributions can be summarized as follows:

\begin{itemize}

\item A new DRL framework {\em stochastic actor-executor critic} (SAEC) is proposed to handle very high dimensional state and action spaces in I2IT task.

\item Even using a sparse reward signal, our SAEC model can be stably trained from scratch on I2IT problems by combining DL-based supervision.

\item  Our SAEC framework is flexible to incorporate many advanced DL methods for various complex I2IT applications.
And this framework enables the agent to simultaneously learn feature representation, control, and image generation in high-dimensional continuous spaces.
\end{itemize}

\section{Background}

\subsection{Image-to-Image Translation}
Image-to-image translation (I2IT) is often addressed by learning a generative process $G$ that maps state $\bf x$ to target $\bf y$, $G:{\bf x} \rightarrow {\bf y}$. The state should be consistent with the target and both of them are images,  such as generating realistic photos from semantic segmentation labels \cite{isola2017image}, or synthesizing completed visually targets from images with missing regions \cite{pathak2016context}. Autoencoder is leveraged in most research work to learn this process by minimizing the reconstruction error ${\cal L}_{rec}$ between the predicted image $\tilde{\bf y}$ and the target $\bf y$ (see Figure \ref{dl-rl}). In addition, the generative adversarial network (GAN) is also vigorously studied in I2IT to synthesis realistic images \cite{isola2017image}.
Subsequent works enhance I2IT performance by using a coarse-to-fine deep learning framework \cite{yu2018generative} that recursively uses the previous {stage's} output as the input to the next stage (see the bottom of the Fig. \ref{dl-rl}(a)). In this way, I2IT task is transformed into a multi-stage, coarse-to-fine solution. Although iteration can be infinitely applied in theory, it is limited by the increasing model size and training instability.

\subsection{Reinforcement Learning}
Reinforcement learning (RL) is an infinite-horizon Markov decision process (MDP), defined by the tuple $({\cal X},{\cal A}, {P},{r}, {\gamma})$. ${\cal X}$ is a set of states, ${\cal A}$ is action space, and $P$ represents the state transition probability given $x \in {\cal X}$ and ${\bf a}$, $r$ is the reward emitted from each transition, and $\gamma \in [0,1]$ is the reward discount factor. The standard objective of RL is to maximize the expected sum of rewards. Maximum entropy RL (MERL) further adds an entropy term:
$ \sum\nolimits_{t = 0}^T  {{\mathbb{E}_{({\bf x}_t, {\bf a}_t)\sim {\rho}_{\pi} }}[{r_{t} + \alpha {\cal H}(\cdot | {\bf x}_t)}]} $, where $\alpha$ is a temperature parameter, ${\rho}_{\pi}$ denotes the marginal trajectory distribution induced by policy $\pi$. MERL model and has proven stable and powerful in low dimensional continuous action tasks, such as games and robotic controls \cite{haarnoja2018soft}.
However, when facing complex visual problems such as I2IT, where observations and actions are high-dimensional, it remains a challenge for RL models \cite{lee2020slac}.

\begin{figure*}[t]
    \centering
    \includegraphics[scale=0.545]{figs/overview-ijcai.eps}
    \caption{The conventional DL frameworks and our SAEC framework. Top of (a): One-stage DL framework. Bottom of (a): Multi-stage DL framework, whose networks are usually different in each stage. (b) Our SAEC framework. $\bf{x}$ and $\bf{y}$ are input and target, $\tilde{\bf y}$ is the predicted output. $\bf{x_t}$ and $\tilde{\bf y}_t$ are temporary inputs and outputs generated in step (stage) $t$. In the SAEC framework, there are two actions: high-level latent-action ${\bf z}_t$ which is sampled from policy $\pi_\phi$: ${\bf z_t} \sim \pi_\phi({\bf z}_t|{\bf x}_t)$, and low-level image-action $\tilde{\bf y}_t$ which is generated from executor $\eta_\psi$ conditioned by ${\bf z}_t$: $\tilde{\bf y}_t=\eta_{\psi}(\tilde{\bf y}_t|{\bf x_t}, {\bf z_t})$. Both actions are high-dimensional continuous.
    }
    \label{dl-rl}

\end{figure*}

\section{Method}

This work reformulates I2IT as a decision making problem. Instead of directly mapping the input to the target image in a single step, we propose to use a light-weight RL model that performs the translation progressively, where new details can be added as the translation progresses. The proposed model {\em stochastic actor-executor-critic} (SAEC) consists of three components: an actor ($\pi_{\phi}$), an executor ($\eta_{\psi}$), and a critic ($Q_{\theta}$). Specifically, the actor $\pi_{\phi}$ and the executor $\eta_{\psi}$ form a DL pathway that directs the agent learning with a DL-based objective. The same actor and the critic $Q_{\theta}$ form a DRL pathway that works as an actor-critic model.
In SAEC, the actor generates a latent action according to a stochastic policy, so as to capture the required image translation control. The critic evaluates the stochastic policy. The executor leverages the latent action and image details to perform image translation and the generated image is then applied to the environment as the new input image at next time.

\subsection{I2IT as MDP}
\label{sec:i2i-mdp}
The key of our method is to formulate I2IT as a Markov decision process (MDP).
As for state ${\bf x}_t \in {\cal X}$, it is defined task specific.
For realistic photo generation or segmentation, the state ${\bf x}_t$ is the pair of the fixed source image $I_f$ and the moving image $I_{m_t}$: ${\bf x}_t=(I_f, I_{m_t})$. A new state is generated by warping the moving image with predicted image action $\tilde{\bf y}_t$: ${\bf x}_{t+1}=(I_f, \tilde{\bf y}_t \circ I_{m_t})$. For face inpainting, the state is composed of the original image ${\bf x}$ and the synthesized missing part $\tilde{\bf y}_t$. The next state is obtained by adding the new predicted image to the missing area: ${\bf x}_{t+1}=(\tilde{\bf y}_{t} \odot {\bf x}_{t})$.

There are two types of actions in our action space: latent action ${\bf z}_t \in {\cal Z}$ and image action $\tilde{\bf y}_t \in {\cal Y}$. These two actions are intrinsically different, in that $\cal Z$ has a high-level of abstraction and its distribution is unknown. ${\bf z}_t$ is sampled from the stochastic policy of the actor $\pi_{\phi}$, ${\bf z}_t \sim \pi_{\phi}({\bf x}_t)$, while $\tilde{\bf y}_t$ is created by the executor based on  ${\bf z}_t$ and the input image. The reward function is flexible and can be any evaluation metric that measures the similarity between the predicted image $\tilde{\bf y}_t$ and the target ${\bf y}$, such as the peak signal-to-noise ratio (PSNR) and the structural similarity index measure (SSIM).

The sequence of actions are implemented by the {\em stochastic actor-executor-critic} (SAEC) model. Specifically, at each time step $t$, the actor samples a latent action ${\bf z}_t$ from state ${\bf x}_t$, then the executor generates image action $\tilde{\bf y}_t$ from the latent action ${\bf z}_t$ and state ${\bf x}_t$. The change of the state will result in a reward $r_t$, and the new state is fed back to the SAEC model to generate a new image action.
Let ${\cal T}_t$ represents a single run from state ${\bf x}_t$ to image action $\tilde{\bf y}_t$ via an actor and an executor:
${\bf z}_t \sim \pi_{\phi}({\bf x}_t), \tilde{\bf y}_t = \eta_{\psi}({\bf x}_t, {\bf z}_t)$.
The whole process of I2IT in MDP can be formulated as follows:

\begin{equation}
\centering
\label{multi-steps}
    {\cal T}({\bf x}) = {\cal T}_t \circ {\cal T}_{t-1} \circ \dotsm \circ {\cal T}_0({\bf x}),
\end{equation}
where $\circ$ is the function composition operation, ${\cal T}_t$ is the $t$th translation step that predicts image from current state ${\bf x}_t$.

\subsection{Policy with Skip Connections}
\label{sec:su-net}

As in an MERL model, the stochastic policy and maximum entropy in our SAEC improve the exploration for more diverse generation possibilities, which helps to prevent agent from producing a single type of plausible output during training (known as mode-collapse). In addition, one specific characteristic in the SAEC model is that skip connections are added from each down-sampling layer of the actor to the corresponding up-sampling layer of the executor, as shown in Fig. \ref{dl-rl}(b). In this way, a natural looking image is more likely to be reconstructed since the details of state ${\bf x}_t$ can be passed to the executor by skip connections. Besides, since both ${\bf z}_t$ and ${\bf x}_t$ can be used by the executor to generate image action, over-exploration of the action space can be avoided in SAEC, where the variance is limited by these passed detail information.
Furthermore, the skip connections also facilitate back-propagation of the DL gradients to the actor. Our experiments show that if the skip connections are removed, the details of the reconstructed image may lost because of down-sampling (max-pooling) and stochastic policies, and the training would be very unstable in such a situation. Combining skip connections and DL-based learning can avoid detail loss and thus stabilize training.  (See experiments section for more details).

\subsection{Stochastic Actor-Executor-Critic}
\label{sec:saec}

To address I2IT, We propose to extend the maximum entropy RL (MERL) framework with an additional executor. Our proposed model has a similar structure with SAC \cite{haarnoja2018soft}, but the latter cannot be applied to the I2IT tasks directly since the delayed and sparse rewards making it difficult to handle the inherent high-dimensional states and actions. To this end, we propose to train the actor and executor in a conventional DL-based manner,
where the training data are collected from the agent-environmental interaction and change dynamically in experience replay. The objective of SAEC is to learn a policy $\pi_{\phi}$ and an executor $\eta_{\psi}$, by maximizing both the conditional likelihood and the expected sum of rewards with an entropy ${\cal H}$.

\begin{algorithm}[t]
    \caption{Stochastic Actor-Executor Critic}\label{alg:sac}

    \begin{algorithmic}
        \Require Environment $ Env $ and initial parameters $\theta_1, \theta_2 $ for the Critics, $\phi, \psi$ for the Actor and Executor.

            \State $\bar{\theta}_1 \gets \theta_1, \bar{\theta}_2 \gets \theta_2, \cal D \gets \varnothing$

            \For {each iteration}
                \State ${\bf x}, {\bf y} \sim Env_{reset}()$

                \For {each environment step}
                    \State ${\bf z}_t \sim \pi_{\phi}({\bf z}_t|{\bf x}_t)$

                    \State $\tilde{\bf y}_t \gets \eta_{\psi}({\bf x}_t, {\bf z}_t)$

                    \State ${\bf x}_{t+1}, r_t \sim Env_{step}(\tilde{\bf y}_t)$

                    \State $\cal D \gets {\cal D}$ $\cup$ $\{({\bf y}, {\bf x}_t, {\bf z}_t, r_t, {\bf x}_{t+1})\}$

                \EndFor

                \For {each gradient step}
                    \State Update Actor and Executor (DL guided):
                     \State \qquad$\phi \gets \phi - \lambda_{DL} \hat{\nabla}_{\phi} {\cal L}_{DL}(\phi)$

                     \State \qquad$\psi \gets \psi - \lambda_{DL} \hat{\nabla}_{\psi} {\cal L}_{DL}(\psi)$

                    \State Update Actor and Critic (RL guided):

                     \State \qquad$\theta_i \gets \theta_i - \lambda_Q \hat{\nabla}_{\theta_i} J_Q(\theta_i)$ for $i \in \{1,2\}$

                     \State \qquad$\phi \gets \phi - \lambda_{\pi} \hat{\nabla}_{\phi} (J_{\pi}(\phi))$

                     \State \qquad$\alpha \gets \alpha - \lambda \hat{\nabla}_{\alpha} J(\alpha)$

                     \State \qquad$\bar{\theta}_i \leftarrow \tau \theta_i + (1-\tau)\bar{\theta}_i$ for $i \in \{1,2\}$

                \EndFor

            \EndFor
        \Ensure $\theta_1, \theta_2, \phi, \psi $

    \end{algorithmic}
\end{algorithm}

\subsubsection{Conditional Generative Model Learning}
The actor and the executor form a conditional generative process that translates the current state ${\bf x}_t$ to target ${\bf y}$. Instead of learning the generative model alone like a VAE, we maximize the conditional likelihood following the stochastic policy $\pi_{\phi}$:

\begin{equation}
\centering
\label{likelihood}
    p({\bf y}|{\bf x}_t)=\int \pi_{\phi}({\bf z}_t|{\bf x}_t)\eta_{\psi}({\bf y}|{\bf x}_t, {\bf z}_t)d{\bf x}_t.
\end{equation}
The empirical objective is to minimize the reconstruction error over all samples:

\begin{equation}
\centering
\label{rec_loss}
    {\cal L}_{rec}=\mathbb{E}_{{\bf x}_t,{\bf y} \sim {\cal D}}[\left \| {\cal T}({\bf x}_t),{\bf y}\right \|_d],
\end{equation}
where $\cal D$ is a replay pool, $\left \| \cdot \right \|_d$ denotes some distance measures, such as $L_1$ and $L_2$.
To synthesis more realistic images, we also extend the actor-executor to a high quality generative model by jointly training an adversarial loss with discriminator $D$:

\begin{equation}
\centering
\label{adv_loss}
    {\cal L}_{adv}= \mathbb{E}_{{\bf x}_t,{\bf y} \sim {\cal D}}[\log (D({\bf y})) + \log (1-D({\cal T}({\bf x}_t)))]
\end{equation}
Our final DL guided objective can be expressed as
\begin{equation}
\centering
\label{dl-loss}
    {\cal L}_{DL} = {\lambda}_{rec} {\cal L}_{rec} + {\lambda}_{adv} {\cal L}_{adv},
\end{equation}
where ${\lambda}_{rec}$ and ${\lambda}_{adv}$ are used to balance the reconstruction and adversarial learning. In general, the DL pathway is very flexible and can readily leverage any other advanced DL losses or techniques into this framework.

\subsubsection{MERL in I2IT}
In RL pathway, the rewards and the soft Q values are used to iteratively guide the stochastic policy improve. Moreover, the latent-action ${\bf z}_t$,
is used to estimate soft state-action value for encouraging high-level policies. Let actor and critic with parameters $\phi$ and $\theta$ be the function approximators for the policy $\pi_\phi$ and the soft Q-function $Q_\theta$, respectively. The critic parameters $\theta$ are trained to minimize the soft Bellman residual:
\begin{equation}
\centering
\label{sac-q}
\begin{aligned}
    {J_Q}({\theta}) & = \mathbb{E}_{({\bf x}_t, {\bf z}_t) \sim {\cal D}} \left[\frac{1}{2} (Q_{\theta}({\bf x}_t, {\bf z}_t) - {\hat Q}_{\bar{\theta}}({\bf x}_{t}, {\bf z}_{t}))^2\right],
\end{aligned}
\end{equation}

\begin{equation}
    \begin{aligned}
        {\hat Q}_{\bar{\theta}}({\bf x}_{t}, {\bf z}_{t}) &= r_t + \gamma \mathbb{E}_{{\bf x}_{t+1} \sim {P}}\left[V_{\bar{\theta}}({\bf x}_{t+1})\right], \notag
    \end{aligned}
\end{equation}
where $V_{\bar{\theta}}({\bf x}_{t})$ is the soft state value function which can be computed by

\begin{equation}
    \begin{aligned}
        V_{\bar{\theta}}({\bf x}_{t})  = \mathbb{E}_{{\bf z}_{t} \sim \pi_\phi}\left[Q_{\bar{\theta}}({\bf x}_{t}, {\bf z}_{t}) - \alpha \log \pi_\phi ({\bf z}_{t}|{\bf x}_{t})\right].
    \end{aligned}
\end{equation}
We use a target network $Q_{\bar{\theta}}$ to stabilize training, whose parameters $\bar{\theta}$ are obtained by an exponentially moving average of parameters of the critic network: $\bar{\theta} \leftarrow \tau \theta + (1-\tau)\bar{\theta}$.
The actor parameters $\phi$ are optimized to learn the policy towards the exponential of the soft Q-function:

\begin{equation}
\centering
\label{sac-pi}
    {J_\pi}({\phi}) = \mathbb{E}_{{\bf x}_t \sim {\cal D}} \left[\mathbb{E}_{{\bf z}_t \sim \pi_\phi}\left[\alpha \log (\pi_\phi({\bf z}_t|{\bf x}_t)) - Q_\theta({\bf x}_t, {\bf z}_t)\right]\right]
\end{equation}
In practice, we use two critics with the same structure but different parameters, ($\theta_1$, $\theta_2$), to mitigate positive bias in policy improvement and accelerate training. Moreover, we automatically tune the temperature hyperparameter $\alpha$ by
\begin{equation}
    \begin{aligned}
        {J(\alpha)} = \mathbb{E}_{{\bf z}_t \sim \pi_t}[-\alpha \log \pi_t({\bf z}_t|{\bf x}_t) - \alpha \bar{{\cal H}}],
    \end{aligned}
\end{equation}
where $\bar{{\cal H}}$ represents a constant entropy that equals to the negative of the dimension of the latent action spaces.

The complete algorithm of SAEC is described in Algorithm \ref{alg:sac}. The training process alternates between learning I2IT from DL pathway and policy control from  RL pathway. The joint training of DL and RL ensures a fast convergence since the supervised or unsupervised loss is a "highway" to facilitate back-propagation of the gradients to the actor and the executor. As the latent-action ${\bf z}_t$ is concatenated into critic (shown in Figure \ref{dl-rl}(b)), the RL objective and DL objective can cooperate to train the stochastic actor fast and stable.

\section{Experiments}

In this section, we demonstrate experimentally the effectiveness and robustness of the SAEC framework on several different tasks of I2IT.

\subsection{Face Inpainting}

 \noindent\textbf{Settings}
In this experiment, we apply SAEC to the problem of face inpainting, which is to fill in pixels in the central area of a face image with synthesized contents that are semantically consistent with the original face and at the same time visually realistic. Celeba-HQ dataset is used in this study, of which $28,000$ images are used for training and $2,000$ images are used for testing. All images have a missing part of size of 64 $\times$ 64 cropped in the center.
We compare the SAEC
with several recent face inpainting methods, including CE \cite{pathak2016context}, CA \cite{yu2018generative}, PEN \cite{zeng2019learning}, PIC \cite{zheng2019pluralistic} and RN \cite{yu2020region}.

The actor-executor for our methods uses the same network structure as the encoder-decoder for CE. The network structures of the discriminator also comes from CE but with a different SNGAN loss. Following the previous work \cite{pathak2016context,yu2018generative,zheng2019pluralistic}, we use PSNR and SSIM as the evaluation metrics.

\begin{figure*}[t]
    \centering

    \includegraphics[scale=0.54]{figs/fi_ssim.eps}
    \caption{Visual comparison of face inpainting over all methods. Our SAEC use SNGAN for auxiliary DL guided learning. $\#$ indicates what reward is used for RL training. Our results have better visual quality even for the large pose face.}
    \label{fi-compare}
\end{figure*}

 \noindent \textbf{Results and Analysis}
As shown in Table \ref{tab:psnr}, by using $L_1$ and adversarial loss in the DL guided learning, our method achieves the best PSNR and SSIM scores comparing with the existing state-of-the-art methods. And as mentioned in the Method section, the reward function is very flexible for our RL framework. Both the PSNR and SSIM based reward are suitable for SAEC and can improve the performance on face inpainting.
The qualitative comparison shown in Figure \ref{fi-compare} illustrates that the SAEC gives obvious visual improvement for synthesizing realistic faces.
The results of SAEC are very reasonable, and the generated faces are sharper and more natural. This may attribute to the high-level action ${\bf z}_t$, which focuses on learning the global semantic structure and then directs the executor with DL guided learning to further improve the local details of the generated image. We can also see that the synthesised images of the SAEC could have very different appearances from the ground truth, which indicates that although our training is based on paired images, the SAEC can successfully explore and exploit data for producing diverse results.

\begin{table}
\centering
\begin{tabular}{lcc}
\toprule
\multicolumn{1}{l}{Method} & PSNR $\uparrow$ & SSIM $\uparrow$ \\ \midrule
CE \cite{pathak2016context}                         & 26.949 & 0.870     \\
CA \cite{yu2018generative}                  & 26.608  & 0.871    \\

PIC \cite{zheng2019pluralistic}                      & 26.307    & 0.893  \\
PEN \cite{zeng2019learning}                      & 26.391    & 0.883  \\
RN \cite{yu2020region}                      & 26.932    & 0.892  \\
\midrule
SAEC (SNGAN + PSNR reward)  & 27.176   & 0.882 \\
\textbf{SAEC (SNGAN + SSIM reward)}           & \textbf{27.327}    & \textbf{0.896} \\
\bottomrule
\end{tabular}
\captionof{table}{Quantitative results of all methods on Celeba-HQ.}
\label{tab:psnr}
\end{table}

\noindent \textbf{Ablation Study}
To illustrate the stability of training GANs in SAEC, we jointly use $L_1$ and several advanced GAN techniques i.e. WGAN-GP \cite{gulrajani2017improved}, RaGAN \cite{jolicoeur2018relativistic},
and SNGAN \cite{miyato2018spectral} for DL guided learning. To validate the effectiveness of RL pathway in SAEC for I2IT, we also separately train an actor-executor model (AE) by jointly optimizing the $L_1$ and SNGAN loss. The results shown in Table \ref{tab:fi-gans} indicate that the proposed SAEC with different GANs are stable, and significantly improve the performance of training AE with SNGAN alone, which further demonstrates the contribution of the RL formulation.

\subsection{Realistic Photo Translation}

\noindent \textbf{Settings}
In this section, we evaluate the SAEC on a set of general I2ITs.
We select three datasets of realistic photo translation including:

\begin{compactitem}

\item CMP Facades dataset for segmentation {\em labels$\rightarrow$images} \cite{tylevcek2013spatial}.

\item Cityscapes dataset for segmentation {\em labels$\rightarrow$images} and {\em images$\rightarrow$labels} \cite{Cordts2016Cityscapes}.

\item Edge and shoes dataset for {\em edges$\rightarrow$shoes} \cite{yu2014fine}.

\end{compactitem}
In these tasks, SAEC uses the same network structure as in face inpainting experiment with PSNR reward and SNGAN loss. We compare our method with pix2pix \cite{isola2017image} and PAN \cite{wang2018perceptual}.
We also compared with pix2pixHD \cite{wang2018high}, DRPAN \cite{wang2019drpan} and CHAN \cite{gao2021complementary}, which are designed for high-quality I2IT. Moreover, we replace MERL with PPO \cite{schulman2017proximal} in the RL pathway as SAEC$^*$. We use PSNR, SSIM and LPIPS \cite{zhang2018unreasonable} as the evaluation metrics.

\begin{table}[t]
\centering
\begin{tabular}{lcc}
\toprule
\multicolumn{1}{l}{Method} & PSNR $\uparrow$  & SSIM $\uparrow$  \\ \midrule
AE + SNGAN         & 26.884   & 0.871  \\
SAEC (+ WGAN-GP)        &27.091   & 0.875 \\
SAEC (+ RaGAN)        & 27.080    & 0.873\\
\textbf{SAEC (+ SNGAN)}                        & \textbf{27.176}   & \textbf{0.882}   \\
\bottomrule
\end{tabular}
\captionof{table}{Quantitative results of the variants methods on Celeba-HQ testing dataset (all trained with PSNR reward).}
\label{tab:fi-gans}
\end{table}

\begin{figure*}[t]
    \centering

    \includegraphics[scale=0.54]{figs/label2ims1.eps}
    \caption{Visual comparison of our method with pix2pix, PAN, pix2pixHD and DRPAN over all tasks.}
    \label{l2i}
\end{figure*}

\begin{table*}[t]
\renewcommand\arraystretch{1.1}
\centering
\scalebox{0.90}{
\begin{tabular}{lcccccccccccc}

\toprule
\multirow{3}{*}{Method} & \multicolumn{3}{c}{Facades label$\rightarrow$image}   &  \multicolumn{3}{c}{Cityscapes image$\rightarrow$label}   &
\multicolumn{3}{c}{Cityscapes label$\rightarrow$image}   &\multicolumn{3}{c}{Edges$\rightarrow$shoes} \\
 & PSNR  & SSIM  & LPIPS $\downarrow$  & PSNR & SSIM & LPIPS $\downarrow$ & PSNR  & SSIM & LPIPS $\downarrow$ & PSNR & SSIM & LPIPS $\downarrow$ \\ \midrule

pix2pix         & 12.290   & 0.225 & 0.438 & 15.891   & 0.457 & 0.287  &15.193  & 0.279 & 0.379 & 15.812   & 0.625 & 0.279  \\

PAN   & 12.779     & 0.249  & 0.387  & 16.317     & 0.566    & 0.228 & 16.408   & 0.391  &0.346  &16.097 &0.658 & 0.228\\

pix2pixHD   & 12.357     & 0.162  & 0.336  & 17.606     & 0.581    & 0.204 & 15.619   & 0.361  & \textbf{0.319}  &17.110 &0.686 & 0.220\\

DRPAN   & 13.101     & 0.276  & 0.354  & 17.724     & 0.633    & 0.214 & 16.673   & 0.403  &0.343  &17.524 &\textbf{0.713} & 0.221\\

CHAN   & 13.137    & 0.231   & 0.402  & 17.459    & 0.641   & 0.222 & 16.739    & 0.401   & 0.373  & 18.065    & 0.692   & 0.236\\

\midrule
SAEC$^*$   & 13.163     & \textbf{0.308}  & 0.366  & 17.168     & 0.616    & 0.221 & 16.685   & 0.410  &0.362  &16.914 &0.695 & 0.225\\
\textbf{SAEC}   & \textbf{13.178}   & 0.296  & \textbf{0.324} & \textbf{17.969}   & \textbf{0.659}  & \textbf{0.203}  & \textbf{16.848}  & \textbf{0.412}  & 0.337  & \textbf{18.178}   & 0.698 & \textbf{0.215} \\
\bottomrule
\end{tabular}}
\captionof{table}{Quantitative results of our SAEC with other methods over all datasets. $\downarrow$ means lower is better, SAEC$^*$ means using PPO.}
\label{tab:pix2pix}
\end{table*}

\noindent \textbf{Quantitative Evaluation}
The quantitative results are shown in Table \ref{tab:pix2pix}. With a similar network structure, the proposed method significantly outperforms the pix2pix and PAN model on PSNR, SSIM and LPIPS over all datasets and tasks. The SAEC even achieves a comparable or better performance than the high-quality pix2pixHD and DRPAN model, which have much more complex architectures and training strategies. Moreover, using MERL instead of PPO obviously improves performance on most tasks. These experiments illustrate that the proposed SAEC is a robust and effective solution for I2IT.

\begin{figure*}[!h]
\centering

\subfigure{
\includegraphics[width=0.24\textwidth]{{figs/facades2_.eps}}
\includegraphics[width=0.24\textwidth]{figs/cityscapes_image2label2_.eps}
 \includegraphics[width=0.24\textwidth]{figs/cityscapes_label2image2_.eps}
 \includegraphics[width=0.24\textwidth]{figs/edges2shoes2_.eps}
}

\caption{Learning curves on different I2IT tasks. SAEC performs consistently better than other modified RL algorithms.}
\label{curves}
\end{figure*}

\noindent \textbf{Qualitative Evaluation}
The qualitative results of our SAEC with other I2IT methods on different tasks are shown in Figure \ref{l2i}. We observe that the pix2pix and PAN sometimes suffer from mode collapse and yield blurry outputs. The pix2pixHD is unstable in different datasets especially on Facades and Cityscapes.
The DRPAN is more likely to produce blurred artifacts in several parts of the predicted image on Cityscapes.
In contrast, the SAEC produces more stable and realistic results. Using stochastic policy and MERL helps to explore more possible solutions so as to seek out the best generation strategy by trial-and-error in the training steps, leading to a more robust agent for different datasets and tasks.

\noindent \textbf{Comparison with other RL Algorithms}
The key components of SAEC are substituted by other structures or other state-of-the-art RL algorithms to test their importance.
We use DDPG and PPO to demonstrate the effectiveness of stochastic policy and maximum entropy RL, respectively.
The learning curves of different variants on four tasks are shown in Figure \ref{curves}.
By using the stochastic policy and the maximum entropy framework, the training is significantly improved.

\section{Conclusion}

In this paper, we present a new RL framework for high-dimensional continuous control and apply it to I2IT problems. This framework contains a {\em stochastic actor-executor critic} (SAEC) structure that handles high dimensional policy control, high-level state representation, and low-level image generation simultaneously. We train SAEC by jointly optimizing the RL and the DL objective, which provides ubiquitous and instantaneous supervision for learning I2IT. Our empirical results on several different vision applications demonstrate the effectiveness and robustness of the proposed SAEC. We will subsequently extend this work to more computer vision tasks involving with high-dimensional continuous spaces.

\section*{Acknowledgements}
This work was supported in part by the National Natural Science Foundation of China under Grant 61602065, Sichuan province Key Technology Research and Development project under Grant 2021YFG0038.

Computer Vision and Pattern Recognition (cs.CV)


Machine Learning-based Ransomware Detection Using Low-level Memory Access Patterns Obtained From Live-forensic Hypervisor

Abstract: Since modern anti-virus software mainly depends on a signature-based static analysis, they are not suitable for coping with the rapid increase in malware variants. Moreover, even worse, many vulnerabilities of operating systems enable attackers to evade such protection mechanisms. We, therefore, developed a thin and lightweight live-forensic hypervisor to create an additional protection layer under a conventional protection layer of operating systems with supporting ransomware detection using dynamic behavioral features. The developed live-forensic hypervisor collects low-level memory access patterns instead of high-level information such as process IDs and API calls that modern Virtual Machine Introspection techniques have employed. We then created the low-level memory access patterns dataset of three ransomware samples, one wiper malware sample, and four benign applications. We confirmed that our best machine learning classifier using only low-level memory access patterns achieved an $F_1$ score of 0.95 in detecting ransomware and wiper malware.

\section{Introduction}

Ransomware is a type of malware that encrypts or exfiltrates victims' files to demand ransom. Many cyber-criminals create new ransomware variants using Ransomware-as-a-Service (RaaS) and ransomware toolkit, and they can easily evade protections of anti-virus software using the built-in obfuscators and packers \cite{sharmeen2020avoiding}. Although anti-virus software vendors frequently update their signature database used to detect binary files of the variants, the signature-based static analysis detection fundamentally cannot cope with a large number of the variants. Beaman et al. presented a literature review on recent state-of-the-art ransomware prevention and detection approaches \cite{beaman2021ransomware}. They analyzed popular ransomware samples and developed their experimental ransomware, AESthetic, that was able to evade detection against eight popular anti-virus programs. Their experiment highlights that current anti-virus software relies heavily on signature-based static analysis detection. They claimed that vendors should invest more into the approaches seen in recent literature, especially in dynamic analysis using behavioral patterns of ransomware.

Another problem of the current cyber-defense landscape is the evasion techniques of the conventional protection mechanisms. Many vulnerabilities of the complex software of modern OSs enable cyber-criminals to evade the detection mechanisms. Therefore, a more resilient multiple-layered ransomware detection mechanism is needed to address the advanced evasion techniques of the state-of-the-art malware.
Microsoft, for example, has introduced Virtualization Based Security, or VBS, in the latest products beginning with Windows 10 and Windows Server 2016 \cite{2021windowsinternals}. Microsoft's VBS uses the Windows hypervisor to create an additional protection layer under the conventional OS-based protection layer. Even when attackers gain access to the OS kernel, the extra protection layer limits the possible exploits. The Hypervisor-protected Code Integrity (HVCI) function of VBS, for example, can check whether the attacker modified the OS kernel and device driver code or not. Even if attackers evaded the first protection layer of an operating system (e.g., anti-virus, sandbox, and firewall), the following protection layer of a hypervisor (e.g., Microsoft's VBS, a hypervisor-based monitoring system presented in this paper) can detect the advanced cyber attacks.

\subsection{Relationship with previous studies}
\label{sec:previous}

Shinagawa et al. released an open-source lightweight hypervisor named BitVisor that transparently enforces security functions to a guest OS \cite{bitvisor}. Although Microsoft's VBS employs a multiple guest OSs model and enforces security functions from a privileged guest OS to a guest OS, BitVisor employs a single guest OS model and enforces security functions directly from the hypervisor to a guest OS. The single guest OS model of BitVisor reduces the performance overhead of a hypervisor-based security mechanism. Hirano et al. presented a BitVisor-based live-forensic hypervisor for collecting low-level storage access patterns \cite{hirano2017waybackvisor}. They showed that the low-level storage access patterns, the behavioral features obtained from the hypervisor, can classify ransomware samples from benign applications using conventional machine learning models \cite{Hirano2019}. The low-level behavioral features of storage access patterns consist of variance of the accessed sector's logical block address, the entropy of written sectors, and throughput on a serial ATA (Advanced Technology Attachment) storage device (e.g., Solid State Drives, SSDs). The proposed system achieved the average $F_1$ score of 96.2\% in detecting ransomware and 94.1\% in detecting ransomware variants. Furthermore, Hirano et al. released the open dataset of the low-level ransomware storage access patterns for researchers who are interested in constructing machine learning models for ransomware detection \cite{hirano2022ransap}. In this paper, we added a novel monitoring function of memory access patterns to the live-forensic hypervisor presented in the previous papers \cite{hirano2017waybackvisor,Hirano2019,hirano2022ransap}.

\subsection{Contributions of this paper}
\label{sec:contribution}

This paper presents a novel machine learning-based ransomware detection method using low-level memory access patterns. The contributions of this paper are as follows.

\begin{itemize}
    \item  We developed a thin and lightweight live-forensic hypervisor that collects low-level memory access patterns with mitigating advanced evasion techniques of malware that exploit OS vulnerabilities.
    \item While many modern ransomware detection methods presented in literature use dynamic features obtained from an operating system layer \cite{beaman2021ransomware} (e.g., sequences of API calls per process ID, file system operations), our system uses only low-level memory access patterns of physical address space obtained from a hypervisor layer.
    \item Many researchers believe that Virtual Machine Introspection (VMI), in general, needs to bridge the semantic gap between OS and hypervisor \cite{more2014virtual}. For example, low-level memory access patterns on RAM alone cannot determine which process or who accessed the memory region. This paper examines how we can use low-level memory access patterns obtained from a hypervisor layer to discriminate ransomware from benign applications without solving the semantic gap problem.
\end{itemize}

\section{Live-forensic hypervisor for collecting memory access patterns}
\label{sec:surveillance}

We first define the memory access patterns and feature vectors for creating machine learning models. Before describing them, we need to understand how hypervisors see memory address space of RAM. When virtual machines are executed on a machine, the hypervisor needs to translate a guest OS's physical address to a host machine's physical address. The memory address translation thus needs to be performed twice, once inside the guest OS from a guest virtual address to a guest physical address, and once inside the hypervisor from a guest physical address to a host physical address. Many modern hypervisor programs employ Second Level Address Translation (SLAT), a hardware-assisted address translation technology for OS virtualization, such as Intel's Extended Page Table (EPT) \cite{intel-sdm} and AMD's Rapid Virtualization Indexing (RVI) to improve the performance of the latter translation.

Intel's EPT works as follows: when a guest OS accesses a memory page of 4KB or 2MB for the first time, a VM exit due to an EPT violation (i.e., a hypervisor's page fault) activates a hypervisor. The hypervisor then creates a new EPT entry to map the guest physical address to a host physical address, followed by the guest OS's address translation to map the guest virtual address to the host physical address. Thus, an EPT violation occurs when there is no EPT entry for a memory page. Another reason for EPT violations is unauthorized access to the pages. An EPT entry of each host physical address has privilege bits of read, write, and instruction fetch (i.e., execute) to prohibit access on the memory page. The privilege bits protect a critical memory region (e.g., kernel code and drivers) from attackers on a guest OS. When a guest OS accesses a memory page that is not permitted in the page's EPT entry, a VM exit occurs due to an EPT violation; the hypervisor then decides whether the guest OS should execute the memory operations or not.

Fig. \ref{fig:flow-surveillance} shows the flow chart of the live-forensic hypervisor that collects memory and storage access patterns. Although many Virtual Machine Introspection (VMI) systems drop specific permission on the particular EPT entries to deliberately cause EPT violations for intercepting memory accesses, the hypervisor presented in this paper deletes all EPT entries and executes TLB shootdown every 30 s for intercepting memory accesses. TLB shootdown is an operation that flushes a Translation Lookaside Buffer (TLB), a cache of address translation, on all CPU cores. The TLB shootdown on BitVisor was implemented by Fukai et al. \cite{Fukai}; we used their implementation as it is. The appropriate interval is discussed in Section \ref{sec:interval}.

When an application or a kernel program on a guest OS accesses the pages that are not both in TLBs and EPT entries, a VM exit due to an EPT violation activates the hypervisor. The hypervisor then reads Exit Qualification to obtain the access type (i.e., read, write, or instruction fetch). It also obtains the page type of the accessed page, including 4KB page, 2MB page, and Memory Mapped Input and Output (MMIO). MMIO is a method of performing input and output between CPU and peripheral devices using memory address space. When the access type is write, and the page type is not MMIO, the developed hypervisor obtains the memory contents at the physical address after executing five main loops of the hypervisor to ensure that the memory contents are updated from the original one. The obtained memory contents are mainly used to calculate entropy in feature engineering. The current implementation of the hypervisor collects only the first 4,096 bytes at the physical address even when a 2MB page was accessed and skips a byte filled with zeros. Please note that the design and implementation of the monitoring function of storage access patterns, which are shown in the grey symbols, were presented in the previous paper \cite{Hirano2019}.

\begin{figure}[tb]
    \centering
    \includegraphics[scale=0.38]{figs/flow-surveillance.pdf}
    \caption{Flow chart of the live-forensic hypervisor that collects memory and storage access patterns.}
    \label{fig:flow-surveillance}
\end{figure}

\section{Design and implementation}

Fig. \ref{fig:setup-procedure} shows the schematic diagram of the hypervisor that collects memory and storage access patterns. Both the memory and storage access pattern monitor were implemented with BitVisor \cite{bitvisor}, a thin hypervisor for enforcing security functions using a single guest OS model. The security functions of BitVisor are transparently executed to a guest OS, and hence any modification of guest OSs and applications are not needed.
Table \ref{tbl:spec-test-machine} shows the specification of the test machine that executes the developed hypervisor. While a guest OS (i.e., Windows 10) uses a GbE card, the hypervisor uses a 10GbE card to transfer memory and storage access patterns to another monitoring machine. The SSDs were formatted in New Technology File System (NTFS) with GUID Partition Tables (GPT).

The hypervisor is booted from a write-protected USB thumb drive to prevent overwriting our trusted computing base, the hypervisor program, from ransomware and wiper malware. The guest OS is then booted from an SSD, a serial Advanced Technology Attachment (ATA) device. The hypervisor monitors memory access patterns using EPT violations as described in Section \ref{sec:surveillance}. Since BitVisor employs a single guest OS model, the guest physical addresses and the host physical addresses are identical except for the memory region that BitVisor uses. The hypervisor also monitors storage access patterns by intercepting the Direct Memory Access (DMA) protocol \cite{Hirano2019}.
The memory and storage access patterns are sent to another machine using User Datagram Protocol (UDP) packets on a 10-gigabit Ethernet connection with jumbo frames of 9,000 bytes. We can therefore collect access patterns even when a ransomware or wiper malware sample encrypted or destroyed data in serial ATA devices (e.g., SSD). BitVisor does not use virtual disk files but uses a serial ATA device as it is. We employ a drive duplicator to overwrite the entire contents of a serial ATA device with the initial content before executing a malware sample.

\begin{figure*}[tb]
    \centering
    \includegraphics[scale=0.50]{figs/setup-procedure}
    \caption{Schematic diagram of the live-forensic hypervisor for collecting memory and storage access patterns.}
    \label{fig:setup-procedure}
\end{figure*}

\begin{table}[tb]
\begin{center}
\caption{Specification of the test machine that executes the live-forensic hypervisor.}
\label{tbl:spec-test-machine}
\begin{tabular}{|l|l|}
\hline
CPU & Intel Cerelon G3920 (2.9 GHz) \\
\hline
RAM & DDR4-2133 8 GiB \\
\hline
Motherboard & ASRock H110M-HDV \\
\hline
Network & Intel Pro1000 (GbE), Intel X550-T1 (10GbE) \\
\hline
SSD & Crucial SSD 250GB (CT250MX500SSD1) \\
    & SSD Samsung SSD 250GB (MZ-7TD250) \\
\hline
OS  &  Windows 10 Enterprise LTSC (10.0.17763.2183) \\
\hline
Hypervisor & BitVisor downloaded on 24th Dec. 2021 (8c129a1) \\
\hline
\end{tabular}
\end{center}
\end{table}

\section{Feature engineering}

We calculated the following five feature vectors on the four memory access types (i.e., read, write, execute, and read/write). The memory access type is obtained from Exit Qualification of a VM exit due to an EPT violation; when both read and write bits are 1, we process the access type as read/write. In particular, a Shannon entropy is calculated only when the access type is write or read/write. We, therefore, create 18-dimensional feature vectors of memory access patterns to make machine learning models.

\begin{itemize}
    \item{Shannon entropy of a written page, $H(t)$ }
    \item{The number of accessed 4KB pages, $C_{4KB}(t)$ }
    \item{The number of accessed 2MB pages, $C_{2MB}(t)$ }
    \item{The number of accessed MMIO pages, $C_{MMIO}(t)$ }
    \item{Variance of accessed physcal addresses, $V(t)$ }
\end{itemize}

where $t$ is the time elapsed after executing a sample. Shannon entropy is a metric to measure uncertainty in a series of bit patterns \cite{6773024}. Ransomware encrypts many files, and the encryption operations increase the entropy of a specific memory region of RAM; hence, Shannon entropy can be a good indicator of ransomware activities. $H(addr)$ is a Shannon entropy of a written memory page at a host physical address $addr$, and it is calculated as follows:

\begin{equation}
\label{eq:entropy}
H(addr) = - \sum_{i=1}^{n} p(x_i) \log_2{p(x_i)}
\end{equation}

where $ p(x_i) $ is a probability of a byte $x_i$, which is an $i$th byte beginning from a physical address $addr$, and $n$ is the byte size of the collected memory data. Since the hypervisor cannot transfer all the contents of a 2MB page with acceptable performance, we employed 4,096 for $n$ in creating the dataset. $H(addr)$ produces a value between 0 and 8, where 8 represents a perfectly even distribution of byte values, where 0 represents a sequence of the same byte values. Before training machine learning models, the Shannon entropy is normalized between 0 and 1. A Shannon entropy at $t$, $H(t)$, is calculated as follows:

\begin{equation}
    H(t) = \frac{1}{N} \sum_{i=1}^{N} H_i(addr)
\end{equation}

where $H_i(addr)$ is a Shannon entropy of an $i$th written page calculated in \eqref{eq:entropy} and $N$ is the number of write accesses in $T_{window}$.

Our feature extractor transforms a set of data points within a time window $T_{window}$ into a single 18-dimensional vector. Fig. \ref{fig:window} show how we calculate the 18-dimensional feature vectors with a window size of $T_{window}$. The feature vectors are calculated by shifting a time window from left to right in time-series memory access patterns. $T_{d}$ is the duration of memory access patterns used for creating a machine learning model. Our feature extractor shifts the time window of $T_{window}$ from $t$ = 0 until $t$ reaches at $(T_d - T_{window})$. When we create feature vectors with $T_{d}$ = 30 s and $T_{window}$ = 10 s, the total number of the 18-dimensional feature vectors will be 20; the model can detect ransomware in 30 s after execution of ransomware. $T_{d}$ is, therefore, referred to as the detection time of ransomware or duration of data points fed into a machine learning model.

\begin{figure}
\centerline{\includegraphics[width=7.0cm]{figs/window.pdf}}
\caption{Window-based feature calculation}
\label{fig:window}
\end{figure}

$C_{4KB}(t)$, $C_{2MB}(t)$, and $C_{MMIO}(t)$ are the number of accessed pages of 4KB, that of 2MB, and that of MMIO, respectively. A variance of host physical address at $t$ is calculated as follows:

\begin{equation}
    V(t) = \frac{1}{N-1} \sum_{i=1}^{N} ( addr_i - \overline{addr} )^2
\end{equation}

where $\overline{addr}$ is the mean of $addr_i$, $addr_i$ is a physical address of an $i$th access on RAM, $N$ is the number of the accesses in $T_{window}$.

\section{Dataset creation}

We created a dataset of the eight classes consisting of four malicious classes and four benign classes. Table \ref{tbl:ransomware-hash} shows the hash values of three ransomware samples and one wiper malware sample used in creating the dataset. WannaCry, Sodinokibi (REvil), and Darkside are ransomware samples; CaddyWiper, on the other hand, is a wiper malware sample that destroys data on a victim's computer by wiping all files; and it was discovered in Ukraine \cite{caddywiper}.

We created decoy files from the Govdocs1 dataset that are available in the open repository \cite{govdocs}; the ransomware and wiper malware samples encrypt and delete the decoy files. The Govdocs1 dataset consists of files obtained from U.S. government websites and includes PDF, JPEG, HTML, and Microsoft Office files. We copied the 9,872 files in the first ten directories of the Govdocs1 dataset to the Desktop of a test machine in creating the dataset.

\begin{table*}[tb]
\begin{center}
\caption{Ransomware and wiper malware samples: SHA-256 hash values and the first submission dates at VirusTotal.}
\label{tbl:ransomware-hash}
\begin{tabular}{|c|c|c|}
\hline
Name & SHA-256 hash & Date \\
\hline
\hline
WannaCry & \texttt{ed01ebfbc9eb5bbea545af4d01bf5f1071661840480439c6e5babe8e080e41aa} & 2017-05\\
Sodinokibi (REvil) & \texttt{0fa207940ea53e2b54a2b769d8ab033a6b2c5e08c78bf4d7dade79849960b54d} & 2019-04\\
Darkside & \texttt{b6855793aebdd821a7f368585335cb132a043d30cb1f8dccceb5d2127ed4b9a4} & 2021-04\\
CaddyWiper & \texttt{a294620543334a721a2ae8eaaf9680a0786f4b9a216d75b55cfd28f39e9430ea} & 2022-03\\
\hline
\end{tabular}
\end{center}
\end{table*}

The dataset includes the four benign classes: Idle, AESCrypt, Zip, and Office suite with a web browser. To create the Idle class dataset, we collected memory and storage access patterns after booting Windows 10 but did not launch any application program. AESCrypt is an open-source symmetric encryption program that uses the Advanced Encryption Standard (AES) algorithm \cite{aescrypt}. We collected memory and storage access patterns when the AESCrypt program (version 3.10) encrypts the 9,872 decoy files. We also collected memory and storage access patterns when the Zip program compresses the directory containing the 9,872 decoy files. The Zip program is a standard compression program included in Windows 10. The memory and storage access patterns of Microsoft Excel, PowePoint, and Firefox were also collected. The version of Excel and PowerPoint is 2019, and that of Firefox is 98.0.2. We created a macro program of Excel that inputs values in cells and creates a graph repeatedly. The slides in PowerPoint are played automatically. Firefox web browser plays a movie on YouTube. We executed the Excel, PowerPoint, and Firefox programs simultaneously in creating the dataset.

Memory and storage access patterns of the above eight classes are collected for 60 s immediately after executing each sample. We collected the access patterns of five trials for each class. In every trial, we copied the original contents to the SSD to recover the initial state before ransomware or wiper malware samples destroyed data on the SSD.

\section{Interval for flushing TLBs and clearing EPT entries}
\label{sec:interval}

Since the developed hypervisor collects memory access patterns using EPT violations, we must determine the appropriate time interval for flushing TLBs on all CPU cores (i.e., TLB shootdown) and deleting all EPT entries. If the hypervisor does not flush TLBs and delete EPT entries at a specific interval, we cannot observe memory accesses except for the first access on each page. Fig. \ref{fig:setup-violation-count} shows the cumulative number of EPT violations immediately after a hypervisor flushed the TLBs on all CPU cores and deleted all EPT entries. The experiment was conducted using the hardware and software shown in Table \ref{tbl:spec-test-machine}. The frequent operations to flush TLBs and to clear EPT entries, in general, cause significant performance degradation of the guest OS. We confirmed that the hypervisor could obtain a sufficient number of EPT violations to monitor memory access patterns in the first 30 s from the experimental result shown in Fig. \ref{fig:setup-violation-count}. In this paper, we determined the time interval of 30 s (i.e., $T_{flush}$ = 30). We, however, need to search for a more appropriate value of $T_{flush}$ through experiments that can achieve both minimizing performance degradation and maximizing the number of observed access patterns.

\begin{figure}[tb]
    \centering
    \includegraphics[scale=0.5]{figs/EPT-violation-count.pdf}
    \caption{Cumulative number of EPT violations.}
    \label{fig:setup-violation-count}
\end{figure}

\section{Classification performance on machine learning models}

Fig. \ref{fig:ata-ept-all-win_10s-60s} shows the $F_1$ scores of the three machine learning models: Random Forest, Support Vector Machine (SVM), and k-Nearest Neighbors (kNN) using scikit-learn 0.23.2. The models are trained using feature vectors calculated from memory and storage access patterns.
\begin{figure*}[tb]
    \centering
    \includegraphics[scale=0.75]{figs/ata-ept-all-win_10s-60s.pdf}
    \caption{$F_1$ score of the three machine learning models (10-fold cross-validation)}
    \label{fig:ata-ept-all-win_10s-60s}
\end{figure*}
The Random Forest model was trained using the \texttt{RandomForestClassifier} class with the following parameters: the number of trees is 10, and the maximum depth of the tree is 10. The SVM model was trained using the \texttt{OneVsRestClassifier} class with the radial basis function kernel. The kNN model was trained using the \texttt{KNeighborsClassifier} class with the number of neighbors $k$ = 5.

To evaluate the effectiveness of the models, we used the unweighted mean of $F_1$ scores under 10-fold cross-validation. $F_1$ score is a harmonic mean of precision and recall. An unweighted mean of $F_1$ score is calculated as follows:

\begin{equation}
    \label{eq:f1}
    \overline{F_{1}} = \frac{1}{N} \sum_{i=1}^{N} \left( 2 \cdot \frac{Precision_i \cdot Recall_i}{Precision_i + Recall_i} \right)
\end{equation}

where $N$ is the number of classes. Precision and recall were calculated as follows:

\begin{equation}
    \label{eq:precision}
    Precision_i = \frac{TP_i}{TP_i + FP_i}
\end{equation}

\begin{equation}
    \label{eq:recall}
    Recall_i = \frac{TP_i}{TP_i + FN_i}
\end{equation}

where $TP_i$ (True positive) is the number of correctly classified features belonging to an $i$th class. $FP_i$ (False positive) is the number of incorrectly classified features belonging to an $i$th class. $FN_i$ (False negative) is the number of incorrectly classified features as not belonging to an $i$th class.

All models are trained using memory and storage access patterns of the first 60 s immediately after executing a sample (i.e., ${T_d}$ = 60 s) with $T_{window}$ = 10 s. The Random Forest model was the best classifier both in 8 and 2 classes. The two-class problem discriminates the four malicious classes (i.e., three ransomware samples and one wiper malware sample) from the four benign samples. Although the $F_1$ scores of memory access patterns are lower than those of storage access patterns, the total $F_1$ scores that used both memory and storage access patterns were improved in all the three models.

We also need to determine the proper time window size, $T_{window}$. Fig. \ref{fig:time-window-ept-60s} shows how $F_1$ scores changed with $T_{window}$ on the Random Forest model that  was trained using memory access patterns with ${T_d}$ = 60 s. The $F_1$ scores saturated at $T_{window}$ above 10; we, therefore, employed $T_{window}$ of 10 s in training the machine learning models.

\begin{figure}[tb]
    \centering
    \includegraphics[scale=0.65]{figs/time-window-ept-60s}
    \caption{Changes of $F_1$ scores with $T_{window}$ (10-fold cross-validation)}
    \label{fig:time-window-ept-60s}
\end{figure}

\section{Performance benchmarking}

Fig. \ref{fig:PCMARK10} shows the results of the PCMARK10 (version 2.1.2525) benchmark that is designed to test typical home user workloads \cite{pcmark10}. For example, the ``Essentials'' workload includes application start-up and web browsing, the ``Productivity'' workload includes spreadsheet and writing, and the ``Digital content creation'' workload includes rendering and video editing.
We first created the baseline of our performance evaluation by measuring a score on the test machine shown in Table \ref{tbl:spec-test-machine} without a hypervisor. We then test the following three systems: (1) the developed hypervisor; the monitoring function is disabled, (2) the developed hypervisor; monitoring function both for memory and storage is enabled, and (3) Microsoft's hypervisor that enabled Virtualization Based Security (VBS) with Hypervisor-protected Code Integrity (HVCI) and credential guard function. Both the developed hypervisor and Microsoft's hypervisor \cite{2021windowsinternals} are Type-1 (i.e., bare-metal) hypervisors.

Both the developed hypervisor and Microsoft's hypervisor caused performance degradation of 18 \% and 24 \% in total score, respectively. BitVisor, a lightweight and thin hypervisor for research purposes that we employed in the development, enforces security functions using a single guest OS model. Microsoft's hypervisor, on the other hand, enforces more complex security functions using a multiple guest OSs model. For example, when a guest OS on a Windows hypervisor loads a kernel driver, the HVCI function of a privileged guest OS is called to verify the code signature of the kernel driver. The kernel driver is loaded on the guest OS only when verification of the code signature is succeeded. Thus, Microsoft's VBS needs at least two guest OSs simultaneously for enforcing security functions. BitVisor, on the other hand, enforces security functions directly from hypervisor software without using any privileged guest OS. This single guest OS model of BitVisor reduces the number of context switching between a guest OS and the hypervisor software and contributes to performance improvement.

\begin{figure}
    \centering
    \includegraphics[scale=0.5]{figs/PCMARK10.pdf}
    \caption{Results of PCMARK10 benchmark tests.}
    \label{fig:PCMARK10}
\end{figure}

\section{Visualization of feature vectors}

Fig. \ref{fig:ransom} shows feature vectors of ransomware and wiper malware. The entropy of CaddyWiper was lower than those of three ransomware samples because CaddyWiper overwrites files with zeros while ransomware samples encrypt files \cite{caddywiper}. However, both ransomware and wiper malware share common behavioral patterns that heavily access many user files in a short time. Fig. \ref{fig:benign} shows feature vectors of benign application samples. The entropy values of AESCrypt and Zip were higher than those of Idle and Office; The compression algorithm of the Zip program increases entropy because it converts redundant bit patterns into less redundant ones. In the Idle class, we observed some background activities of system programs such as SearchIndexer.exe. Since the hypervisor flushes TLBs and deletes all EPT entries every 30 s, we observed spikes around 30 s on the number of EPT violations of 4KB and 2MB.

\begin{figure*}[tb]
    \centering
    \includegraphics[scale=0.7]{figs/ransom.pdf}
    \caption{Feature vectors of ransomware and wiper malware samples ($T_{window}$ = 1 s and $T_d$ = 60 s).}
    \label{fig:ransom}
\end{figure*}

\begin{figure*}[tb]
    \centering
    \includegraphics[scale=0.7]{figs/benign.pdf}
    \caption{Feature vectors of benign application samples ($T_{window}$ = 1 s and $T_d$ = 60 s).}
    \label{fig:benign}
\end{figure*}

\section{Discussion}

Kumara and Jaidhar presented the VMI technique that gathers details about the running processes of malware by introspecting the semantic view of a guest OS \cite{AJAYKUMARA201799}. Cheng et al. presented a lightweight live memory forensic framework based on hardware virtualization \cite{CHENG201723}. They showed a forensic method to obtain accurate information on malware processes using the information inside and outside the operating system. They also presented a forensic technique to monitor memory modifications by controlling page-grained permissions using EPT violations. Many state-of-the-art papers, including Sharmeen et al. \cite{sharmeen2020avoiding} and Leon et al. \cite{leon2021hypervisor}, have employed process-level dynamic features. While these papers \cite{CHENG201723, AJAYKUMARA201799, sharmeen2020avoiding, leon2021hypervisor} show a malware analysis method using process-level information, this paper presented a ransomware detection method using only low-level memory access patterns. Our approach is also different from other hypervisors that mainly focus on dumping memory contents \cite{vis, hypersleuth}.

\section{Conclusion}

We developed a thin and lightweight live-forensic hypervisor that collects low-level memory access patterns with mitigating advanced evasion techniques of malware that exploit OS vulnerabilities. Furthermore, we presented a design of novel feature vectors of memory access patterns. Our best machine learning classifiers achieved $F_1$ scores of 0.93 in classifying the eight classes and 0.95 in detecting ransomware and wiper malware. Our empirical results show that the developed hypervisor can detect ransomware and wiper malware using only low-level memory access patterns. The current dataset, however, does not include memory access patterns on various conditions, such as the different specifications of chipset, CPU, and RAM. We also need to examine the memory access patterns in more realistic situations, for example, when multiple samples are executed simultaneously.

\section*{Acknowledgment}

This work was supported by JSPS KAKENHI Grant Number JP20K11825. The authors would like to thank Takamichi Omori, Keisuke Makihara, and Hiroki Mizuno for their support in developing the hypervisor-based monitoring system. The authors gratefully thank the original developers of BitVisor.

\bibliographystyle{IEEEtran}