MAPPER: Multi-Agent Path Planning with Evolutionary Reinforcement Learning in Mixed Dynamic Environments

Abstract: Multi-agent navigation in dynamic environments is of great industrial value when deploying a large scale fleet of robot to real-world applications. This paper proposes a decentralized partially observable multi-agent path planning with evolutionary reinforcement learning (MAPPER) method to learn an effective local planning policy in mixed dynamic environments. Reinforcement learning-based methods usually suffer performance degradation on long-horizon tasks with goal-conditioned sparse rewards, so we decompose the long-range navigation task into many easier sub-tasks under the guidance of a global planner, which increases agents' performance in large environments. Moreover, most existing multi-agent planning approaches assume either perfect information of the surrounding environment or homogeneity of nearby dynamic agents, which may not hold in practice. Our approach models dynamic obstacles' behavior with an image-based representation and trains a policy in mixed dynamic environments without homogeneity assumption. To ensure multi-agent training stability and performance, we propose an evolutionary training approach that can be easily scaled to large and complex environments. Experiments show that MAPPER is able to achieve higher success rates and more stable performance when exposed to a large number of non-cooperative dynamic obstacles compared with traditional reaction-based planner LRA* and the state-of-the-art learning-based method.

\section{INTRODUCTION}
\label{sec:intro}
Driven by the need for flexible and efficient manufacturing, an increasing number of affordable mobile robots are expected to be deployed in warehouse environments for transportation purposes. One key component to support the applications of large scale robots is the multi-agent path planning technology. Many research efforts have been devoted to this field in recent years from different perspectives.

Generally, multi-agent planning methods can be classified into two categories: centralized methods and decentralized methods. When all the moving agents' intentions (e.g. future trajectories, goals) are known in a static environment, a centralized planner could generate collision-free paths for all the agents \cite{sharon2015conflict}. However, the computational burden may be a significant concern as the number of agents grows, and the agent's performance may degrade when exposed to unknown dynamic objects  \cite{mellinger2012mixed}. Besides, in practice, centralized methods heavily rely on stable and fast communication networks and powerful servers, which would be costly to be deployed in large scale environments with a large number of robots. Therefore, in this paper, we focus on decentralized methods, where reliable communication can not be established between agents.

For decentralized methods, each agent independently makes decisions based on its own local observations and policies. A natural question is: what should the agent know and assume about other agents or dynamic obstacles around it? Some approaches assume all obstacles are static and re-plan at a high frequency to avoid collision \cite{koenig2005fast}, while other people assume homogeneous policy for agents and constant velocity for dynamic obstacles \cite{van2008reciprocal}. However, we argue that in practice, it is difficult to perfectly estimate neighbouring decision-making agents' intentions without communication. Therefore, instead of using traditional path planning procedures, some recent approaches use reinforcement learning to solve robot the navigation problem by implicitly learning to deal with such interaction ambiguity with surrounding moving obstacles \cite{chen2017socially, everett2018motion, mehta2016autonomous, sartoretti2019primal}.

Though learning-based approaches have shown great potential to model complex interactions in dynamic environments, most of them make assumptions about the homogeneity or motion models of surrounding moving obstacles \cite{long2018towards, chen2017decentralized}. In this paper, we focus on planning in mixed dynamic environments where moving obstacles can either be cooperative or non-cooperative. Also, inspired by state-of-the-art trajectory prediction methods \cite{cui2019multimodal}, we propose an image-based spatial-temporal dynamic obstacle representation, which doesn't need explicit motion estimation and can be generalized to the arbitrary number of agents.

Reinforcement learning agent is usually difficult to achieve satisfying performance in long-horizon tasks with sparse rewards, as in the long-range goal-conditioned navigation problem \cite{eysenbach2019search}.
Therefore, one insight in this paper is using mature planning methods to guide the reinforcement learning-based local planning policy. In this way, agents can learn from complicated local interaction with dynamic obstacles while persistently moving towards a long-range goal. In addition, to ensure the multi-agent training stability and performance, we propose an evolutionary reinforcement learning method that can be easily scaled to large and complex environments.

The major contributions of this paper are:
\begin{enumerate}
    \item We investigate the multi-agent path planning problem in mixed dynamic environments without the homogeneity assumption.

    To model the dynamic obstacles' behavior, we propose an image-based representation which improves our agents' robustness to handle different types of dynamic obstacles.
    \item We decompose a difficult long-range planning problem into multiple easier waypoint-conditioned planning tasks with the help of mature global planners. Experiments show that this approach can greatly improve the performance of reinforcement learning-based methods.
    \item We propose an evolutionary multi-agent reinforcement learning approach that gradually eliminate low-performance policies during training to increase training efficiency and performance.
\end{enumerate}

The structure of this paper is as follows. Section~\ref{sec:related} introduces related works about multi-agent path planning in dynamic environments. Section~\ref{sec:background} provides some preliminaries for our problem formulation. The detail of our multi-agent path planning with the evolutionary reinforcement learning (MAPPER) approach is presented in section~\ref{sec:approach}, followed by section~\ref{sec:experiment} which shows the experiment results of MAPPER in various grid world simulations with mixed dynamic obstacles. Finally, we give a brief conclusion in section~\ref{sec:conclusion}.
\section{Related Work}
\label{sec:related}

\subsection{Decentralized Path Planning in Dynamic Environment}
Decentralized planning methods can be broadly classified into two categories: reaction-based and trajectory-based.
For reaction-based approaches, we need to specify avoidance rules for dynamic obstacles and re-plan at each step based on the rules, such as D* lite \cite{koenig2005fast} and velocity obstacle (VO) based methods \cite{snape2011hybrid, bareiss2015generalized}.
Trajectory-based approaches usually estimate surrounding dynamic objects' intentions and then search collision-free paths in the space-temporal domain \cite{fan2018baidu}. These methods either require perfect information of surrounding dynamic obstacles (e.g. velocities and positions) or assume that all the moving agents adopt the same planning and control policy so that they are homogeneous \cite{van2011reciprocal, van2008reciprocal}. However, such assumptions may not hold in many real-world applications when involved with sensing uncertainty and heterogeneous moving obstacles, such as pedestrians. Beside, increasing local interaction complexity may lead to oscillation behaviors or \textit{freezing robot problem} \cite{trautman2010unfreezing}. Also, in practice, VO-based and trajectory-based approaches usually have several components to process sensor data, such as object-intention estimation and trajectory prediction. Each component may have many hyper-parameters that are sensitive to environment settings, which need extra human efforts to tune.
In order to reduce the amount of hand-tuning parameters and deal with sensing uncertainties, some researchers proposed learning-based methods to solve the planning problem.

\subsection{Reinforcement Learning-based Planning}
Reinforcement learning-based collision-avoidance algorithms for the single-agent case have been extensively studied in recent years. Deep neural networks are usually used to approximate agent's policy and value function. Some people propose to learn the navigation policy in a completely end-to-end fashion, which directly maps raw sensor data to the agent's action \cite{pfeiffer2017perception,tai2017virtual}. However, we believe that extracting object-level representation can improve the policy generalization ability, because different sensor data sources may encode the same object-level information. Chen et al. first estimate dynamic obstacle's physical states (e.g. velocity and positions) under certain motion model assumptions, and then feed them into neural network to obtain future actions~\cite{chen2017socially, chen2017decentralized} . However, the agents' number is restricted and can hardly be deployed in large scale environments. \cite{everett2018motion} addresses the problem of a varying number of dynamic obstacles with LSTM and removes the homogeneity assumption for surrounding agents. However, it still requires explicit estimation of surrounding agents' states and suffers performance degradation in tasks with a large number of agents. For multi-agent case, PRIMAL \cite{sartoretti2019primal} is the most similar work with ours, which also uses image-based representation and target goal as input sources. However, non-cooperative dynamic obstacles and temporal information are not considered in their work. Besides, their centralized training approach takes a long time even with the help of imitation learning. Our approach differs from theirs in that: 1) we encode both spatial and temporal information of surrounding obstacles in observation representations; 2) we consider planning in mixed dynamic environments; 3) we propose a decentralized evolutionary training method which can converge much faster and can be generalized to arbitrary number of training agents; 4) we use mature global planner to guide the local policy to solve long-range navigation problem. We will use the reinforcement learning method in PRIMAL as an experiment baseline in section~\ref{sec:experiment}.

\section{Background}
\label{sec:background}

\subsection{Problem Formulation}
We model the multi-agent planning problem under the Markov decision processes (MDPs) framework. An $N$-agent MDPs can be represented by the state space $\mathcal{S}$, which describes all the possible state configurations of the agents, the action space $\mathcal{A}_1, ..., \mathcal{A}_N$, which defines each agent's possible actions, and the observation space for each agent $\mathcal{O}_1, ..., \mathcal{O}_N$. In this paper, we consider the partially observable situation, which means each agent can not observe all the state. The agent $i$ receives its own observation $\boldsymbol{o}_i:\mathcal{S} \mapsto \mathcal{O}_i$ and produces an action from its stochastic policy $\boldsymbol{\pi}_{\theta_i}:\mathcal{O}_i \times \mathcal{A}_i \mapsto [0,1]$, where the policy is parameterized by $\theta_i$. All the agents' actions will produce new state that follows the state transition function $\mathcal{T} : \mathcal{S} \times \mathcal{A}_1 \times ... \times \mathcal{A}_N \mapsto \mathcal{S}$. For each time step, agent $i$ will receive rewards based on state and its action $r_i:\mathcal{S} \times \mathcal{A} \mapsto \mathbb{R}$. The initial states is determined by the distribution $\rho:\mathcal{S} \mapsto [0,1]$. The objective is to maximize the expected return $R_i = \sum_{t=0}^{T} \gamma^tr_i^t$ of each agent $i$, where $\gamma$ represents the discount factor and $T$ is the time length. The detail representations of observation space, action space and rewards will be introduced in section~\ref{sec:approach}.

\subsection{Advantage Actor Critic Algorithm}
We use advantage actor-critic (A2C) method \cite{mnih2016asynchronous} as the basis of our multi-agent evolutionary reinforcement learning framework to solve the planning problem. A2C uses stochastic policy, which is essential in our multi-agent scenario because the equilibrium policies in multi-agent MDPs are usually stochastic \cite{bucsoniu2010multi}. Additionally, policy gradient-based methods usually have better convergence property than value-based methods \cite{mnih2016asynchronous}. The objective of A2C is to find the policy $\boldsymbol{\pi}_{\theta}( a |\boldsymbol{o})$ that can maximize the expected return $\mathbb{E}_{\pi_\theta}R(\tau) = \mathbb{E}_{\pi_\theta}\sum_{t=0}^{T} \gamma^tr(\boldsymbol{o}_t, a_t)$ over the episode $\tau = (\boldsymbol{o}_0, a_0, \dots, \boldsymbol{o}_T, a_T)$, where $a$ is the action and $\boldsymbol{o}$ is the observation. Given this objective function, the policy gradient can be computed as:
\begin{equation}
    \nabla_\theta J(\theta) = \mathbb{E}_{\boldsymbol{\pi}_\theta}[\nabla_\theta \log \boldsymbol{\pi}_{\theta}( a |\boldsymbol{o}) R(\tau)]
\end{equation}
To reduce the gradient variance, we employ a value function $V^{\boldsymbol{\pi}_{\theta}}(\boldsymbol{o})$ as the baseline and replace the expected return $R(\tau)$ with an advantage function $A^{\boldsymbol{\pi}_{\theta}}(\boldsymbol{o}, a)$. Then we can rewrite the gradients as:
\begin{equation}
    \nabla_\theta J(\theta) = \mathbb{E}_{\boldsymbol{\pi}_\theta}[\nabla_\theta \log \boldsymbol{\pi}_{\theta}( a |\boldsymbol{o}) A^{\boldsymbol{\pi}_{\theta}}(\boldsymbol{o}, a)],
\end{equation}
where the advantage function $A^{\boldsymbol{\pi}_{\theta}}(\boldsymbol{o}, a)$ has an unbiased estimation:
\begin{equation}
    A^{\boldsymbol{\pi}_{\theta}}(\boldsymbol{o}, a) = \mathbb{E}_{\boldsymbol{\pi}_\theta}(r + \gamma V^{\boldsymbol{\pi}_{\theta}}(\boldsymbol{o}^\prime) | \boldsymbol{o}, a) - V^{\boldsymbol{\pi}_{\theta}}(\boldsymbol{o})
\end{equation}
The policy $\boldsymbol{\pi}_\theta$ is usually termed as the \textit{actor} to produce actions based on current observations, and the value function $v^{\boldsymbol{\pi}_{\theta}}$ is the \textit{critic}, which is used to estimate the advantage function $A^{\boldsymbol{\pi}_{\theta}}$ that indicates the quality of produced actions. In this paper, we approximate the policy and value function with neural networks, which will be introduced in section~\ref{archi}.

\section{Approach}
\label{sec:approach}

This section shows how the multi-agent path planning problem is modeled into an evolutionary reinforcement learning framework. We firstly introduce the observation representation, action space, and reward design of each agent. Then, we detail the model architecture and training procedures.

\subsection{Observation Representation}
\label{obs}
In many real-world mobile robot applications, people usually use the single beam LiDAR for localization and obstacle detection purposes, which is cheap and reliable \cite{grisetti2007improved, pierson2019dynamic}. A common map representation based on the LiDAR data is called the cost map, which discretizes a 2D map into a fixed resolution grids and assigns a cost to each grid \cite{reid2013cooperative}. The cost and obstacle information can be continuously updated by the local observation of sensor data. Therefore, to mimic such common map representations in practice, we consider a partially observable grid world environment, where each agent has its own visibility that limited by the sensing range and there is no communication between agents. We argue that such a fully decentralized partially observable setting is feasible and important if we need to deploy our approach to the real-world with large scale robots. We assume that each agent is able to detect and distinguish surrounding agents and dynamic objects within its sensing range and estimate their relative positions. Also, we assume that each agent can access the static environment map so that it can plan a trajectory in this map.

We split the observations into three channels to encode different types of information. As shown in Fig.~\ref{fig:framework}, the first channel stores current observed static obstacles, surrounding agents and dynamic objects' positions, which are represented by different values. This channel is the basic reflection of sensing data and is corresponding to the cost map representation, which could be used in many traditional search-based planning algorithms \cite{koenig2005fast}. The second channel is the trajectory of surrounding agents and dynamic obstacles, which encodes the time sequence information. Inspired by the state-of-the-art trajectory prediction method in the autonomous vehicle field, we encode the trajectory with different grayscales in time \cite{cui2019multimodal}. For example, the point on a trajectory in the earlier time has a smaller value than the later one. The third channel is the reference path planned by a global planner based on the static environment map. The reference path update frequency could be much lower than our reinforcement learning-based local planner. We will demonstrate the importance of those observation representations in the experiment section~\ref{sec:experiment}.

\begin{figure}[htb]
\centering
\includegraphics[width=8.3cm]{framework.png}
\caption{MAPPER model architecture overview.}
\label{fig:framework}
\end{figure}

\subsection{Action Space}
In this paper, we consider an 8-connected grid environment, which means the agent can move to 8 directions: south, north, west, east, southwest, northwest, southeast and northeast. The agent can also choose to wait at the current grid. Thus, the action space contains 9 discrete options in total. At each time step, the agent will move one step following the direction that is selected. However, if the target grid is already occupied, the agent will not be able to move and will stay at the current position.

\subsection{Reward Design}
The objective of robot navigation is to reach the goal position with minimum steps while avoiding collision with obstacles. Therefore, the first part of the reward consists of step penalty $r_s$, collision penalty $r_c$ and goal-reaching reward $r_g$.
To encourage exploration, we penalize slightly more for waiting than moving if the agent has not reached the goal. A similar training trick is also used in \cite{sartoretti2019primal}. To prevent agents from adopting oscillating policies, we set penalty $r_o$ to agents that return to the positions they come from last time. The detailed values of these reward components in our experiment can be found in Table~\ref{reward}.

Since our local planning policy is guided by a reference path planned by global planner, we introduce an additional off-route penalty $r_{f}$ if the agent deviates from the reference path. The intuition is that if there are no dynamic obstacles around the agent, it should be able to follow the reference path. To obtain the off-route penalty,
we need to calculate the Euclidean distance between the agent's position and the closest point's position on the reference path. Denote the position of the agent as $\boldsymbol{p}_a \in \mathbb{R}^2$. Denote the reference path as a set of coordinates $\mathcal{S} = \{\boldsymbol{p}_{start},...,\boldsymbol{p}_{goal}\}$,
the penalty is calculated by
$
    r_{f} = -\min_{\boldsymbol{p} \in \mathcal{S}} ||\boldsymbol{p}_a - \boldsymbol{p} ||_2
$.
Then the final reward is
$
    R = r_s+r_c+r_o+r_g+\lambda r_{f}
$,
where the $\lambda$ controls the weight of off route reward.

\begin{table}[h]
\caption{Reward Design}
\label{reward}
\begin{center}
\begin{tabular}{|c |c|}
\hline
\textbf{Reward} & \textbf{Value}\\
\hline
step penalty $r_s$ & -0.1 (move) or -0.5 (wait) \\
\hline
collision penalty $r_c$ & -5 \\
\hline
 oscillation penalty $r_o$
 &  -0.3\\
\hline
 off-route penalty $r_{f}$
 &  -$\min_{\boldsymbol{p} \in \mathcal{S}} ||\boldsymbol{p}_a - \boldsymbol{p} ||_2$\\
\hline
goal-reaching reward $r_g$
 &  30\\
\hline
\end{tabular}
\end{center}
\end{table}

\begin{algorithm}
\caption{Multi-Agent Evolutionary Training Approach}
\label{alg:EA}
\begin{algorithmic}[1]
\REQUIRE ~ Agents number $N$; discount factor $\gamma$; evolution interval $K$; evolution rate $\eta$;
\STATE Initialize agents' model weights $\Theta = \{\Theta_1,...,\Theta_N\}$
\REPEAT
\STATE Set accumulated reward $R^{(k)}_1,...,R^{(k)}_N = 0$
\STATE // \textit{update model parameters via A2C algorithm}
\FOR{$k=1,...,K$}
\FOR{each agent $i$}
\STATE Executing the current policy $\boldsymbol{\pi}_{\Theta_i}$ for $T$ timesteps, collecting action, observation and reward $\{a_i^{t}, \boldsymbol{o}_i^t, r_i^t \}$, where $t\in [0, T]$
\STATE Compute return $R_i = \sum_{t=0}^{T} \gamma^tr_i^t$
\STATE Estimate advantage $\hat{A}_i = R - V^{\boldsymbol{\pi}_{\Theta_i}}(\boldsymbol{o}_i)$
\STATE Compute gradients $
    \nabla_{\Theta_i} J = \mathbb{E}[\nabla_{\Theta_i} \log \boldsymbol{\pi}_{\Theta_i}\hat{A}_i]$
\STATE Update $\Theta_i$ based on gradients $\nabla_{\Theta_i} J$
\ENDFOR
\STATE $R^{(k)}_i = R^{(k)}_i + R_i$
\ENDFOR
\STATE Normalize accumulated reward to get $\Bar{R}^{(k)}_1,...,\Bar{R}^{(k)}_N$
\STATE Find maximum reward $\Bar{R}_{j}^{(k)}$ with agent index $j$
\STATE // \textit{Evolutionary selection}
\FOR{ each agent $i$}
\STATE Sample $m$ from uniform distribution between $[0,1]$
\STATE Compute evolution probability $p_i = 1-\frac{\exp (\eta\Bar{R}_i^{(k)})}{\exp (\eta\Bar{R}_j^{(k)})}$
\IF{$m < p_i$}
\STATE $\Theta_i \xleftarrow{} \Theta_j$
\ENDIF
\ENDFOR
\UNTIL converged
\end{algorithmic}
\end{algorithm}

\subsection{Model Architecture}
\label{archi}
We use deep neural networks to approximate the policy and the value function in our A2C method.
The model architecture is illustrated in Fig.~\ref{fig:framework}. We have two input sources to be processed independently before being concatenated as a combined feature. The first one is the three channels image represented observation, which has been introduced in section ~\ref{obs}. The image channels are passed through several blocks, which contain convolution layers, and max-pooling layers. After the final block, the extracted feature will be flattened to one feature embedding.

We notice that reinforcement learning may hardly solve long-term tasks to get the reaching goals rewards \cite{eysenbach2019search}. Therefore, instead of using final goals as one input source, we use the waypoint coordinates as sub-goals of our task, which is computed by the global planner. More specifically, the global planner, which is the A* planner in our case, will generate a reference path from the start point to the goal. Then the agent will choose waypoints on the reference path based on a certain distance interval threshold and attempt to reach them one by one. Once the agent approaches its current waypoint goal within a pre-defined range, it will begin to head to the next waypoint.

The currently selected waypoint can be viewed as a sub-goal. It will be passed through a fully connected layer, and then be fed together with the observation feature embedding to two shared fully connected layers. The output feature of the two shared layers will then be passed through two separate neural networks. The lower one is a two-layers policy network with softmax activation, which produces the probability of choosing each action. The upper one is the value function network, which outputs the expected value of the current state.

\subsection{Multi-Agent Evolutionary Reinforcement Learning}
Although reinforcement learning has achieved great success in many single-agent tasks \cite{mnih2013playing}, it is still hard to directly apply those methods to the multi-agent case.
One challenge is the scalability issue: as the number of agents grows, the environment becomes more complicated and the variance of policy gradients may grow exponentially \cite{lowe2017multi}.

Inspired by evolutionary algorithm that has been successfully applied to many optimization problems \cite{simon2013evolutionary}, we adopt a decentralized evolutionary approach based on A2C algorithm, which can be applied to arbitrary number of agents training procedure. Evolutionary algorithm usually contains three stages: crossover, mutation and selection. Let's denote the model parameters of agent $i$ as $\Theta_i$. We firstly initialize $N$ agents with random weights for their own model. Then the mutation process begins by training each agent's model separately using A2C algorithm. After $k$ episodes training, agent $i$ will accumulate the rewards over the last $k$ episodes, and we denote it as $R_i^{(k)}$. Denote $R_{max}^{(k)} = \max_{i \in \{1,...,N\}} R_i^{(k)}$ and $R_{min}^{(k)} = \min_{i \in \{1,...,N\}} R_i^{(k)}$. We normalize the accumulated reward for agent $i$ by:
$
    \Bar{R}_i^{(k)} = \frac{R_i^{(k)}}{R_{max}^{(k)} - R_{min}^{(k)}}
$.
Assume agent $j$ has the maximum normalized reward $\Bar{R}_{j}^{(k)} = \max_{i \in \{1,...,N\}} \Bar{R}_i^{(k)}$, then we start the crossover and selection stages. Each agent $i$ has the probability $p_i$ to reserve its original model weights  and $1-p_i$ probability to replace its weights with the weights of agent $j$. The probability is calculated by
$
    p_i = 1-\frac{\exp (\eta\Bar{R}_i^{(k)})}{\exp (\eta\Bar{R}_j^{(k)})}
$.
where $\eta$ controls the evolution rate. Larger $\eta$ means agents with lower rewards are more likely to be updated. The core idea of our evolutionary method is very simple: gradually eliminate bad policies while maintaining good ones. The full MAPPER training process is shown in Algorithm~\ref{alg:EA}.

\section{Experiment and Discussion}
\label{sec:experiment}
\subsection{Experiment Settings}
\begin{figure}[htb]
\centering
\includegraphics[width=8.5cm]{maps_new.jpg}
\caption{Grid world simulation environment demonstration.}
\label{fig:map}
\end{figure}

We evaluate our approach in grid world simulation environment, just as Fig.~\ref{fig:map} shows. Gray blocks are static obstacles and black blocks are agents' goals. Orange circles represent agents. Each agent has a 7-grid sensing range in our experiment setting, which means the size of the observation image is $15 \times 15 \times 3$. Blue triangles present dynamic obstacles, where each dynamic obstacle will navigate to a randomly selected goal using LRA* algorithm~\cite{coop}. To increase the dynamic obstacle movement pattern diversity, we randomly select 50\% dynamic obstacles that will ignore the presence of surrounding agents, which would be more challenging for our agents because of their non-cooperative nature.

Existing centralized multi-agent path planning methods, such as conflict based search \cite{sharon2015conflict}, break down in mixed dynamic environments because of the unpredictable nature of non-cooperative moving obstacles. Therefore, we resort to decoupled reaction-based planning approaches. One benchmark we adopt is a modified local repair A* (LRA*) algorithm that re-plans at every time step, where we replace A* with D* lite \cite{koenig2005fast} implementation because the latter is more computationally efficient in dynamic environments \cite{coop}.
Each LRA* agent takes into account local observation, updates the cost map accordingly, and searches for a route to the destination.
Then, a coordinator that has access to every agent's future plan resolves conflicts between agents and adjust these paths. LRA* behaves similarly to our MAPPER method in that they both react based on local observation, but note that we do not require access to all agents' future plan information.

Another baseline is PRIMAL \cite{sartoretti2019primal}, a reinforcement learning-based decentralized planner. We modify the original PRIMAL to adjust to our experiment setting because the original model and observation representation do not consider non-cooperative dynamic obstacles. More specifically, we use the same observation representation and network architecture as ours, but keep the original A3C training procedures and goal-conditioned approach as PRIMAL, so we name it PRIMAL* in the rest of the paper.
We also conduct experiments that remove part of features of our MAPPER method, which are removing the moving dynamic obstacles' trajectory (w/o traj) and removing the global planner guidance feature (w/o guid). We evaluate the performance of each method in terms of the \textbf{success rate} in different experiment settings.

\begin{table*}[t]
\centering
\caption{Comparison of success rate over different experiment settings}
\label{compare}
\begin{tabular}{|c|c|c|c|c|c|c|c|}
\hline
\multicolumn{3}{|c|}{Environment Setting} & \multicolumn{5}{c|}{Success Rate}                              \\ \hline
map size     & agent     & dynamic obstacle    & MAPPER      & MAPPER w/o traj & MAPPER w/o guid & PRIMAL*        & LRA*           \\ \hline
20x20        & 15        & 10             & \textbf{1.0}  & 0.971    & 0.877   & 0.964   & 0.996 \\ \hline
20x20        & 35        & 30             & \textbf{1.0}  & 0.961   & 0.836 & 0.980   & 0.999 \\ \hline
20x20        & 45        & 30             & \textbf{0.999} & 0.854   & 0.607  & 0.971   & 0.997 \\ \hline
60x65        & 70        & 100            & \textbf{1.0}  & 0.256  & 0.516  & 0.352 & \textbf{1.0}   \\ \hline
60x65        & 130       & 140            & \textbf{1.0}  & 0.473   & 0.221    & 0.404 & 0.992 \\ \hline
120x130      & 150       & 40             & \textbf{0.997} &          0.324    &  0.211      & 0.389     & 0.994 \\ \hline
\end{tabular}
\end{table*}

\subsection{Training Details}
Inspired by the idea of curriculum learning \cite{forestier2017intrinsically}, we divide the whole training procedure into two stages and start from easier tasks. We begin by initializing a small population of agents and dynamic obstacles, and sample goals within a certain distance to let agents learn a short-range navigation policy. Then we increase the agents and dynamic obstacles number, and sample goals in the whole map.

The training parameters are the same for MAPPER and its variants. We set off-route penalty weight $\lambda = 0.3$, the evolution rate and interval $\eta=2, K=50$, the discount factor $\gamma=0.99$, and the learning rate $lr= 0.0003$. For PRIMAL*, we observe that it is sensitive to the learning rate and will not converge if we set the same learning rate as MAPPER. Therefore, we set the learning rate for PRIMAL* as 0.00005 after several experimental explorations.
For the first stage, we initialize 4 agents and 10 dynamic obstacles in a $20\times 20$ map with 7 grid goal-sample range, as shown in Fig.~\ref{fig:map} left. For the second stage, we train models with 20 agents and 30 dynamic obstacles in a more complex $32\times 32$ map without goal-sample limitation, as shown in Fig.~\ref{fig:map} right.

\subsection{Results}

\begin{figure}[t]
\centering
\includegraphics[width=8cm]{training.jpg}
\caption{Success rate and average reward  comparison of variants of MAPPER and PRIMAL* algorithms}
\label{fig:train}
\end{figure}

\begin{figure}[t]
\centering
\includegraphics[width=8cm]{goal.png}
\caption{Comparison of MAPPER and its variants with different goal sample range.}
\label{fig:goal}
\end{figure}

The training figures for the first stage are shown in Fig~\ref{fig:train}. For the second stage training, we find that MAPPER without dynamic obstacle trajectory (MAPPER w/o traj) and MAPPER without global planner guidance (MAPPER w/o guid) can hardly converge if we sample goals from the whole map, so we limit the goal sample range to 15 grids. For PRIMAL*, the proper learning rate depends on agents number because of its centralized training nature, so we keep the agents size and learning rate as in the first stage. Since the training settings are different for second stage, the training figures are not presented in Fig~\ref{fig:train}.  But from the first stage training plot, we can see MAPPER has the most stable performance (smallest variance) and fastest convergence. The final average reward and success rate of MAPPER are also superior to the other methods in comparison.

To demonstrate the effectiveness of our observation representation, we evaluate trained models in a $65 \times 65$ size simulation environment with 10 agents, 10 dynamic obstacles, and different goal sample range. The success rate when we increase the goal range is shown in Fig.~\ref{fig:goal}.
We can see the performance of other variants of MAPPER is sub-optimal, while MAPPER agents will not be influenced by the goal range. Specifically, if we remove the global planner guidance feature, the agent's performance declines a lot when the distance to goal is increased, which means decomposing the long-range navigation task to several easier waypoint-conditioned tasks is necessary.
Though removing dynamic obstacle trajectory feature will not be influenced a lot when the goal range is changed, however, it shows worse capability to handle interactions with dynamic obstacles in a large environment.

We also evaluate the trained models as well as LRA* in various environment settings without goal sample range limitation to see their generalization capability. The performance is shown in Table~\ref{compare}.
Note that LRA* needs to access all the agents' (not dynamic obstacles) intention information and resolve conflicts before taking actions, while MAPPER only needs local observations.
We observe that in simple tasks where only a few moving obstacles are around the MAPPER agent, the agent will behave similar to following the reference path from the global planner. However, when the dynamic obstacle density is increased and the reference path is blocked, MAPPER agent performs aggressively to get out of surrounding obstacles and then moves towards its goal. We can see the success rate for MAPPER is the highest and is consistently above 0.99 in various experiment settings.

The MAPPER variant without dynamic obstacle trajectory works well when there are few dynamic obstacles but performs poorly when the complexity of the environment increases. It can be seen that the waypoints guidance is an important aspect of the MAPPER algorithm and the variant without waypoints guidance has low success rate even in a $20\times 20$ grid world with 15 agents and 10 dynamic obstacles.

\section{CONCLUSION}
\label{sec:conclusion}
This paper proposes a decentralized partially observable multi-agent path planning with evolutionary reinforcement learning (MAPPER) method to learn an effective local planning policy in mixed dynamic environments. We model dynamic obstacle's behavior with an image-based representation and decompose the long-range navigation task into many easier waypoint-conditioned sub-tasks. Furthermore, we propose a stable evolutionary training approach that could be easily scaled to large and complex environments while maintaining good convergence property compared with centralized training methods. The experiment result shows that MAPPER outperforms traditional method LRA* and learning-based method PRIMAL* in terms of success rate among various experiment settings. However, MAPPER may still collide with other agents or dynamic obstacles in complex environments in order to reach the goal. So the future direction would be to investigate safety-critical learning-based planning methods.

\section*{ACKNOWLEDGMENT}

The authors acknowledge the support from the Manufacturing Futures Initiative at Carnegie Mellon University made possible by the Richard King Mellon Foundation.
\newpage
\bibliographystyle{IEEEtran}

Robotics (cs.RO); Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Multiagent Systems (cs.MA)


Towards Class-Oriented Poisoning Attacks Against Neural Networks

Abstract: Poisoning attacks on machine learning systems compromise the model performance by deliberately injecting malicious samples in the training dataset to influence the training process. Prior works focus on either availability attacks (i.e., lowering the overall model accuracy) or integrity attacks (i.e., enabling specific instance-based backdoor). In this paper, we advance the adversarial objectives of the availability attacks to a per-class basis, which we refer to as class-oriented poisoning attacks. We demonstrate that the proposed attack is capable of forcing the corrupted model to predict in two specific ways: (i) classify unseen new images to a targeted "supplanter" class, and (ii) misclassify images from a "victim" class while maintaining the classification accuracy on other non-victim classes. To maximize the adversarial effect as well as reduce the computational complexity of poisoned data generation, we propose a gradient-based framework that crafts poisoning images with carefully manipulated feature information for each scenario. Using newly defined metrics at the class level, we demonstrate the effectiveness of the proposed class-oriented poisoning attacks on various models (e.g., LeNet-5, Vgg-9, and ResNet-50) over a wide range of datasets (e.g., MNIST, CIFAR-10, and ImageNet-ILSVRC2012) in an end-to-end training setting.

\section{Introduction}
In recent years, machine learning has demonstrated superior performance in various fields including computer vision~\cite{he2016deep,krizhevsky2012imagenet}, natural language processing~\cite{bahdanau2014neural,devlin2018bert,young2018recent}, autonomous vehicle~\cite{bojarski2016end,chen2015deepdriving}, robotics~\cite{kober2013reinforcement,sunderhauf2018limits}, and healthcare~\cite{esteva2017dermatologist}. However, it has also been shown that machine learning models are vulnerable to various types of attacks, such as evasion attacks~\cite{athalye2018obfuscated,carlini2017towards,dong2018boosting,goodfellow2014explaining,moosavi2016deepfool,szegedy2013intriguing} and poisoning attacks~\cite{biggio2012poisoning,mei2015using,munoz2017towards,shafahi2018poison,steinhardt2017certified,munoz2019poisoning,yang2017generative}. Evasion attacks occur at the inference phase, which cause misclassification without altering the model. The adversarial example is one such attack, which fools a well-trained neural network to misclassify specific inputs by adding imperceptible perturbations to the original images~\cite{goodfellow2014explaining}. In contrast, poisoning attacks corrupt the model by injecting malicious training data points. Compared to evasion attacks, poisoning attacks have the potential to render more severe or even catastrophic damages to real-world applications where continuously model updates are required or training data are accessible. For instance, as recommendation systems usually take customers' feedback to update the model constantly, the adversary can possibly utilize such vulnerability to manipulate recommendation content to achieve the adversarial goals.

Prior research on poisoning attacks can be broadly classified into two categories: availability attacks that degrade overall model accuracy (i.e., denial-of-service attacks)~\cite{biggio2012poisoning,xiao2012adversarial,mei2015using,munoz2017towards,mozaffari2014systematic,jagielski2018manipulating,steinhardt2017certified,yang2017generative,munoz2019poisoning} and integrity attacks that cause misclassification on specific instances (i.e., backdoor attacks)~\cite{chen2018detecting,shafahi2018poison,zhu2019transferable,turner2019label,yao2019latent}. However, the existing literature of poisoning availability attacks lacks an extensive study on multi-class classification. Even those considered multi-class classification, are rarely evaluated against the state-of-the-art learning models such as ResNet or large-scale dataset such as ImageNet. Furthermore, the adversarial goal of prior works on poisoning availability attack is constrained only in degrading the overall accuracy.

In this paper, we extend the literature by developing and applying novel poisoning availability attack strategies against deep neural networks to a per-class basis. We advance the adversarial objectives by formulating two attack tasks: (i) corrupting the overall model performance by forcing the model to classify all or most new inputs as a targeted class, which is denoted as the \emph{supplanter class}, and (ii) corrupting performance of a specific class, which is named as the \emph{victim class}, while retaining the accuracy of other classes. We propose an efficient gradient-based framework for the poisoning availability attacks, which generates more effective poisoned data samples by enlarging/dwindling the feature information of images. The idea is partially inspired by the explanation of probabilities that are assigned to incorrect answers~\cite{hinton2015distilling}. We leverage this inherent structural similarity among different classes to develop the novel class-oriented poisoning attacks.

\textbf{Our Contributions.} In this paper, we made the following technical contributions.
\begin{itemize}[leftmargin=.3in,nolistsep,noitemsep]
\item {This paper introduces the concept of the class-oriented poisoning attack, which manipulates the model behaviour on a per-class basis.}
\item {While the majority of prior work focuses on binary classification tasks and conventional learning algorithms such as regression and support vector machine, our poisoning availability attacks extend to multi-class classification tasks with deep neural networks.}
\item {We define two new performance metrics at the class level: \emph{change-to-target (CTT)} rate and \emph{change-from-target (CFT)} rate, for evaluating the effectiveness of poisoning availability attacks, along with the overall accuracy degradation.}
\item {This paper, to the best of our knowledge, takes the first step to systematically evaluate the effectiveness of poisoning availability attack on a large-scale dataset (ImageNet).}

\end{itemize}

\section{Problem Settings}
\subsection{Poisoning Availability Attack}
\textbf{Related work.} The existing literature has exploited the poisoning availability attack against various learning algorithms such as clustering~\cite{biggio2013data}, LASSO~\cite{xiao2015feature}, collaborative filtering~\cite{li2016data} and SVM~\cite{biggio2012poisoning}. The closest works to this paper are~\cite{munoz2017towards} and~\cite{yang2017generative}, which extended poisoning availability attack to multi-class classification against regression models and shallow neural networks, respectively. However, these are not class-oriented. The prior works have also developed a series of approaches for poisoned data generation, including influence function~\cite{koh2017understanding}, minimax duality~\cite{koh2018stronger}, Karush-Kuhn-Tucker (KKT) conditions~\cite{mei2015using}, autoencoder~\cite{yang2017generative} and generative adversarial network (GAN)~\cite{munoz2019poisoning}. Different from these prior works, we propose a novel approach for class-oriented poisoned data generation by retaining/eliminating feature information in images, while targeting at multi-class classification tasks with deep neural networks.

We consider a continuous or online learning scenario where data samples arrive in a sequential order. Let $\boldsymbol{x} \in \mathcal{X}(\mathcal{X} \in \mathbb{R}^d)$ be a $d$-dimensional input and $y \in \mathcal{Y}$ be the corresponding label for the input. The objective of the classification task is to build up the mapping $\mathcal{F}$: $\mathcal{X} \xrightarrow{} \mathcal{Y}$. We denote the pre-trained base classifier's parameters as $\theta$. The model parameters are updated to $\theta^*$ with the incoming new steam of data: $\theta \xrightarrow{(\boldsymbol{x},y)} \theta^*$.

Poisoning availability attacks are typically formulated as a bi-level optimization problem:
\vspace{-0.1cm}
\begin{align}
\mathop{\arg\max}\limits_{\theta^* \in \Theta}&\sum_{(\boldsymbol{x},y) \in \mathcal{D}_{tr}} L\left[\mathcal{F}_{\theta^*} \left( \boldsymbol{x} \right), y, \theta^*\right] \\
\textbf{s.t.}\quad  \theta^* \in \mathop{\arg\min}_{\theta^* \in \Theta}& \sum_{(\boldsymbol{x},y) \in \mathcal{D}_{tr} \cup \mathcal{D}_p} L\left[ \mathcal{F}_\theta \left( \boldsymbol{x} \right), y,  \theta^*\right],
\end{align}
\vspace{-0.3cm}

where $D_{tr}$ is the clean training dataset, $D_p$ is the poisoned dataset, $\Theta$ is the possible parameter space, and $L[\cdot]$ is the loss function. The outer maximization stands for adversarial objective, while the inner minimization represents the training process with poisoned dataset.

\subsection{Class-Oriented Adversarial Objectives}
Rather than only focus on maximizing the overall loss, we take the first step towards extending the adversarial objective to a per-class basis. We impose two new adversarial objectives in addition to the goal of degrading overall accuracy, which are formulated as two optimization problems accordingly.

\textbf{Problem 1: All-Supplanter (AS) attack.} The goal of the AS attack is to misclassify all or most inputs as a targeted object class, which we named as \emph{supplanter class}. For a broader real-world example, the adversary can apply the AS attack to compromise a recommendation system such that no matter what the customer is looking for, only a particular item/brand (the supplanter class) will be recommended. The AS attack problem can be formulated as:
\vspace{-0.1cm}
\begin{align}
\mathop{\arg\max}_{\theta^* \in \Theta}& \sum_{(\boldsymbol{x},y) \in \mathcal{D}_{tr}} L\left[ \mathcal{F}_{\theta^*} \left( \boldsymbol{x} \right),  y, \theta^*\right]
\end{align}
\vspace{-0.2cm}
\begin{align}
\textbf{s.t. }& \theta^* \in \mathop{\arg\min}_{\theta^* \in \Theta} \sum_{(\boldsymbol{x},y) \in \mathcal{D}_{tr} \cup \mathcal{D}_p} L\left[ \mathcal{F}_\theta \left( \boldsymbol{x} \right), y,  \theta^*\right]   \\
& \theta^* \in \mathop{\arg\min}_{\theta^* \in \Theta} \sum_{(\boldsymbol{x},y) \in \mathcal{D}_{tr}, {y = y_s}} L\left[\mathcal{F}_{\theta^*} \left( \boldsymbol{x} \right), y, \theta^*\right],
\end{align}
\vspace{-0.1cm}

where $y_s$ represents the label of the supplanter class.

\textbf{Problem 2: Only-Victim (OV) attack.} The goal of the OV attack is to compromise the classification accuracy for only the inputs from a specific class, which is denoted as \emph{victim class}. For a broader example again, the adversary can apply the OV attack to a high privilege class in an access control system, resulting in a denial-of-service to only the users in this class. The OV attack problem is formulated as:

\vspace{-0.2cm}
\begin{align}
\mathop{\arg\max}_{\theta^* \in \Theta}& \sum_{(\boldsymbol{x},y) \in \mathcal{D}_{tr},{y=y_v}} L\left[ \mathcal{F}_{\theta^*} \left( \boldsymbol{x} \right), y, \theta^*\right] \end{align}
\vspace{-0.2cm}
 \begin{align}
\textbf{s.t. }& \theta^* \in \mathop{\arg\min}_{\theta^* \in \Theta} \sum_{(\boldsymbol{x},y) \in \mathcal{D}_{tr} \cup \mathcal{D}_p} L\left[ \mathcal{F}_\theta \left( \boldsymbol{x} \right), y,  \theta^*\right]   \\
& \theta^* \in \mathop{\arg\min}_{\theta^* \in \Theta} \sum_{(\boldsymbol{x},y) \in \mathcal{D}_{tr}, {y \neq y_v}} L\left[\mathcal{F}_{\theta^*} \left( \boldsymbol{x} \right), y, \theta^*\right],
\end{align}
\vspace{-0.2cm}

where $y_v$ is the label of the victim class.

\subsection{Class-Oriented Evaluation Metric}

We propose two class-oriented evaluation metrics to assess the performance of the proposed class-oriented poisoning attacks.

\textbf{Change-to-Target (CTT)} rate is designed as an evaluation metric for the \textbf{AS attack}, which indicates the percentage of images that are classified as a targeted supplanter class ($\mathcal{C}_s$) due to the poisoning attack. CTT rate for a class $\mathcal{C}_k$ is formally defined over a test dataset $\mathcal{D}_t$ as:

\begin{equation}
    CTT(\mathcal{C}_k)  =\frac{\sum_{\substack{(\boldsymbol{x_i},y_i) \in \mathcal{D}_t \\{y_i=y_k}}}\big{(}\mathcal{F}_{\theta^*}(\boldsymbol{x_{i}})==y_s\big{)} -\sum_{\substack{(\boldsymbol{x_i},y_i) \in \mathcal{D}_t \\{y_i=y_k}}}\big{(}\mathcal{F}_{\theta}(\boldsymbol{x_{i}})==y_s \big{)} }{N_{k}},
    \label{equ_CTT_perclass}
\end{equation}

where $N_k$ is the total number of images in the class $\mathcal{C}_k$ and $y_k$ is the corresponding categorical label, while $\mathcal{F}_{\theta}$ and $\mathcal{F}_{\theta^*}$ are the model mappings before and after the poisoning attack, respectively. Then, the overall CTT rate over $\mathcal{D}_t$ can be calculated by weighted averaging the CTT rates of all non-supplanter classes:
\vspace{-0.1cm}
\begin{equation}
    CTT  =\frac{\sum_{\substack{k=1 \\ k\neq s}}^{K}  \big{(}N_k \cdot CTT(\mathcal{C}_k)\big{)}}{\sum_{\substack{k=1 \\ k\neq s}}^{K} N_k},
    \label{equ_CTT}
\end{equation}

where $K$ is the total number of classes.

\textbf{Change-from-Target (CFT)} rate is specifically used for evaluating the \textbf{OV attack}, which indicates how many images from a targeted class are misclassified due to the poisoning attack. Similarly, CFT rate for a class $\mathcal{C}_k$ is defined by Equation~(\ref{equ_CFT}).
\begin{equation}
   CFT(\mathcal{C}_k) =\frac{\sum_{\substack{(\boldsymbol{x_i},y_i) \in \mathcal{D}_t \\{y_i=y_k}}}\big{(}\mathcal{F}_{\theta}(\boldsymbol{x_{i}})==y_k\big{)}-\sum_{\substack{(\boldsymbol{x_i},y_i) \in \mathcal{D}_t \\{y_i=y_k}}}\big{(}\mathcal{F}_{\theta^*}(\boldsymbol{x_{i}})==y_k\big{)} }{N_k}
   \label{equ_CFT}
\end{equation}

\subsection{Adversarial Model}
Following the recent works on poisoning availability attack~\cite{koh2017understanding,mei2015using}, we consider the same adversarial model where the adversary has the knowledge of the learning algorithm and model architecture. Meanwhile, the attacker is assumed to be capable of injecting crafted poisoned data and assigning labels to poisoned data. Although the assumption is less practical in the real-world applications, it allows us to assess the effectiveness of poisoning attacks in the worst-case scenario, which is the same as in the literature of poisoning availability attacks~\cite{burkard2017analysis,biggio2012poisoning,munoz2019poisoning,li2016data,munoz2017towards}.

\section{Adversarial Goal and Evaluation Metrics}

We consider two tasks for the poisoning attacks on a per-class basis, which are distinct from prior indiscriminate poisoning attacks:
\begin{itemize}
\item \textbf{Task 1 (targeted supplanter class):} corrupting the overall model performance and manipulating the model to classify all inputs into a supplanter class.
\item \textbf{Task 2 (targeted victim class):} only corrupting performance of a selected victim class, while maintaining the accuracy of other classes.
\end{itemize}

To achieve the adversary goals, we extend the prior indiscriminate poisoning attack to the class level. To showcase the idea, Figure~\ref{Task1&2} shows a representative example of our adversarial goal. While prior methods yield misclassification in a random manner at the class level, our proposed attacks cause misclassification into the intended class. Additionally, we introduce new metrics (i.e., CTT and CFT) to impose additional constraints to guide the attack.

\textbf{Definition: CTT rate and supplanter class}
A supplanter class is a targeted class that after carrying out a successful poisoning attack, all or most images will be classified as this class.

CTT rate stands for change-to-target rate, which indicates the percentage of images that are classified into the supplanter class after the poisoning attack. It can be formally expressed as in Equation~\ref{equ_CTT},  where $X$ is the entire dataset,  $N$ is total number of images in $X$, $y_s$ is the label of supplanter class, and $P(\cdot)$ is the neural network prediction with a specific input.
\begin{equation}
    CTT  =\frac{\sum_{0}^{n}(P(x_i)==y_s) }{N}
    \label{equ_CTT}
\end{equation}

\textbf{Definition: CFT rate and victim class}
A victim class is the targeted class that suffers significantly performance degradation after the poisoning attack. For instance, if class `0' is selected as the victim class, images that originate from this class should no longer be classified as 0 after the attack. Meanwhile, classification of images from other classes should not be largely affected.

CFT rate stands for change-from-target rate. It indicates how many images from a targeted class are misclassified after the attack, which can be mathematically expressed as in Equation~\ref{equ_CFT}, where $C$ is the image set of an arbitrary class(any class can be set as a targeted class to evaluate the CFT rate), $N_c$ is the number of images in the set that are correctly classified before attack, and $y_c$ is the original label of the targeted class.
\begin{equation}
   CFT(C) =\frac{\sum_{i=0}^{n_c}(P(x_i)==y_c)_{before}-\sum_{i=0}^{n_c}(P(x_i)==y_c)_{after} }{N_c}
   \label{equ_CFT}
\end{equation}

\textbf{Definition: seed image and poisoning label}
Seed images are used to generate poisoned data for initiating a poisoning attack, which can be either selected from the original training and validation datasets, or collected by synthesizing test images. Seed images can be arbitrarily selected if the victim class is non-targeted. Poisoning labels are defined as the ground-truth labels of the supplanter class.

\section{Class-Oriented Poisoning Attack Methods}

\subsection{All-Supplanter Attack}

Intuitively, we expect retraining images with the label of the supplanter class, similar to the flipped-label poisoning attack~\cite{xiao2012adversarial}, would have the potential to shift predictions of all classes towards the supplanter class. However, our experimental results demonstrate that such attacks, when applied to multi-class classification tasks, neither effectively degrade the overall accuracy nor achieve the class-oriented adversarial goal of the AS attack for the neural network models (see \emph{Section 4}).

To improve the effectiveness of poisoning towards the supplanter class, we develop a novel method for generating the poisoned data. We leverage the fact that the probabilities assigned to other object classes of a well-trained model also reveal how much feature information of these incorrect classes are associated with the corresponding image by the model~\cite{hinton2015distilling}. Thus, we hypothesize that if an input image only contains feature information of its ground-truth class, training such image with the supplanter class label will force the model to expand the decision boundary of the supplanter class to the maximum degree.

Inspired by prior works that exploit the penultimate layer (formally called logit outputs) to distill knowledge of neural networks~\cite{hinton2015distilling}, catch features~\cite{ilyas2019adversarial}, and defend against adversarial examples~\cite{kannan2018adversarial}, we follow a similar methodology to control the feature information through the logit outputs. Our algorithm starts with a seed image that is arbitrarily picked from any class other than the supplanter class, and then attempts to retain the feature information associated with the original ground-truth class and reduce the feature information of other classes by enlarging/dwindling the corresponding logit outputs. We here denote $f(\cdot)$ as the logit output function of the neural network and $f_{y_k}$ as the corresponding logit to the categorical label $y_k$. The objective of our algorithm can be expressed by the following minimization objective function:
\begin{equation}

    \mathop{\arg\min}_{\boldsymbol{x_p}\in D_p} \Big{(}L[f_{y_o}(\boldsymbol{x_o}),+\infty,\theta] + \sum_{\substack{k =1\\k \neq o}}^{K}{\lambda_k\cdot L_k[f_{y_k}(\boldsymbol{x_o}),-\infty,\theta]} \Big{)},
    \label{objective_function}
\end{equation}

where $\boldsymbol{x_o}$ is the seed image, $y_o$ is the corresponding categorical ground-truth label, $f_{y_o}(x_o)$ is the logit output of the ground-truth class, $f_{y_k}(\boldsymbol{x_o})$ is the logit output of other classes, and finally $\boldsymbol{x_p}$ stands for the poisoned image. $\lambda_k$ is used to control the importance of loss terms. Alternatively, from the perspective of entropy, we expect such optimization would also reduce the following information entropy $H[\cdot]$:
\begin{equation}
    H\big{[}\sigma\big{(}f(\boldsymbol{x_p})\big{)}\big{]}=-\sum_{k=1}^{K}p_k\cdot log(p_k) \xrightarrow{} 0,
    \label{entropy}
\end{equation}
where $p_k$ is the probability for each class that converted from the logit  $f_{y_k}$ using \emph{softmax}.

However, solving the minimization problem can be computationally expensive, especially for high-dimensional datasets that have hundreds or thousands of classes. Based on the facts that (i) the classification result is determined by the largest probability and (ii) only the supplanter class is the target, we consider an approximation that simplifies the task to retain feature information of the ground-truth class and eliminate feature information of the supplanter class. In other words, we only focus on the two most important classes instead of all classes, i.e.,
\begin{equation}

    \mathop{\arg\min}_{\boldsymbol{x_p}\in D_p}\big{(}L[f_{y_o}(\boldsymbol{x_o}),+\infty,\theta] + \lambda\cdot L[f_{y_s}(\boldsymbol{x_o}),-\infty,\theta]\big{)}.
    \label{objective_function_simplified}
\end{equation}

\begin{comment}

Similarly, the evaluation function can be written as the following:
\vspace{-0.1cm}
\begin{equation}
    H(\sigma(f(\boldsymbol{x_p}))\approx -[p_o \cdot log(p_o) + p_s \cdot log(p_s)] \xrightarrow{} 0
    \label{entropy2}
\end{equation}

 where $p_s$ and $p_o$ represent the probability of the ground-truth class and the supplanter class, respectively.

\end{comment}

By solving the minimization using gradient descent, the poisoned image $\boldsymbol{x_p}$ is updated through one backward pass:

\vspace{-0.1cm}
\begin{equation}
    \boldsymbol{x_p} = \boldsymbol{x_o} - \epsilon \cdot sign\Big{(}\nabla_{\boldsymbol{x_o}}\big{(}L[f_{y_o}(\boldsymbol{x_o}),+\infty,\theta] + \lambda\cdot L[f_{y_s}(\boldsymbol{x_o}),-\infty,\theta]\big{)}\Big{)},
    \label{poisoned_data_update}
\end{equation}

where $\epsilon$ is a hyper-parameter chosen by the attacker. This update step can also be executed for several rounds to more aggressively increase the feature information of the ground-truth class and reduce the feature information of the supplanter class. Algorithm~\ref{algo:poisoned data generation} presents the details of poisoned data generation for the AS attack.

{\centering
\begin{algorithm}
\DontPrintSemicolon
    \KwIn{$\boldsymbol{x_o}$: seed image, $y_o$: seed image label, $y_s$: supplanter class label\\
    $T$: max number of optimization iterations \\
    hyper-parameters $\lambda$, $\epsilon$}
    \KwOut{poisoned image $\boldsymbol{x_p}$, poisoned label $y_p$}

    {\bf Initialization:}  $\boldsymbol{x_{p_0}} =  \boldsymbol{x_o} - \epsilon \cdot sign\Big{(}\nabla_{\boldsymbol{x_o}}\big{(}L[f_{y_o}(\boldsymbol{x_o}),+\infty,\theta] + \lambda\cdot L[f_{y_s}(\boldsymbol{x_o}),-\infty,\theta]\big{)}\Big{)} $

    \While{t<T}{
    Compute the gradient: $\nabla = \nabla{\boldsymbol{x_{p_t}}}\big{(}L[f_{y_o}(\boldsymbol{x_{p_t}}),+\infty,\theta] +\lambda\cdot L[f_{y_s}(\boldsymbol{x_{p_t}}),-\infty,\theta]\big{)}$

    Update the poisoned image: \textit{Clip}\{$\boldsymbol{x_{p_{t+1}}} = \boldsymbol{x_{p_t}} - \epsilon \cdot sign(\nabla)$\}

    \uIf{$f_{y_s}(\boldsymbol{x_{p_{t+1}}})$ $>$ $f_{y_s}(\boldsymbol{x_{p_{t}}})$ \textbf{or} $f_{y_o}(\boldsymbol{x_{p_{t+1}}})$ $<$ $f_{y_o}(\boldsymbol{x_{p_{t}}})$ }{

    \textbf{break}
    }

    }

   \textbf{Assign} $y_p = y_s$

    \textbf{Return} $\boldsymbol{x_p}, y_p$

\caption{Poisoned Data Generation (All-Supplanter)}
\label{algo:poisoned data generation}
\end{algorithm}
}

\subsection{Only-Victim Attack}

Compared to the AS attack, the OV attack is fundamentally more challenging, as it requires to not only compromise the accuracy of the targeted victim class but also maintain the performance of the other classes, i.e.,  high CFT rate for the victim class while low CFT rates for the other classes. However, as observed in our preliminary experiments as well as prior poisoning availability attacks~\cite{liu2016teaching,park2017resilient}, poisoning with data from only one class will also largely shift the distribution of other classes. Hence, a set of poisoned data from more than one class is necessary to achieve the adversarial goal of the OV attack. Intuitively, simply training images from all classes except the victim class may achieve this adversarial goal. However, our preliminary results reveal that such methods always require a huge amount of training data and very large training epochs, which is very inefficient in achieving the adversarial objective.

We propose another gradient-based algorithm for the OV attack. The entire procedure of our poisoned data generation is presented in Algorithm~\ref{algo:poisoned data generation_t2}. We modify the method from the AS attack to craft the poisoned dataset as follows: (i) pick a same number of arbitrary images from each class, (ii) enlarge/dwindle feature information of the corresponding classes for each image, as detailed in Algorithm~\ref{algo:poisoned data generation_t2}, and (iii) assign the original ground-truth labels to the non-victim classes and the targeted label $y_p$ to the victim class. Specifically, for images from the victim class, we apply the same operations as in Algorithm~\ref{algo:poisoned data generation}, while for images from other classes, we only increase the feature information of their original classes. The objective of OV attack can be expressed as
\vspace{-0.1cm}
\begin{equation}

    \begin{cases}
    \mathop{\arg\min}\limits_{\boldsymbol{x_p}\in D_p}\big{(}L[f_{y_v}(\boldsymbol{x_o}),+\infty,\theta]+ \lambda\cdot L[f_{y_p}(\boldsymbol{x_o}),-\infty,\theta]\big{)} , & \text{if  $\boldsymbol{x_o} \in \mathcal{C}_v$}\ \\
    \mathop{\arg\min}\limits_{\boldsymbol{x_p}\in D_p}\big{(}L[f_{y_o}(\boldsymbol{x_o}),+\infty,\theta]\big{)}, & \text{otherwise}
    \end{cases}
    \label{objective_function_3}
\end{equation}

Similarly, the poisoned images are updated through a backward pass in Equation~(\ref{poisoned_data_generation_2}):
\vspace{-0.1cm}
\begin{equation}
    \ \boldsymbol{x_p} =
    \begin{cases}
    \boldsymbol{x_o} - \epsilon \cdot sign\Big{(}\nabla_{\boldsymbol{x_o}}\big{(}L[f_{y_v}(\boldsymbol{x_o}),+\infty,\theta] +\lambda\cdot L[f_{y_p}(\boldsymbol{x_o}),-\infty,\theta]\big{)}\Big{)}, & \text{if  $\boldsymbol{x_o} \in \mathcal{C}_v$}\ \\
    \boldsymbol{\boldsymbol{x_o}} - \epsilon \cdot sign\Big{(}\nabla_{\boldsymbol{x_o}}\big{(}L[f_{y_o}(\boldsymbol{x_o}),+\infty,\theta]\big{)}\Big{)}, & \text{otherwise}
    \end{cases}
    \label{poisoned_data_generation_2}
\end{equation}

\begin{algorithm}
\DontPrintSemicolon
\KwIn{$\boldsymbol{x_k} \in \boldsymbol{X}$: the set of seed images, {$y_k \in Y$}: the set of labels associated to $X$ \\
    $y_p$: poisoned label, $T$: max number of optimization iteration \\
    hyper-parameters $\lambda$, $\epsilon$}
\KwOut{poisoned dataset $\boldsymbol{X_p}$, poisoned label set $Y_p$}
  \uIf{$\boldsymbol{x_k}$ $\in$ $\mathcal{C}_v$}{
    \textbf{Apply} Algorithm 1\;

    \textbf{Assign} $y_k = y_p$
  }
  \Else{
  \textbf{Initialization:} $\boldsymbol{x_{p_0}} = \boldsymbol{\boldsymbol{x_o}} - \epsilon \cdot sign\Big{(}\nabla_{\boldsymbol{x_o}}\big{(}L[f_{y_o}(\boldsymbol{x_o}),+\infty,\theta]\big{)}\Big{)}$

    \While { $ t < T$} {
Compute the gradient: $\nabla = \nabla{\boldsymbol{x_{p_t}}}\big{(}L[f_{y_o}(\boldsymbol{x_{p_t}}),+\infty,\theta]\big{)}$

Update the poisoned image: \textit{Clip}\{$\boldsymbol{x_{p_{t+1}}} = \boldsymbol{x_{p_t}} - \epsilon \cdot sign(\nabla)$\}

}
  }

\textbf{Assign} $\boldsymbol{X_p} = \boldsymbol{X}$ ; $Y_p = Y$

\textbf{Return} $\boldsymbol{X_p}$, $Y_p$

\caption{Poisoned Data Generation (Only-Victim)}
\label{algo:poisoned data generation_t2}
\end{algorithm}

\section{Experiments}
\subsection{Experimental Settings}
We apply our proposed poisoning attacks to multi-class image classification tasks using three widely-used datasets (MNIST, CIFAR-10, and ImageNet-ILSVRC2012) against classic neural network models (LeNet-5, Vgg-9, and ResNet-50, respectively). All the evaluations are performed on the corresponding validation datasets. For comparison with prior works, we implement the flipped-label attack (FL)~\cite{xiao2012adversarial} and the direct gradient method (DGM)~\cite{yang2017generative} as the baseline attacks for MNIST and CIFAR-10. We also examine our poisoning availability attacks on ImageNet against a pre-trained ResNet-50 model~\cite{he2016deep}, which we hope to serve as a baseline comparison for future works of poisoning availability attacks. To illustrate the effectiveness of our methods as well as facilitate a fair comparison, we minimize the impact of significant model shifts due to large learning rates and better align with practical continuous or online learning scenarios, by setting the initial learning rate of the poisoning attack close to the final learning rate of the base model and applying the same decay strategy during the attack. All networks are implemented with TensorFlow and experiments are run on NVIDIA Tesla V100 GPUs.

\vspace{-0.1cm}
\subsection{Experimental Results of the All-Supplanter Attack}
To examine the true power of our AS attack, we only generate a single instance for the poisoning process. We present the results of MNIST and CIFAR-10 in Figure~\ref{Error-CTT}, and ImageNet in Table~\ref{resnet-CTT}. Hyper-parameters in Algorithm 1 are set as: $\lambda =1, \epsilon = 0.3$ for MNIST and CIFAR-10, and $\lambda =1, \epsilon = 0.5$ for ImageNet. It can be seen that our proposed AS attack is highly effective in degrading the overall model accuracy as well as increasing the CTT rate for the supplanter class. For example, we improve the test error from 20\% to above 80\% and the CTT rate from 10\% to 99\% on the CIFAR-10 dataset within less than 20 attack iterations. On the other hand, our method requires much fewer iterations to reach at the maximum poisoning effect than the baseline attack, whose test error and CTT rate are only increased by $\sim$20\% and $\sim$10\%, respectively, even after 50 attack iterations. For ImageNet, the proposed AS attack achieves a CTT rate of 94.07\% within 10 attack iterations. Note that including more poisoned data in each attack iteration will obviously enhance the poisoning effect of the proposed AS attack.

For scenarios where the adversary has no knowledge of the training process, it is also important to study the impact of different learning rates on the attack. Our conclusion is consistent with prior works~\cite{wang1808data} that smaller learning rates yield less effectiveness for the attack. However, while it requires more attack iterations to arrive at the maximum poisoning effect, our method still significantly outperforms the baseline attacks when learning rate is small, as indicated in Figure~\ref{Error-CTT}(c) and Figure~\ref{Error-CTT}(d).

 \begin{figure*}[htbp]
    \centering
    \resizebox{1.0\textwidth}{!}{
    \includegraphics{error-ctt_final.pdf}}
    \caption{Error and CTT rate comparison. Classes `4' and `deer' are selected as the supplanter class for MNIST and CIFAR-10, respectively. (a) and (c) are error rates, while (b) and (d) are CTT rates with different learning rates.}
    \label{Error-CTT}
\end{figure*}

\begin{table*}[hbtp]
\centering
\caption{Comparison of CTT and accuracy before and after the AS attack on ImageNet.}
\vspace{0.1cm}
  \begin{tabular}{ c | c  c  c }
      \toprule[1pt]

         & Top-1 Accuracy & Top-5 Accuracy & CTT\\
      \midrule[1pt]

    Vanilla ResNet-50  & 74.87\% & 92.02\% & --\\

    Poisoned by AS Attack  & {2.15\%} & 16.91\% & {94.07\%}\\

      \bottomrule[1pt]
  \end{tabular}

\label{resnet-CTT}

\end{table*}

\begin{comment}

Another intriguing observation is that some pairs of seed image and poisoning label naturally perform better than others. For example, as shown in Table~\ref{tab:accuracy_ct_rate_comparison}, the pair class `1' (seed image) and `3' (supplanter image) has already achieved an accuracy degradation to 25.86\%, while increasing the CTT rate to 83.58\% through baseline attack. However, we found that no matter how the baseline attack performs, our method can always yield a stronger result by running through a large number of pairs. The details are presented as follows:

\begin{table*}[htbp]

\begin{scriptsize}
    \begin{tabular}{ccccccccccc}
     \toprule
     Accuracy & 0 & 2 & 3 & 4 & 5 & 6 & 7 & 8 & 9 & Average \\
     \midrule
     Baseline & 73.16\% & 59.43\% & 25.86\% & 76.43\% & 37.41\% & 72.41\% & 74.02\% & 50.46\% & 31.75\% & 55.66\% \\
     Our Approach & 17.43\% & 17.31\% & 12.41\% & 20.61\% & 12.41\% & 28.89\% & 18.42\% & 10.13\% & 10.21\%	& 16.42\% \\
     \toprule
     CTT Rate & 0 & 2 & 3 & 4 & 5 & 6 & 7 & 8 & 9 & Average \\
     \midrule
     Baseline & 28.95\% & 48.49\% & 83.58\% & 29.42\% & 70.20\% & 27.33\% & 33.21\% & 58.83\% & 76.34\%	 & 50.71\% \\
     Our Approach & 92.65\%	 & 93.27\% & 97.60\% & 87.92\% & 96.74\% & 79.90\% & 91.47\% & 99.64\% & 99.99\%	 & 93.24\% \\
     \bottomrule
    \end{tabular}
    \vspace{0.2cm}
    \end{scriptsize}
    \caption{ MNIST: accuracy and CTT rate comparison of a seed image (from class `1') paired with different poisoned labels.}
    \label{tab:accuracy_ct_rate_comparison}
\end{table*}

\end{comment}

\subsection{Experimental Results of the Only-Victim Attack}
Since the adversarial goal of the OV attack is to subvert only one class without degrading the performance of other classes, there are two important metrics for this task: 1) CFT rate of the victim class should be as high as possible, and 2) CFT rates of the non-victim classes should be as low as possible. Note that the highest achievable CFT rate is upper-bounded by the accuracy of the victim class in the vanilla model. We use the same hyper-parameters as in the AS attack. The poisoned images in the OV attack only account for 10\% of the training data. We present the CFT rate of each class by poisoning attacks on CIFAR-10 in Table~\ref{CFT_CIFAR10_classes}, where class `frog' is selected as the victim class. It can be seen that DGM
is able to keep the CFT rates of all non-victim classes relatively low; however, its CFT rate of the victim class is only 39.2\%. While the FL attack achieves a CFT rate of 85.1\% for the victim class, it also significantly increases the CFT rates of non-victim classes, which reveals the difficulty of achieving the class-oriented adversarial goal by using prior poisoning attack methods. Compared to the baseline attacks, our proposed approach can effectively increase the CFT rate of the victim class and also retain the CFT rates of the non-victim classes to be less than 6.5\%. Besides, in some cases, the proposed approach even improves the accuracy of some non-victim classes, which is indicated by the negative CFT rates in our experimental results. This performance is beyond our expectation, which however does not contradict the class-oriented OV adversarial objective, i.e., only degrading the accuracy of the victim class.

For ImageNet, due to the large number of object classes, we present the distribution of the final CFT rates, as shown in Figure~\ref{fig:CFT_resnet}. While successful in maintaining low CFT rates for non-victim classes, our method achieves a CFT rate of 62\% for the victim class on ImageNet. Note that we can improve the performance by increasing the percentage of poisoning images in the training data. Therefore, it can be concluded that the proposed OV attack, although still accomplishes the adversarial goal, is less effective when compared to the performance on CIFAR-10. A possible rationale behind this phenomena is that it becomes harder to completely decouple the feature information of one class from all other classes during the poisoning, with the number of classes scaling up. On the other hand, our experimental results also reveal the importance of studying poisoning availability attack on large-scale/dimensional dataset, which is lacking in the existing literature.

\begin{table*}[hbtp]
\centering

\caption{CFT rate of each CIFAR-10 class by poisoning.}
\vspace{0.1cm}
    \begin{tabular}{c|c c c c c}
     \toprule
     Class / Attack& airplane & automobile & bird & cat & deer \\
    \midrule
     DGM & 0.1\% & -0.1\% & 12.0\% & 18.2\% & 3.7\% \\
     FL & 19.5\% & 15.0\% & 51.9\% & 53.3\% & -21.3\% \\
     \textbf{OV (proposed)}  & -0.2\% & 2.9\% & 6.5\% & -3.9\% & -1.4\% \\
     \midrule
      Class / Attack& dog & \cellcolor[HTML]{C0C0C0}frog (victim) & horse & ship & truck\\
     \midrule
     DGM & 6.8\% & \cellcolor[HTML]{C0C0C0}{39.2\%} & -0.1\% & 2.1\% & -4.3\% \\
     FL Attack & 41.1\% & \cellcolor[HTML]{C0C0C0}85.1\% & 26.3\% & 13.2\% & 19.3\% \\
     \textbf{OV (proposed)}   & 5.0\% & \cellcolor[HTML]{C0C0C0}71.6\% & 1.4\% & -1.7\% & 3.1\% \\
     \bottomrule

    \end{tabular}
     \label{CFT_CIFAR10_classes}
\end{table*}

\begin{figure}[htbp]
     \centering

     \resizebox{1\textwidth}{!}{
     \includegraphics{hist_imagenet2.pdf}}
     \caption{The CFT rate distribution of ImageNet classes by the OV attack.}
     \label{fig:CFT_resnet}
 \end{figure}

\section{Possible Defenses}
Since the main objective of this paper is to extend the attack to a per-class basis rather than enhance the adversarial effect against various defensive solutions, in general, we expect the proposed attacks to have similar performance as prior poisoning availability attacks, when evaluated under possible defenses.

Data sanitization is a defensive technique against poisoning attack that works by distinguishing and removing outliers (poisoned data) from the training dataset~\cite{cretu2008casting,rubinstein2009antidote}. However, it has been shown that a broad range of data sanitization can be easily compromised or bypassed~\cite{koh2018stronger}. Therefore, we can also leverage such techniques for our proposed attack to evade the detection. In fact, in most of the recent works on poisoning attacks, data sanitization is no longer considered as a certified defensive strategy~\cite{koh2018stronger,munoz2019poisoning,jagielski2018manipulating,wang1808data}.

Alternatively, a possible countermeasure is to periodically check the accuracy and/or loss of the learning models~\cite{munoz2017towards,yang2017generative}. Although expensive in terms of cost and time, these approaches are intuitively effective based on the fact that poisoning availability attack aims at degrading the accuracy. Therefore, the threat models of poisoning availability attacks usually consider the retraining or learning process to be unmonitored, which makes these countermeasures impractical. We expect the worst-case performance of our proposed methods against such countermeasures to be consistent with prior poisoning availability attacks~\cite{mei2015using,li2016data,munoz2017towards,mozaffari2014systematic,yang2017generative,munoz2019poisoning}.

Since the poisoned data are tailored to maliciously influence the training process of a learning model, we suggest exploiting averaged stochastic gradient classifier~\cite{wang1808data} and combinational models, such as bagging~\cite{li2016data}, where the classification results are no longer dependent on a single model, to defend against the attack. However, the overhead for deploying multiple classifiers should also be carefully considered.

\section{Conclusions}
This paper introduced the concept of class-oriented poisoning attack. We formulated two attack problems, i.e., ``All-Supplanter'' and ``Only-Victim'', which seek to compromise the model behavior on a per-class basis. Accordingly, we defined two new metrics to evaluate the performance of poisoning attacks at the class level. Our proposed gradient-based algorithms successfully achieved the class-oriented adversarial objectives through manipulating the feature information contained in images for poisoned data generation. The effectiveness of the proposed methods are comprehensively studied in our experiments.

Machine Learning (cs.LG); Cryptography and Security (cs.CR); Machine Learning (stat.ML)


Colloidal glass transition observed in confinement

Abstract: We study a colloidal suspension confined between two quasi-parallel walls as a model system for glass transitions in confined geometries. The suspension is a mixture of two particle sizes to prevent wall-induced crystallization. We use confocal microscopy to directly observe the motion of colloidal particles. This motion is slower in confinement, thus producing glassy behavior in a sample which is a liquid in an unconfined geometry. For higher volume fraction samples (closer to the glass transition), the onset of confinement effects occurs at larger length scales.

\section{Introduction}

Glasses are typically formed by rapidly quenching the
temperature of a liquid, resulting in an amorphous
liquid-like microstructure with macroscopic solid-like
behavior.  Upon approaching the glass transition, the
temperature might be changed by only a factor of two while
simultaneously the viscosity of the liquid grows by many
orders of magnitude \cite{reviews}.  A conceptual microscopic
explanation for the viscosity growth is the idea of dynamic
length scales: in order for molecules in the material to
rearrange, they must move together as a group.  As the glass
transition is approached, the increasing
size of these groups relates to the increasing macroscopic viscosity
\cite{reviews,mckenna05,ediger00,weeks00,kegel00,weeks07}.

An important way to probe these length scales is to
study the behavior of glass-forming systems when they
are confined, to constrict the range of accessible length scales
\cite{mckenna05,kob00,lowen99,robbins92,yamamoto00,jackson91,barut98,richert94,morineau02}.
Intriguingly, the glass transition temperature $T_g$
can be both smaller or larger in confined geometries
\cite{jackson91,barut98,richert94}, even for the same
material \cite{mckenna05,kob00,morineau02}.
Experiments and simulations suggest that
the interaction between the confining surface and the sample
is crucial.  For strong interactions (or atomically rough
surfaces) the glass transition happens ``sooner,'' that is,
confinement increases $T_g$ by slowing motion near the surfaces
\cite{mckenna05,kob00,lowen99,richert94,morineau02}.
Likewise, for systems that weakly interact with the walls,
$T_g$ is typically smaller
\cite{mckenna05,kob00,jackson91}.  However, a clear explanation
of these phenomena is still lacking.  As it is difficult to
get details out of experiments \cite{mckenna05}, the use
of computer simulations to visualize the motion is important
\cite{kob00,lowen99,yamamoto00,robbins92}.

We use confocal microscopy to directly visualize the motion
of colloidal particles, which serve as a model system for
the glass transition in confinement.  Colloids undergo a
glass transition in bulk samples as the solid particle volume
fraction $\phi$ is increased
\cite{weeks00,kegel00,pusey86,vanmegen96}.
At high volume fraction near the colloidal glass
transition ($\phi_g \approx 0.58$), particles
move in rearranging groups characterized by a length scale
of $\sim$3-6 particle diameters \cite{weeks00,weeks07}, similar to
simulations \cite{yamamoto00}.  In this manuscript we study a
mixture of two sizes of colloidal particles confined between two
quasi-parallel plates, with a plate gap as small
as 3.0 large-particle diameters.  In our system confinement
induces the glass transition ``sooner,'' at concentrations for
which the bulk behavior is still liquid-like.  Studying the
glass transition in confinement may help us understand the
glass transition in the bulk \cite{mckenna05}.  Furthermore,
understanding the properties of confined liquids has relevance
for lubrication \cite{robbins92,granick99}, dusty plasmas
\cite{teng03}, and the flow of glassy complex fluids through
microfluidic devices \cite{granick99}.

Our colloidal samples are poly-methyl(methacrylate)
particles, sterically stabilized to prevent aggregation
\cite{weeks00,pusey86}.  We use a mixture of two particle
sizes, with radii $a_{\rm small} = 1.18$ $\mu$m and $a_{\rm
large} = 1.55$ $\mu$m.  While the particle polydispersity is
low ($\sim$5\%), the mean particle radii are only known to
within $\pm 0.02$ $\mu$m.  The mixture of two particle sizes
prevents crystallization which would otherwise be induced by
the walls \cite{lowen99,teng03,murray98}.  In each sample,
the small particles are dyed with rhodamine dye, and the large
particles are undyed.  We use a mixture of cyclohexylbromide
and decalin as our solvent, to match the density and index
of refraction of the particles; the particles are slightly
charged in this solvent \cite{dinsmore01}.  The viscosity of
the solvent is 2.25 mPa$\cdot$s.  We examine four different
samples A-D, with properties listed in Table I.

\begin{table}
\caption{\label{table1}
Characteristics of the four samples studied.
The number ratio $N_{\rm small}/N_{\rm large}$ is determined
by counting particles in several fields of view using DIC
(differential interference contrast) microscopy.
The total volume fraction $\phi_{tot}$ is determined using
confocal microscopy, by counting
the number of small particles seen in a given imaging volume,
using the known number ratio to determine the number of large
particles present, and then using the particle sizes and the
imaging volume size to compute $\phi_{tot}$.  Additionally
$\phi_{tot}$ and $N_{\rm small}/N_{\rm large}$ was confirmed in
samples B--D by direct 3D confocal microscopy observation, where
the particle sizes could be easily distinguished and counted;
the results were in agreement with the DIC measurements.
The volume fractions of the small species and large species,
$\phi_s$ and $\phi_l$, are calculated from the other two
quantities.
The uncertainties of $N_{\rm small}/N_{\rm large}$ are $\pm
5\%$, and the uncertainties of $\phi_{tot}$ are $\pm 8\%$.
In particular, note that samples A and C likely do not have
the same volume fraction, but it is unclear which has the larger
$\phi_{tot}$.  Samples B, C, and D are prepared by dilutions
of one stock sample and thus all have the same $N_{\rm
small}/N_{\rm large}$.
}
\begin{ruledtabular}
\begin{tabular}{ccccc}
Sample & $N_{\rm small}/N_{\rm large}$ & $\phi_s$ & $\phi_l$ &
$\phi_{tot}$\\
\hline
A & 3.5 & 0.26 & 0.16 & 0.42 \\
B & 3.0 & 0.13 & 0.10 & 0.23 \\
C & 3.0 & 0.24 & 0.18 & 0.42 \\
D & 3.0 & 0.26 & 0.20 & 0.46 \\
\end{tabular}
\end{ruledtabular}
\end{table}

We observe our samples using confocal microscopy
\cite{dinsmore01,prasad07}.  As the larger particles are
not dyed, we only see the smaller particles.  We use a fast
confocal microscope (VT-Eye from Visitech, International)
with a 63$\times$ air objective (N.A.~0.70) to scan a volume
$50 \times 50 \times 20$ $\mu$m$^3$ once every 2.0~s over a
period of an hour.  We analyze the images offline to locate the
positions of visible (smaller) particles, with a resolution
of 0.05~$\mu$m in $x$ and $y$ (parallel to the walls) and a
resolution of 0.1~$\mu$m in $z$ (parallel to the optical axis).
We then track their motion in 3D \cite{dinsmore01}.

Our sample chambers are made by placing a microscope
coverslip at a slight angle, supported by a small piece of
mylar film (thickness 100 $\mu$m) at one end and resting
directly on the microscope slide at the other end
\cite{murray98,palberg05}.  The ends and sides are
sealed with UV-curing epoxy.  Thus a thin wedge-shaped chamber
is formed with an angle $\approx 0.4^\circ$, ensuring that
locally the walls are essentially parallel, and allowing us
to study a single sample at a variety of different confinement
thicknesses \cite{murray98,palberg05}.

The glass surfaces of the coverslip and slide are untreated.
In experiments with sample A, we find that some colloidal
particles stick to these surfaces.
The stuck particle coverage
is typically 10\% - 20\% of the area.
In a second series
of experiments done with samples B--D, no particles were stuck.
Reassuringly, we find little dependence of
the behavior on the number of stuck particles in the results
discussed below \cite{stuck}.

For sample A, measuring the positions of the stuck particles
allows us to accurately measure the sample thickness.
While the uncertainty in locating individual particle positions
in $z$ is 0.1~$\mu$m, by averaging data from tens of
stuck particles over hundreds of images we locate their mean
$z$ position to better than 0.005~$\mu$m.  Thus the effective
thickness $H$ of each experimental data set is determined to
within 0.01~$\mu$m, and is the range in $z$ available to the
{\em centers} of the visible particles.  In this manuscript
our thicknesses are reported in terms of $H$.  The true
surface-to-surface thickness of a sample chamber is found by
adding $2a_{\rm small} = 2.36$~$\mu$m to $H$.

For the first series of experiments, we study the behavior of
sample A ($\phi \approx 0.42$) as a function of thickness.
We quantify the particle motion
by calculating the mean square displacement (MSD),
$\langle \Delta x^2 \rangle = \langle (x_i(t+\Delta t) -
x_i(t))^2 \rangle$,
where the average is taken over all particles $i$ and all
initial times $t$, and a similar formula applies for $\langle
\Delta y^2 \rangle$ and $\langle \Delta z^2 \rangle$.  We find
that $\langle \Delta x^2 \rangle \approx \langle \Delta y^2
\rangle$ for all our experiments; we report our results
for the $x$ direction, the direction over which the
sample chamber has constant thickness.  We first consider the
results for motion parallel to the confining plates, $\langle
\Delta x^2 \rangle$, shown in Fig.~\ref{msd}(a).  The upper
bold line shows motion in an unconfined
region and is reproducible for all chamber thicknesses $H >
20$~$\mu$m.  For this sample, the motion in the unconfined
region is nearly diffusive, with the MSD growing almost
with slope 1 on the log-log plot.
This behavior is similar to monodisperse samples with a volume
fraction of $\phi \leq 0.4$ \cite{weeks00}.  In other words,
this sample is far from the glass transition, $\phi_g \sim 0.6$
\cite{sciortino04,vanmegen96}.
In thinner regions, the motion slows, as seen in the
sequence of solid curves below the top bold curve in
Fig.~\ref{msd}(a).  This slowing starts at a thickness of
$H \sim 16$~$\mu$m (2nd curve from top) and slows dramatically
for thinner samples; note that Fig.~\ref{msd}(a) shows a
log-log plot and thus for the thinnest region shown (bottom
curve, $H=6.92$~$\mu$m), to move a distance $\langle x^2 \rangle =
(a_{\rm small}/3)^2$ it takes a time scale 180 times larger
than for the bulk region data ($\Delta t = 500$~s as compared to 2.8 s).

\begin{figure}
\centerline{
\epsfxsize=8.0truecm
\epsffile{fig1.ps}}
\caption{Mean square displacements.  (a) Data for sample A,
showing motion parallel to the walls, for thicknesses $H$ =
bulk, 16.28 $\mu$m, 11.06 $\mu$m, 9.41 $\mu$m, and 6.92 $\mu$m
(from top to bottom).  The dashed line has a slope of 1 and
indicates the expected motion for a very dilute bulk suspension
of particles with radius $a_{\rm small}$.
(b) Similar to (a), but for motion perpendicular to the walls.
Data are ordered by thickness as $\Delta t \rightarrow \infty$,
as in (a).
}
\label{msd}
\end{figure}

These results suggest that confinement induces glassy behavior,
with the influence of confinement beginning at $H \approx
16$~$\mu$m~$\approx 14 a_{\rm small} \approx 10 a_{\rm large}$
for this sample.  For the lower curves in Fig.~\ref{msd}(a),
the characteristic behavior of a ``super-cooled'' sample is
seen:  at shorter lag times ($\Delta t < 100$~s),
the MSD has a plateau, while at longer lag times, the MSD begins to rise
again \cite{kob00,weeks00}.  (For short time scales, particles
are trapped in cages formed by their neighbors, causing the
plateau in $\langle x^2 \rangle$.  At longer time
scales, these cages rearrange \cite{weeks00,weeks07}.)  For the thinnest region
(bottom curve), the particles remain localized for the duration
of the experiment.

The slowing is also seen in motion perpendicular to
the walls, quantified by $\langle \Delta z^2 \rangle$,
shown in Fig.~\ref{msd}(b).  Moreover, in comparison with
Fig.~\ref{msd}(a), it is seen that the motion perpendicular to
the walls, $\langle \Delta z^2 \rangle$, is slowed even more so
than motion parallel to the walls, $\langle \Delta x^2 \rangle$.
This is not surprising, given that particles close to the walls
cannot move toward the walls at all, whereas motion parallel
to the walls is less restricted.

\begin{figure}
\centerline{
\epsfxsize=8.0truecm
\epsffile{fig2.ps}}
\caption{
(a) Particle number density $n_{\rm small}(z)$ as a function
of distance $z$ across the sample cell.  Additional particles
are stuck to the walls of the sample cell (not shown in
the plot) which have centers located at $z=0.00$ $\mu$m
and $z=H=11.06$~$\mu$m.  This data corresponds to the middle
curve in Fig.~\ref{msd}(a), that is, a sample with moderately
slowed dynamics.
(b) Mean square displacement parallel to the walls ($x$) and
perpendicular to the walls ($z$), as a function of $z$.
The displacements are calculated using $\Delta t = 100$~s.  The vertical dotted
lines indicate the positions of the peaks
from part (a).  Data shown are from sample A ($\phi \approx 0.42$).
}
\label{dz}
\end{figure}

More than merely constricting motion, the walls also induce a
layering of particles, as seen in Fig.~\ref{dz}(a), similar to
simulations \cite{lowen99,archer07}.  The layering is most pronounced
immediately adjacent to the walls.  The centers of these
peaks are not at the precise distance $a_{\rm small}$ from
the walls, but are slightly offset toward the interior of the
sample.  (The centers of the stuck particles indicate
the maximum possible extent in $z$ that particles could be
located, and correspond to the ``feet'' of the data shown in
Fig.~\ref{dz}(a) at $z=0$ and 11.06~$\mu$m.  These particles
are not counted in $n_{\rm small}$ shown in Fig.~\ref{dz}(a).)

The layers influence the dynamics, as seen in Fig.~\ref{dz}(b),
which shows how $\langle \Delta x^2 \rangle$ and $\langle
\Delta z^2 \rangle$ depend on $z$.  The displacements are
calculated using $\Delta t = 100$~s, as a representative time
scale over which particles begin to move out of this cage,
although the results do not depend on this choice and are similar
for caged behavior ($\Delta t = 2.0$~s for example).
Particles in the layers [the peaks of $n(z)$] have smaller
vertical displacements, as seen by the dips in $\langle \Delta
z^2 \rangle$ (heavy line).  The implication is that particles
in layers are in a preferred structure and less likely to
move elsewhere \cite{lowen99,archer07}.

Surprisingly, the layers do not appear to influence the
motion parallel to the walls, as seen by $\langle \Delta
x^2 \rangle$ (thin line), which does not depend on $z$.
(The slight dip in $\langle \Delta x^2 \rangle$ seen at the
largest values of $z$ is not reproducible in other data sets.)
This seems counterintuitive as hydrodynamic interactions
with the wall normally result in reduced motion for nearby
particles \cite{dufresne00}.  We speculate that the cage
dynamics dominate particle motion, rather than hydrodynamic
influences \cite{squires}.  For example, if a particle is pulled by an
external force in a direction parallel to the walls, other
particles would be forced to rearrange, which is probably
the most significant contribution to the drag.
Particle rearrangements would be even more constrained for a
particle pulled perpendicular to the wall, thus explaining why
we observe slower $z$ motion \cite{squires}.
Simply put, the high volume fraction likely results in
hydrodynamic screening.

Thus while confinement causes the layering of particles near
the walls, this layering does {\em not} appear directly
responsible for the slowing of the particle motion.
Rather, the layering seems to be an additional influence on
the motion in the direction perpendicular to the walls, as
seen in Fig.~\ref{dz}(b), but only a minor influence compared
to the overall fact of confinement.  Note that
results do not appear to depend on having an integral
number of well-defined layers between the walls \cite{robbins92}.
The overall dynamics slow smoothly and monotonically as
the confining dimension decreases.

Our observation that the layers closest to the wall have
slower motion perpendicular to the walls agrees qualitatively
with previous experiments \cite{barut98,richert94,morineau02}
and simulations \cite{kob00} which suggested that surface
layers may be glassier than the interior.  However, we note
in our experiment this is strongly directionally dependent.
The slowing is most easily seen if $\langle \Delta z^2 \rangle$
can be measured independently of the other two directions.

As noted earlier, the growth of dynamic length scales has been
observed as the glass transition is approached in a bulk material
\cite{reviews,mckenna05,ediger00,weeks00,kegel00,weeks07}.
For our colloidal samples, this implies that samples with a
larger $\phi$ should exhibit stronger confinement effects.
To check this, we took data from samples B, C, and D at
various thickness.  Qualitatively the data resemble that
shown in Fig.~\ref{msd}(a).  To capture the $H$ dependence,
Fig.~\ref{xh} shows the values of $\langle \Delta x^2 \rangle$,
at fixed $\Delta t = 100$~s, as a function of $H$ for the
different samples.  Consider the solid triangles, corresponding
to sample D.  For $H > 50$~$\mu$m, $\langle \Delta x^2 \rangle$
is essentially constant.  At $H <
50$~$\mu$m, the data start showing a strong $H$ dependence,
suggesting a confinement length scale of $H^* \approx
50$~$\mu$m.  For the solid symbols, an increase in $H^*$
is seen as $\phi$ increases, from approximately 10~$\mu$m
to 50~$\mu$m, confirming that there is a growing length
scale as the glass transition is approached.  These length
scales are significantly larger than those seen for dynamical
heterogeneities in monodisperse samples, which are 4 -- 8
$\mu$m \cite{weeks07}.  However, this agrees with simulations
which found a confinement length scale significantly larger
than the mobile cluster size \cite{kob00}.  In Fig.~\ref{xh},
sample A has a smaller value of $H^*$ relative to sample C,
which may be due to the excess of small particles in sample A;
see Table I.

\begin{figure}
\centerline{
\epsfxsize=8.0truecm
\epsffile{fig3.ps}}
\caption{
Value of $\langle \Delta x^2 \rangle$ at $\Delta t = 100$~s, as a
function of thickness $H$, for samples with $\phi$ as indicated.
The open circles correspond to sample A with
$N_{\rm small}/N_{\rm large}=3.5$,
while the solid symbols correspond to samples B--D with
$N_{\rm small}/N_{\rm large}=3.0$.
The lines are drawn to guide the eye.  The
plateau for each data set indicates behavior corresponding to the
bulk, whereas the downturn at low $H$ gives an idea of the length
scale at which confinement becomes important.
}
\label{xh}
\end{figure}

We find that confinement slows the motion of colloidal particles
and thus induces a glass transition to occur sooner than normal,
in other words, at volume fractions for which the bulk behavior
is liquid-like.  Simulations suggest the roughness of the walls
is crucial to this slowing \cite{kob00,lowen99} and we plan to
vary this in future experiments.  However, we note that our data
show slowing both with completely smooth walls (samples B,
C, and D) and walls with isolated stuck particles (sample A).
In contrast to our work, rough walls in simulations are composed
of particles fixed in a liquid-like structure
\cite{kob00,lowen99}.  This prevents layering of adjacent particles
and restricts motion parallel to the walls.  Thus the glass
transition in confined samples occurs sooner (at higher
temperatures \cite{kob00} or lower densities \cite{lowen99}).
In our experiments, particle motion parallel to the wall is
not noticeably inhibited, as seen in Fig.~\ref{dz}.  Yet,
we still find the glassy behavior occurs sooner:  at constant
volume fraction, the dynamics are slower as the confining
dimension decreases.  Thus it seems that the important effect
in our experiments is simply the restriction of motion perpendicular
to the wall, close to the surface of the wall.

We thank A.~Schofield and W.~C.~K.~Poon for our colloidal
samples, and J.-L.~Barrat, E.~R.~Dufresne, M.~F.~Hsu,
W.~Kob, J.~Saldana, and T.~M.~Squires for helpful discussions.
Initial support for this project was provided by the Donors
of The Petroleum Research Fund, administered by the American
Chemical Society (37712-G7).  The work of CRN, KVE, and ERW
was also supported by the National Science
Foundation under Grant No.~DMR-0239109.

Soft Condensed Matter (cond-mat.soft); Materials Science (cond-mat.mtrl-sci)