MD Simulation of Hundred-Billion-Metal-Atom Cascade Collision on Sunway Taihulight

Abstract: Radiation damage to the steel material of reactor pressure vessels is a major threat to the nuclear reactor safety. It is caused by the metal atom cascade collision, initialized when the atoms are struck by a high-energy neutron. The paper presents MISA-MD, a new implementation of molecular dynamics, to simulate such cascade collision with EAM potential. MISA-MD realizes (1) a hash-based data structure to efficiently store an atom and find its neighbors, and (2) several acceleration and optimization strategies based on SW26010 processor of Sunway Taihulight supercomputer, including an efficient potential table storage and interpolation method, a coloring method to avoid write conflicts, and double-buffer and data reuse strategies. The experimental results demonstrated that MISA-MD has good accuracy and scalability, and obtains a parallel efficiency of over 79% in an 655-billion-atom system. Compared with a state-of-the-art MD program LAMMPS, MISA-MD requires less memory usage and achieves better computational performance.

\section{Introduction}
As a sustainable, clean and renewable energy source, nuclear energy is a promising solution to global energy crisis and environmental pollution.
Nevertheless, how to ensure the safety of the nuclear reactor is a crucial issue.
Many components of a reactor are exposed to a radiation environment that is of high temperature and high pressure, and is full of high-energy neutrons.
Particularly, reactor pressure vessel (RPV), a unique and non-replaceable part, is the last protection of the fission reactor core.
Its integrity directly determines the service time of the whole reactor.

Radiation damage to the key material of RPV is a major threat to the integrity of RPV.
Existing research revealed that the damage is initiated when a given lattice atom, namely, primary knock-on atom (PKA), is struck by a high-energy neutron \cite{stoller_primary_2012,knaster_materials_2016,nordlund_primary_2018,nordlund_historical_2019}.
Then, PKA will continue to perform a sequence of collisions with other atoms.
Afterwards, the system will generate the secondary, the third, and the subsequent higher-order knock-ons until all the energy initially imported to the PKA has been dissipated \cite{stoller_primary_2012,knaster_materials_2016}.
This process, called \textit{cascade collision}, usually stops within tens of picosecond ($1 \si{\ps} = 10^{-12}\,\si{\second}$) approximately.
It is almost impossible to observe in experiment.

To study material evolution behavior at atomic scale,
molecular dynamics (MD) \cite{peng_shockwave_2018,fu_molecular_2019} can be applied to the simulation of the \textit{cascade collision}.
Due to the high \change{computational} cost required by the simulation on large spatial scale and long temporal scale, parallel molecular dynamics simulation is especially essential.
There have been a number of parallel molecular dynamics programs \cite{plimpton1993fast, duan_redesigning_2018, niethammer_ls1_2014}.
Nevertheless, existing programs are limited in the functionality
\add{(e.g. direct defect analyzing, adaptive timestep length, batch execution of simulations)} and the execution efficiency.
Consequently, they are not applicable to the large-scale \textit{cascade collision} simulation.

This paper proposes MISA-MD, a new parallel MD implementation, to achieve the efficient and accurate \emph{cascade collision} simulation
on large spatial scale and long temporal scale.
\change{There are various potential functions used in MD simulation under diffrent fields,
such as Tersoff potential \cite{tersoff_new_1988} and Lennard-Jones (L-J) potential \cite{lennard1931cohesion}, for calculating the interaction among atoms.
To improve the simulation accuracy, MISA-MD adopted Embedded Atom Method (EAM) potential \cite{daw_embedded-atom_1984},
a complex but pretty accurate potential function, which can provide an effective interatomic description for metallic system.
}To improve the runtime performance, MISA-MD designed and realized a new hash based data structure for efficient atom storage and quick neighbor atom indexing.
Several acceleration and optimization strategies were also applied to ensure that MISA-MD can make full use of SW26010 processors on Sunway Taihulight supercomputer.

The key contributions of this paper are summarized as follows:
\begin{enumerate}
    \item A new hash based data structure for storing and indexing lattice-based atoms;
    \item A new method of EAM potential tables storage and interpolation on SW26010 processor;
    \item A new coloring method to avoid the write conflicts on computing processing elements (CPEs) of SW26010 processor.
    \item New double-buffer DMA and data reuse strategies to reduce the overhead of data transmission from main memory to local device memory (LDM) of CPEs.
\end{enumerate}

The remainder of this paper is organized as follows.
\cref{sec:background} presents the background knowledge on MD and Sunway Taihulight supercomputer, and discusses the related work.
\cref{sec:data-structure} introduces the new atom storage and neighbors indexing data structure.
\cref{sec:sunway} discusses the implementation on Sunway TaihuLight supercomputer and the optimization strategies.
\cref{sec:results} presents a performance analysis and validation of MISA-MD.
The last section summarizes the conclusion and the future work.

\section{Background} \label{sec:background}

\subsection{Molecular Dynamics Method and Challenges}
Molecular dynamics is a classic method for simulating particle systems.
It has been widely used in many domains, such as material science, chemistry, and biomolecular science \cite{rapaport_art_2004}.
A generic MD workflow is shown in \cref{fig:md_algorithm}.
In MD, an atom/molecule is treated as a particle.
For initialization, each particle is created with an initial coordinate and a velocity.
Then, the computation falls into a loop of time steps to solve Newton's equations of motion.
In each time step, the computation updates every particle as follows.
First, calculate the force applied to each particle to solve its acceleration $a$.
The force can be calculated by interactional potential functions (e.g. Lennard-Jones potential and EAM potential) in a particular system.
We will discuss the potential function in \cref{sec:eam-pot} in details.
Second, update the particle velocity via the integral of acceleration in a short $\Delta t$,
where $\Delta t$ is the length of a time step.
Third, update the particle coordinate using the integral of velocity in $\Delta t$.
Finally, the corresponding ensemble, \change{such as NVT or NPT ensemble}, is applied,
and some physical quantities may also be calculated before the loop moves into the next time step.

\begin{figure}[!t]
    \centering
    \begin{adjustbox}{scale=0.65,center}
        \subimport{tikz/}{md-diagram.tex}
    \end{adjustbox}
    \caption{A standard workflow of MD applications.
    In the time step loop, it will repeat to calculate force,
    update velocities and position and perform constraint condition as they need.}
    \label{fig:md_algorithm}
\end{figure}

The interaction forces to be computed among particles can typically fall into two categories:
long-range and short-range \cite{law_algorithm_2019}.
Long-range force calculation \change{requires interactions of each particle contributed by the global system}.
While in short-range force calculation, for any particle $p$,
only its neighbor particles that are located within a cut-off radius is considered to contribute to the force that applied to $p$.
In fact, short-range force calculation is \change{adopted} by most particle systems in MD simulation,
including particle interaction modeled by EAM or L-J potential.

Although most MD implementations are parallel,
it is still challenging to realize the simulation of large spatial scale and long temporal scale.
For spatial scale, the simulation is mainly restricted by memory storage.
Given a fixed memory capacity, the more particles can be stored (i.e, the less memory is required by a single particle),
the larger spatial scale MD simulation can reach.
For temporal scale, the faster the computation of each time step is,
the more time steps can be computed within the same time limitation.
For MD simulation with computation of short-range interaction forces,
the time cost of each time step is mainly affected by neighbor particle indexing and the force calculation.
To reduce the time cost of each time step, we must manage the particle data
and index neighbor particles as efficiently as possible.

\subsection{EAM Potential} \label{sec:eam-pot}

Different from pair potentials (e.g. Lennard-Jones and Morse), embedded-atom method potential is able to provide a better interatomic description for metallic system.
In EAM, the energy of an atom $i$ is determined by two aspects \cite{daw_embedded-atom_1984}:
(1) the embedded energy of the atom embed in electron cloud which represents the \emph{many-body effects} in the interaction,
and (2) the pairwise potential of atomic interaction.
EAM potential can be more complex compared with pair potentials.
In \cref{eq:eam_base}, $E_i$ denotes the energy of atom $i$,
and $r_{ij}$ represents the distance between atom $i$ and its neighbor atom $j$.
Three kinds of functions are presented in \cref{eq:eam_base}:
\begin{enumerate}
    \item $\rho_{\beta}(r)$: It describes the contribution to the electronic density
    at the site with distance of $r$ from atom $j$ whose type is $\beta$.
    \item $F_{\alpha}(\rho)$: The embedding energy function $F$ returns
    the energy associated with placing an atom of type $\alpha$ in the electron environment described by $\rho$.
    \item $\phi_{\alpha \beta }(r_{ij})$: $\phi_{\alpha \beta }$ is pair-wise potential function, which describes
    the pair potential between two atoms $i$ of type $\alpha$ and $j$ of type $\beta$.
    And $r_{ij}$ denotes the distance of atom $i$ and $j$.
\end{enumerate}

\begin{equation}\label{eq:eam_base}
    E_{i}=F_{\alpha}\left(\sum_{j\neq i} \rho_{\beta}(r_{ij})\right) + {\frac {1}{2}}\sum_{i\neq j}\phi _{\alpha \beta }(r_{ij})
\end{equation}

The total force $\vec{F_i}$ applied to atom $i$ can be expressed by
\change{
the negative gradient of potential energy of the simulation system, as descripted in \cite{plimpton_parallel_1992, zhigilei2014introduction}.
Then, we can calculate the force that is applied to atom $i$ from neighbor atom $j$ using \cref{eq:eam_force_ij},
}
where:
    $F_i(\rho)$ and $\rho_i(r)$ denote the embedding energy function and electronic density function
    associated with type of atom $i$ respectively;

    $\phi_{ij}(r)$ denotes the pairwise potential function associated with types of atom $i$ and $j$;
    $\vec{r}_i$ and $\vec{r}_j$ represent the position vector of atom $i$ and $j$, respectively, and $r_{ij}$ is the distance of atom $i$ and $j$;
    $\bar\rho_i$ denotes the electronic density contributed by all neighbor atoms of atom $i$.

\begin{figure*}[!t]
    \normalsize
    \begin{equation} \label{eq:eam_force_ij}
        \begin{split}
        \vec{F}_{ij}  = - \left[
            \left.\frac{\partial F_i (\rho) }{\partial \rho}\right|_{\rho=\bar\rho_i }
            \left.\frac{\partial \rho_j (r) }{\partial r} \right|_{ r =r_{ij} } +
            \left.\frac{\partial F_j (\rho) }{\partial \rho}\right|_{\rho=\bar\rho_j }
            \left.\frac{\partial \rho_i (r) }{\partial r} \right|_{ r =r_{ij} } +
            \left.\frac{\partial \phi_{ij}(r) }{\partial r}\right|_{ r=r_{ij} }
            \right]
            \frac{ \vec{r}_i -\vec{r}_j }{ r_{ij} }
        \end{split}
    \end{equation}
\end{figure*}

In a system with $n$ atom types, $n \times \left(n + 1\right)/2 + n + n + n$ functions should be determined,
including
$n \times \left(n + 1\right)/2$ partial derivative of pairwise potential functions,
$n$ partial derivative of embedding energy functions,
$n$ electronic density functions and $n$ partial derivative of electronic density functions.
For example, a system with two different atom types (i.e., $\alpha$ and $\beta$),
we have to calculate following 9 functions:
$\rho_{\alpha}(r)$, $\rho_{\beta}(r)$,
$\dfrac{\partial \phi_{\alpha \alpha}(r)}{\partial r}$,
$\dfrac{\partial \phi_{\alpha \beta}(r)}{\partial r}$,
$\dfrac{\partial \phi_{\beta \beta}(r)}{\partial r}$,
$\dfrac{\partial F_{\alpha}(\rho)}{\partial \rho}$,
$\dfrac{\partial F_{\beta}(\rho)}{\partial \rho}$,
$\dfrac{\partial \rho_{\alpha} (r)}{\partial r}$,
$\dfrac{\partial \rho_{\beta} (r)}{\partial r}$.

To calculate the force of each atom using EAM potential according to \cref{eq:eam_force_ij},
the following steps must be performed:
\begin{enumerate} \label{sec:eam_computation_steps}
    \item Calculate electronic density contribution from neighbor atoms for each atom $i$:
    $\bar \rho_i = \sum_{k\neq i} \rho_{k}\left(r_{ik}\right)$;
    \item Calculate value of partial derivative of embedding energy function for each atom $i$ at $\bar \rho_i$:
    $ \left.\frac{\partial F_i (\rho) }{\partial \rho}\right|_{\rho=\bar\rho_i } $;
    \item Calculate value of partial derivative of pairwise potential function for every two atoms $i$ and $j$ in cut-off radius:
    $ \left.\frac{\partial \phi_{ij}(r) }{\partial r}\right|_{r =r_{ij} } $;
    \item For every two atoms $i$ and $j$ in cut-off radius,
    calculate each other's contribution value of partial derivative of electronic density function via:
    $ \left.\frac{\partial \rho_i (r) }{\partial r} \right|_{ r =r_{ij} } $ and
    $ \left.\frac{\partial \rho_j (r) }{\partial r} \right|_{ r =r_{ij} } $;
    \item Here, all terms in \cref{eq:eam_force_ij} are avaiable,
    then we can calculate force $\vec F_{ij}$ of two atoms $i$ and $j$ in cut-off radius via \cref{eq:eam_force_ij}.
    For each atom $i$, traverse all its neighbor atoms $j$, add force contributed by $j$ to atom $i$.
    Then we can obtain the total force for each atom.
\end{enumerate}

\subsection{Sunway TaihuLight and SW26010 Many-Core Processor}
Sunway TaihuLight \cite{fu_sunway_2016,yang_application_2018} is a supercomputer with a peak performance of 125.3 PFlops
and a sustained Linpack performance of 93 PFlops.
It was manufactured by \emph{National Research Center of Parallel Computer Engineering and Technology (NRCPC)}, and is hosted at National Supercomputing Center in Wuxi.
It comprises 40960 homegrown SW26010 many-core processors \cite{yang_application_2018}
which are connected by a 2-level fat-tree topology network.

The SW26010 processor uses on-chip heterogeneous many-core architecture.
A SW26010 consists of four core groups (CGs) that are connected via the network on chip (NoC) and a system interface (SI) \cite{noauthor_homegrown_2015}.
A CG consists of a management processing element (MPE), a computing processing element (CPE) cluster, and a memory controller (MC).
Every CG has its own memory space, and the main memory is connected to the MPE and CPE cluster through the MC.
Each CPE cluster contains 64 CPEs that are arranged in an $8 \times 8$ grid.
The SI is used to connect the processor itself and other devices outside.

The MPE and CPE are both 64-bit reduced instruction set computer (RISC) cores.
CPE is simpler than MPE and can only run in user mode.
Moreover, CPE is a core with only \SI{64}{KiB} user-controlled local device memory (LDM),
while MPE is connected with a 8 GB DDR3 memory via MC.
For memory accessing, CPE can access main memory directly by global load/store instructions (gld/gst),
or by direct memory access (DMA) which can be much faster than gls/gst.
At CPE cluster level, each CPE has 8 column communication buses and 8 row communication buses
to provide fast register communication across the $8 \times 8$ CPE grid,
which is a significant capability to share data inside CPE cluster.

\subsection{Related Work}
Many molecular dynamics programs were developed by numerous research teams.
Sandia National Labs developed LAMMPS \cite{plimpton1993fast} for classical molecular dynamics simulation.
However, the overhead of maintaining the neighbor list is very high during a large-scale simulation (e.g. $10^{10}$ particles).
Duan et al.\cite{duan_redesigning_2018} adapted LAMMPS for Sunway many-core architecture.
The new program achieved over 2.43 PFlops performance for a Tersoff simulation using 16,384 nodes.
However it only supports L-J potential and Tersoff potential, rather than the EAM potential that is more suitable for metallic systems.
GROMACS\cite{berendsen1995gromacs} is a widely-used molecular dynamic program in chemical and biomolecular.
SW\_GROMACS\cite{zhang_sw_gromacs_2019} accelerated GROMACS on Sunway TaihuLight supercomputer.
\new{Besides GROMACS, another significant MD software package for biomolecular simulation is NAMD \cite{phillips_namd_2002},
it is written using the Charm++ parallel programming model \cite{kale_charm_1993} and is the recipient of Gordon Bell award in 2002.}
ls1 Mardyn\cite{niethammer_ls1_2014} is another massively parallel molecular dynamics program
using L-J potential\cite{lennard1931cohesion} to solve the interaction among particles.
Its main target is the simulation of thermodynamics and nanofluidics.
It achieved a $2.1 \times 10^{13}$-particle simulation on Hazel Hen supercomputer by using 172,032 CPU cores\cite{tchipev_twetris:_2019}.
However, both GROMACS and ls1 Mardyn cannot simulate the metallic systems with EAM potential.
Crystal MD\cite{hu_crystal_2017,hu_kernel_2017} is a parallel MD program for metal with BCC structure.
It carried out a four-trillion-atom simulation on Sunway TaihuLight supercomputer.
\new{SPaSM \cite{germann_25_nodate} is a MD implementations and can achieve simulation of more \change{than} 100 billions on BlueGeneL.}
But there is a lack of functionality of cascade collision simulation for both CrystalMD and SPaSM.

To support indexing of neighbor atoms within a cut-off radius, several classical data structures were proposed,
such as the neighbor list (or verlet lists)\cite{verlet_computer_1967} used in LAMMPS \new{, NAMD and GROMACS},
and link cell \cite{hockney_quiet_1974, niethammer_ls1_2014} used in ls1 Mardyn\new{ and SPaSM}.
In the neighbor list, each atom will create and maintain a reference to its neighbor atoms
and store them in a list.
The neighbor list requires a lot of memory.
Link cell splits the simulation box into a number of cells.
In force computation step, for any atom $a$ in cell $c$,
link cell must traverse all atoms in cell $c$ and all atoms in neighbor cells of $c$.
which may index many unnecessary atoms out of cut-off radius.
Thus new design of data structure for efficient atoms storing and quick neighbor atoms indexing is essential.

\section{Hash Based Data Structure} \label{sec:data-structure}

\begin{figure}
    \centering
    \begin{adjustbox}{scale=0.76,center}
        \subimport{tikz/}{hash-based-data-structure.tex}
    \end{adjustbox}
    \caption{Atoms data structure and \change{indexing} method}
    \label{fig:md_data_structure}
\end{figure}

To support efficient atom storing and quick indexing of neighbor atoms,
a new hash-based data structure is designed for MISA-MD.
The hash-based data structure consists of two parts.
The first one is Lattice List{} used to efficiently store atoms,
and the second one is Neighbor Offset Index{} targeting quick neighbor atom indexing.
In lattice list{}, a lattice array will be established as a hash array,
and a red-black tree will also be setup for storing elements with hash collision.
The hash function maps each atom to its nearest lattice in cartesian coordinate system.
\cref{fig:md_data_structure} illustrates this idea in a 2d simulation box.
For neighbor offset index{}, the offset value of lattice id is pre-calculated for later neighbor atoms indexing.
To traverse neighbor atoms of a atom, the id of nearest lattice of the atom will be calculated first,
then we can obtain neighbor lattices by adding the nearest lattice id and offset lattice id.
\cref{fig:data_structure_offset} shows the lattice offset for calculating neighbor atoms.

\begin{figure}
    \centering
    \begin{adjustbox}{scale=1.25,center}
        \subimport{tikz/}{md-lattice-coordinate.tex}
    \end{adjustbox}
    \caption{Lattice Offset Index for calculating neighbor atoms for BCC structure in 2d domain.
    The number in brackets under each lattice size is the coordinate of the corresponding lattice site}
    \label{fig:data_structure_offset}
\end{figure}

The complete process of MD calculation based on BCC structure
using lattice list{} and neighbor offset index{} data structure is list as following.
\paragraph{Hash Initialization}
At the beginning, a hash array for lattice will be allocated.
In our cascade collision simulation, body centered cubic (BCC) structure lattice is constructed for iron-based or tungsten-based material,
in which there is an additional lattice site located in the center of each cube unit.
Each element in hash array corresponds to a BCC lattice site.
For example, in the 2d simulation domain\footnote{In fact, it is a 3d domain in practice.},
the \textit{coordinate} of each lattice site can be recorded, as shown in \cref{fig:data_structure_offset},
and the coordinate can then be converted to hash array index by:
$$ index = 2 \times box_x \times y + x$$
where $box_x$ and $box_y$ are the lattice size of simulation box of current MPI process at $x$ and $y$ dimension, respectively.
and $x$ and $y$ are the corresponding lattice coordinates at each dimension.
In \cref{fig:data_structure_offset}, $box_x$ and $box_y$ are both equal to 5,
a lattice with coordinate $(3,4)$ is mapped to the 43rd ($2 \times 4 \times 5 + 3 =43$) element in hash array.
And in 3d simulation domain, the index can be calculated by:
\begin{equation}
    index = 2 \times box_x \times box_y \times z + 2 \times box_x \times y + x
\end{equation}
The advantage of this BCC lattice to index mapping method is that
we can store all lattice associated atoms in a compact memory mode.
We call the calculated index of hash array as \textit{lattice id}.
Meanwhile, the lattice id can also be converted to lattice coordinate easily.

The hash function can calculate \change{lattice} coordinate of the nearest lattice site of a atom and
return lattice id of the nearest lattice site.
Hash clash occurs when two distinct atom coordinates passed into the hash function produce identical lattice id outputs.
A empty red-black tree will be created for storing hash clash atoms and indexed by lattice id.
In this case, one of these atoms is stored in hash array and the left ones are stored in the red-black tree
when hash clash occurs.

Atoms are organized by this hash table.
In system initialization step of simulation, all atoms are placed at the corresponding lattice site.
In other words, there is no hash clash.
It is because that any two atoms must correspond to different lattice sites.
\change{Therefore}, all atoms data are stored in hash array at initialization step.

\paragraph{Neighbor Offset Index{} Initialization}
Under this regular lattice structure, list of offset lattice ids for indexing neighbor lattices in cut-off radius can be pre-calculated.

\paragraph{Iterate Neighbor Atoms}
We can calculate force or other interactions of each atoms in simulation domain
via lattice list{} and neighbor offset index{}.
\cref{alg:iterate_atoms_via_data_structure} show the algorithm of iterating all atoms and neighbor atoms
to calculate force.

\input{algorithm_hash_iterate}

Note that, for \change{obtaining} interaction contributions of neighbor atoms from neighbor MPI processes conveniently,
ghost region is involved.
The atoms in ghost region is also indexed by lattice list{} together with local atoms located in real simulation domain.

In the algorithm, the calculation consists of three parts:
interactions of atoms from both hash array, interactions of atoms from hash array and hash clash,
and interactions of atoms from both hash clash.
In the first part of interactions computation, just iterate over the hash array and
traverse neighbor lattices of each lattice to obtain interaction contribution from neighbor atoms that are stored in hash array.
In interactions calculation of atoms from hash array and hash clash,
iterate atoms in hash clash, if the atom is not in ghost area, find its nearest lattice via hash function,
then we can traverse neighbor atoms in hash array by the nearest lattice as center lattice.
At last, it falls into the interactions of atoms in hash clash.
It iterate the atoms that are not in ghost region.
For each atom $a$ in iteration, we can find the nearest lattice of this atom via hash function.
Then we can calculate ids of neighbor lattices by id of the nearest lattice and precalculated offset lattice ids,
and select atoms in hash clash with the neighbor lattice ids,
and add interactions contribution of these ones to atom $a$.

\paragraph{Hash Update}
After finishing each simulation step, the atoms coordinate can get changed,
thus updating of hash index is necessary.
\cref{alg:update_hash_index} presents the steps of hash updating.
The hash updating can be divided into two steps:
1) Traverse all elements (or atoms) in hash array.
For each valid element, re-calculate its lattice id via hash function.
If array index of the current element in hash array does not equal to the new calculated lattice id,
just copy this element to hash clash and tag this element as invalid.
2) Traverse all \change{of} atoms in hash clash.
For each atom in hash clash that is not located in ghost region, we can calculate its lattice id via hash function.
If the element in hash array indexed by the re-calculated lattice id is invalid,
then copy the atom data to hash array, set the element in hash array as valid, and remove this atom from hash clash.

These two steps above in \cref{alg:update_hash_index} can make sure:
1) If two or more than two atoms have the identical hash value,
only one of them is stored in hash array, and the left ones are stored to hash clash.
2) If there is a lattice id (index value in hash array) but no atom in simulation domain corresponds to this lattice id,
the element indexed by the lattice id in hash array must be invalid.
3) For each non-ghost atom in hash clash, if there is a invalid element in hash array that is indexed by the hash value of this atom,
the algorithm can remove the atom from hash clash and add it back to hash array.

\add{
The advantage of the hashed-based approach is that it can support quick neighbor searching
and efficient atom storing, which use less memory compared than neighbor list method.
Denote $M$ as the number of neighbor in cut-off radius and $N$ as the number of atoms in simulation system.
The memory occupation can be divided as two parts: system atoms and neighbor indexing.
For system atoms, both the new hash-based approach and the neighbor list method
need $O\left(N\right)$ memory to save system atoms.
But for neighbor indexing, the memory cost of the new hash-based approach will be $O\left(M\right)$,
and memory cost of the neighbor list method will be $O(MN)$.
However, the disadvantage is also obvious.
The hash-based approach is specific to solids and would not work well for the system where the atoms are not on a regular lattice, such as liquids.
Because it will cost expensive computation to handle hash claims when the atoms are not on a regular lattice.
}
\input{algorithm_hash_update}

\section{Sunway Acceleration and Optimizations} \label{sec:sunway}
To achieve efficient MD simulation on large-scale clusters and make full use of the computation ability of SW26010 processor,
accelerating computation of MISA-MD is essential.
Results from profile tools show that the EAM potential computation in MISA-MD,
consume more than 80\% computation time.
In SW26010 processor, 256 CPEs in 4 core group contribute more than 90\% peak performance of the overall processor\cite{lin_evaluating_2018}.
Therefore, the code for accelerating EAM interaction computation on sunway CPEs is implemented.
In current version of MISA-MD, we only consider the acceleration for interaction of atoms from hash array
due to the fact of low rate of hash clash.

Moreover, it is of great significance to accelerate EAM computation on SW26010.
Because it is another significant approach to extend MD simulation to longer temporal scale besides quick neighbor atoms indexing.

\subsection{Potential Tables Storage and Interpolation}
In EAM computation, the $n \times \left(n + 1\right)/2 + n + n + n$ functions for $n$ different atom types
are determined by spline interpolation
from a table with dispersed values called ``potential table''.

For each to be determined function as well as its partial derivative,
$N$ equidistant dispersed points are presented in potential table.
For instance, to determine electronic density function of atom type $\beta$,
point set $P_\beta=\{(r_1,\rho_1), (r_2,\rho_2), (r_3,\rho_3), \cdots, (r_N,\rho_N) \}$ is given in table,
where $r_i$ is distance and $\rho_i$ is value of electronic density under distance $r_i$, $i=1,2,\cdots,N$.
We are expected to find a electronic density function $\rho_{\beta}(r)$ and
its partial derivative function $\dfrac{\partial \rho_{\beta} (r)}{\partial r}$
for $ r \in \left[r_1,r_N \right]$.

To perform interpolation on a point set $P=\{(x_1,y_1), (x_2,y_2), (x_3,y_3), \cdots, (x_n,y_N) \}$
with condition of $x_1 < x_2 < \cdots < x_N$ and $x_2 - x_1 = x_3 - x_2 = \cdots = x_N - x_{N-1} = h$,
a cubic polynomial function $S_i(x) = a_i(x-x_i)^3+b_i(x-x_i)^2+c_i(x-x_i)+d_i$
is selected for each segment specified by two adjacent point to approximate the undetermined function in spline interpolation.
Then we can calculate the coefficients $a_i$, $b_i$, $c_i$, and $d_i$ of $S_i(x)$ on each segment,
as well as coefficients $e_i$, $f_i$ and $g_i$ for partial derivative of $S_i(x)$:
$S_i'(x)=e_i(x-x_i)^2+f_i(x-x_i)+g_i$.
After each functions is determined, then these seven coefficients on each segment can be stored for later EAM computation.

In our cases, to simulate cascade collision of a pure Fe system using MISA-MD,
a potential table file from \cite{bonny_ternary_2009} is provided, which contains 3 potential tables
(one pair-wise potential table for Fe-Fe, one for embedding energy for Fe and one for electronic density for Fe).
Each potential table above contains \SI{5000}{} dispersed data values used for interpolation to
determine corresponding potential function and it partial derivative function.
This means that, we need at least \SI{820}{KiB} memory to store the interpolation coefficients for single-atom-type system
(3 potential tables $\times$ 7 coefficients $\times$ 5000 segments $\times$ 8 bytes per double precision floating point value),
which is much large than \SI{64}{KiB} LDM of CPE.
Further, if the target system is an alloy system containing multiple types of atoms, the storage of coefficients can be much larger.
Thus, the interpolation coefficients can not be loaded into the LDM of CPE at one time.

To overcome the small LDM of CPE, we copy the origin values of one table to LDM of CPE
before using it, whose size is only \SI{39}{KiB}.
This thought is \change{inspired} by \cite{li_massively_2018}.
Then, all the coefficients of $S_i(x)$ and $S_i'(x)$ can be calculated on the fly
using the origin data and interpolation method.
But the shortcoming of this method is obvious.
It can only calculate single-atom-type system,
because for alloy system, the LDM of CPE cannot hold all necessary origin data.

To deeply optimize the memory storage of origin potential table data,
we exchange order of offset list iteration and atoms iteration in \cref{alg:iterate_atoms_via_data_structure}
(move the offset list iteration to the outermost loop and move atoms iteration of x,y,z dimension to inner loop).
Because the origin potential table is associated with atoms distance $r$,
in each offset list iteration, we can obtain the minimum and maximum distance of neighbor atoms in atoms iteration \change{firstly},
and then we can only load part of the origin potential data that is in this distance range, rather than the whole origin data.
With partial potential data loading in each offset list iteration,
we can free up more memory space that can be used for atoms storage.
But the number of potential data copies from main memory to LDM can increase due to the partial potential data loading.

To overcome the large overlap of potential data copies,
we divide the origin data of potential tables into some blocks
and each CPE pre-load a block of origin potential tables at initialization step.
Then the origin potential tables are stored on CPEs distributedly.
And in latter offset list iteration,
the partial potential data is loaded from some specific CPEs via fast register communication, instead of main memory.

\subsection{Tasks Assignment on CPEs}
In fact, Newton's third law can be applied for force calculation.
As described at the end of \cref{sec:eam_computation_steps}, thanks to Newton's third law,
the amount of calculation of atoms' electronic density contribution at step 1) and
and force contribution at step 5) as well as pair-wise potential at step 3) and derivative of electronic density at step 4)
can halve.
But the problem of threads' write conflict on sunway CPEs must be solved before involving this trick.
When we assign computation tasks of one MPI processor to 64 CPE \change{threads}
\add{(In sunway native programming, one CPE can launch a light-weight working thread using Athread library \cite{cai_openacc_2018}.)},
spatial decomposition of simulation domain is achieved.
We may split the domain owned by the MPI processor into 64 blocks,
and each CPE \change{thread} would carry out the EAM calculation of one block of them.
But the write conflict will occur when writing calculated results back to main memory on MPE
if there is common neighbor atom for two adjacent CPE \change{threads},
as shown in \cref{fig:write_conflict_of_CPE_thread}.

To overcome this difficulty, we proposed a coloring method for sunway CPE \change{threads},
which is \change{inspired} by SDC (Spatial Decomposition Colouring) method\cite{liu_efficient_2011,hu_efficient_2009}
used in OpenMP program model on multi-core platforms.
\cref{fig:coloring_of_CPE_thread} shows an example of the coloring result.
In coloring method, we split the domain space into 128 blocks in z dimension,
where 128 is twice of the number of CPEs,
and each block is colored as red or blue and make sure there is no adjacent blocks with the same color.
Then the calculation is divided into two steps to avoid write conflict.
In the first step, each CPE \change{thread} is assigned a block with red color,
and the next step, each CPE \change{thread} is assigned a block with blue color.
With these two steps, all blocks can be calculated without write conflict.

\begin{figure}
    \centering
    \begin{adjustbox}{scale=0.8,center}
        \subimport{tikz/}{write-conflict.tex}
    \end{adjustbox}
    \caption{Write Conflict of \change{CPE threads}}
    \label{fig:write_conflict_of_CPE_thread}
\end{figure}

\begin{figure}
    \centering
    \begin{adjustbox}{scale=1,center}
        \subimport{tikz/}{color-athread.tex}
    \end{adjustbox}
    \caption{Coloring \change{CPE threads}}
    \label{fig:coloring_of_CPE_thread}
\end{figure}

\subsection{Double-Buffer DMA and Data Reuse Strategies}
Naturally, the domain block assigned to a CPE may contain a large number of atoms which
can not be \change{transmitted} to CPE at one time due to the restriction of \SI{64}{KiB} LDM.
Thus, the domain block is \change{split} into numbers of small data block for CPE computation.
In traditional method, CPE would request a memory block of atoms data from main memory,
then it start computation once the atoms data transmission finish.
After computation completes, atoms data will put back to main memory and then
prepare to access next data block for next round computation repeatedly
until all data blocks assigned to this CPE is completed.
We found that the repeatedly DMA data accessing has a high overhead, which may leading a poor performance.
Therefore, a double-buffer DMA strategy is taken to overlap the DMA data transmission and EAM potential computation,
as shown in \cref{fig:sunway_double_buffer}.
In double-buffer DMA strategy, the LDM is divided into three blocks,
one is for origin potential data, the second and the third ones are both buffer memory for each other.
Expect for the initial DMA getting and the last DMA putting operation,
when performing computation of current round, the DMA putting operation of previous round or
the DMA getting operation of next round can be performed simultaneously.
By overlaping DMA data transmission and EAM potential computation,
the computation efficiency on CPEs is greatly improved.

\begin{figure}
    \centering
    \begin{adjustbox}{scale=0.6,center}
        \subimport{tikz/}{sunway-double-buffer.tex}
    \end{adjustbox}
    \caption{Double-buffer DMA strategy}
    \label{fig:sunway_double_buffer}
\end{figure}

Double buffer strategy can overlap DMA and computation effectively.
To reduce the overhead of DMA further, another optimization, called data reuse, is applied.
Considering atoms data block of current calculation round,
a part of them can be actually the ghost atoms data of next calculation round,
and its ghost data can also be the `real' atom data of next round.
Thus, we can keep the these two part of data in the LDM of CPE for usage of next round calculation,
which can reduces the amount of data transferred to LDM in each DMA operation.

\section{Evaluation} \label{sec:results}
This section presents the evaluation on single core group performance, scalability, and accuracy of cascade collision simulations.
As a comparison, we also compared memory usage and performance with LAMMPS on Sunway and Intel platforms.

\subsection{Accuracy and Case Study}

\begin{figure}
    \centering

    \begin{subfigure}[]{0.49\textwidth}
        \begin{adjustbox}{scale=1,center}
            \includegraphics[width=\textwidth]{figures/frenkel-number-misa-md-401}
        \end{adjustbox}
        \caption{Evolution of number of displacement cascades' defects for different PKA energy from MISA-MD }
        \label{fig:frenkel_defects_misa-md}
    \end{subfigure}
    \begin{subfigure}[]{0.49\textwidth}
        \begin{adjustbox}{scale=1,center}
            \includegraphics[width=\textwidth]{figures/frenkel-number-lammps-401}
        \end{adjustbox}
        \caption{Evolution of number of displacement cascades' defects for different PKA energy from LAMMPS}
        \label{fig:frenkel_defects_lmp}
    \end{subfigure}
    \caption{Evolution of number of displacement cascades' defects for different PKA energy at 600K temperatures}
    \label{fig:frenkel_number_evolution}
\end{figure}

In order to validate the accuracy of MISA-MD,
we have performed several cascade collision simulations of different PKA energies and directions.
In our cases, the PKA energy settings are \SI{1.0}{\kilo\electronvolt}, \SI{5.0}{\kilo\electronvolt}, \SI{10.0}{\kilo\electronvolt},
\SI{15.0}{\kilo\electronvolt}, \SI{20.0}{\kilo\electronvolt}, \SI{25.0}{\kilo\electronvolt} and \SI{30.0}{\kilo\electronvolt},
and PKA directions includes $\langle 1,2,2 \rangle$, $\langle 1,3,5 \rangle$ and $\langle 2,3,5 \rangle$.
In each simulation instance, a $80 a_0 \times 80 a_0 \times 80 a_0$ (where $a_0$ is lattice constant) simulation box containing \SI{1024000}{} Fe atoms is involved,
and the primary knock-on atom is placed at position of $(40a_0,40a_0,40a_0)$ (\change{center} of simulation box) with a corresponding velocity associated with PKA energy.
For each PKA energy and each PKA, simulations are performed three times to obtain the average results and
analyze the number of frenkel pairs in simulating box over evolution time.
As shown in \cref{fig:frenkel_defects_misa-md}.
Besides, we also tested on LAMMPS using EAM potential with the same input parameters.
\cref{fig:frenkel_defects_lmp} shows the evolution of frenkel pairs numbers for different PKA energies at 600K temperatures from simulation results of LAMMPS.

Comparing \cref{fig:frenkel_defects_misa-md} and \cref{fig:frenkel_defects_lmp},
their evolution curves of frenkel pairs number analyzed from simulation results have the same trend under varies of PKA energies.
The number of frenkel pairs of MISA-MD and LAMMPS both grow to a peak rapidly after a short time of PKA generation
and then quench from peak value to stable value.
As a case of this process, \cref{fig:frenkel_defects_visualization} visualizes
process of cascade collision simulated by MISA-MD,
in which the spatial distributions of frenkel pairs at different evolution stages
under PKA energy of 15kev and incidence direction of $\langle 1,3,5 \rangle$ are presented.
Moreover, with a large PKA energy, the peak would appear later with a larger peak,
and a higher value of frenkel pairs survives at stable stage.
By comparing peak time, peak value, stable time and stable value of MISA-MD and LAMMPS quantitatively,
they are all \change{similar} and within acceptable tolerances.
And the number of survived frenkel pairs pairs also meet the classical NRT model established by Norget \cite{norgett_proposed_1975}.

\begin{figure}
    \centering
    \begin{subfigure}[]{0.24\textwidth}
        \begin{adjustbox}{scale=0.8,center}
            \includegraphics[width=\textwidth]{figures/1}
        \end{adjustbox}
        \caption{system configuration after \SI{0.1}{\ps} of PKA generation within 424 Frenkel pairs.}
        \label{fig:frenkel_defects_1}
    \end{subfigure}

    \begin{subfigure}[]{0.24\textwidth}
        \begin{adjustbox}{scale=0.8,center}
            \includegraphics[width=\textwidth]{figures/9}
        \end{adjustbox}
        \caption{system configuration after \SI{0.8}{\ps} of PKA generation within 3307 Frenkel \change{pairs}.}
        \label{fig:frenkel_defects_2}
    \end{subfigure}
    \begin{subfigure}[]{0.24\textwidth}
        \begin{adjustbox}{scale=0.8,center}
            \includegraphics[width=\textwidth]{figures/20}
        \end{adjustbox}
        \caption{system configuration after \SI{5.5}{\ps} of PKA generation within 190 Frenkel \change{pairs}.}
        \label{fig:frenkel_defects_3}
    \end{subfigure}
    \begin{subfigure}[]{0.24\textwidth}
        \begin{adjustbox}{scale=0.8,center}
            \includegraphics[width=\textwidth]{figures/40}
        \end{adjustbox}
        \caption{system configuration after \SI{26.0}{\ps} of PKA generation within 26 Frenkel \change{pairs}.}
        \label{fig:frenkel_defects_4}
    \end{subfigure}
    \caption{MISA-MD visualization of system evolution at different time under PKA energy of 15kev and direction of $\langle 1,3,5 \rangle$.}
    \label{fig:frenkel_defects_visualization}
\end{figure}

\subsection{Single Core Group Evaluation}
To test the computational efficiency of code accelerated by $8 \times 8$ CPEs,
we performed performance test on single core group with and without CPEs acceleration.
In this test case, a $200 a_0 \times 200 a_0 \times 256 a_0$ (where $a_0$ is lattice constant) simulation box was involved
and simulations of 10 time steps was performed.
\cref{tab:single_node_evaluation} shows the execution time and speedup of case \change{with} MPE only
and case with MPE plus 64 CPEs,
which indicates our MISA-MD code obtains approximate 56 acceleration speedup with \change{computation} on CPEs.

\begin{table}[h]
    \caption{Speedup of MISA-MD on single core group with and without CPEs.}
    \begin{tabular}{lll}
        \hline
                 & MPE only & MPE+64CPEs  \\
        \hline
        Execution time (seconds)  & 1192.33  &    21.2939   \\
        Speedup    &  1 & 55.99 \\
        \hline
    \end{tabular}
    \label{tab:single_node_evaluation}
\end{table}

\subsection{Comparison with LAMMPS}
In this subsection, we compare the memory usage and computational performance of MISA-MD and LAMMPS
on Sunway and Intel platforms.
For LAMMPS, the current latest version, \textit{stable\_3Mar2020}, is selected to be compared.
As the EAM computation of LAMMPS does not supported Sunway CPEs architecture,
thus, we \change{perform} performance tests of LAMMPS and MISA-MD using MPEs only on Sunway platform.

\subsubsection{Memory usage}
The memory usage of MISA-MD and LAMMPS is compared at runtime to demonstrate the superiority of our hash based data structure.
We first apply the same scale of BCC box to MISA-MD and LAMMPS,
and monitor the average memory usage of both programs.
In case 1 and case 2 in \cref{tab:memory_usage_of_md}, simulation box of $600 \times 600 \times 600$ containing $4.32 \times 10^8$ atoms is \change{applied},
and 120 MPI processors is involved for both MISA-MD and LAMMPS.
The results of case 1 and case 2 show that the memory usage of LAMMPS is 2.7 times larger than MISA-MD.
In the second test case set, as shown in case 3 and case 4 in \cref{tab:memory_usage_of_md},
we use the full node memory for both MISA-MD and LAMMPS simulation,
and \change{record the maximum} simulation box they can reach.
The results show that, given the same memory resource, MISA-MD can simulate much larger scale than LAMMPS.
\change{In more detail}, the number of atoms MISA-MD can simulate is 4.39 times than LAMMPS can reach.

\begin{table}[h]
    \caption{Memory Usage of MISA-MD Compared with LAMMPS on TianHe-2}
    \begin{threeparttable}[t]
    \centering
    \begin{tabular}{llcccc}
        \toprule
         No.    & Program  & Cores & Box & Mem usage & Mem per atom \\
                &          &       &         & (GiB/core)& (Bytes/atom) \\
        \midrule

        1   & MISA-MD & 120 & $600^3$  & 0.64 & 191.2 \\
        2   & LAMMPS  & 120 & $600^3$  & 1.77 & 527.2 \\
        3   & MISA-MD & 120 & $1094^3$ & 2.66 & 130.9 \\
        4   & LAMMPS  & 120 & $668^3$  & 2.66 & 574.9 \\
        \bottomrule
    \end{tabular}

   \end{threeparttable}
    \label{tab:memory_usage_of_md}
\end{table}
For computing efficiency, LAMMPS uses neighbor lists (Verlet lists) to keep track of nearby particles.
In the above test cases, the average number of each atom’s neighbor is 56 in LAMMPS.
Thus, it would takes $56 \times 4 = 224$ bytes to index neighbor atoms for each atom,
in which each neighbor index takes 4 bytes.
In test case 4 in \cref{tab:memory_usage_of_md}, the neighbor index memory occupies $38.86\%$ of the total memory.
While in MISA-MD, each atom only occupy 104 bytes for storing basic atoms information (such as velocity, electron charge density),
and neighbor atoms is indexed by index offsets which only consume several KiB memory.
Therefore, the new hash based data structure in MISA-MD can efficiently reduce the memory usage
and can expand MD simulation to larger scale.

\subsubsection{Computational Performance}
\begin{figure}
    \centering
    \begin{subfigure}[]{0.49\textwidth}
        \begin{adjustbox}{scale=0.70,center}
            \subimport{tikz/}{md-perf-compare-intel.tex}
        \end{adjustbox}
        \caption{Computational performance of MISA-MD and LAMMPS on Tianhe 2.}

        \label{fig:strong_scalability}
    \end{subfigure}

    \qquad
    \begin{subfigure}[]{0.49\textwidth}
        \begin{adjustbox}{scale=0.70,center}
            \subimport{tikz/}{md-perf-compare-sw.tex}
        \end{adjustbox}
        \caption{Computational performance of MISA-MD and LAMMPS on Sunway Taihulight.
        Acceleration of MISA-MD using CPEs is disabled for principle of fairness.}

        \label{fig:strong_scalability}
    \end{subfigure}

    \caption{Computational performance comparison of MISA-MD and LAMMPS on Sunway Taihulight and Intel platforms
    with $x$ axis logarithmic scaled.
    Simulation time values are annotated on the top of bars.}
    \label{fig:md_perf_compare}
\end{figure}

The execution time is another impact to expand the scale of MD simulation.
We expect simulation program can execute as fast as it could,
thus it can simulate more time steps in the same spatial scale or
finish as soon as possible in the same temporal and spatial scale under the given computing resources.

To compare the performance between MISA-MD and LAMMPS,
test cases with fixed number of atoms are designed for both MISA-MD and LAMMPS simulation.
We start our tests for MISA-MD and LAMMPS on Sunway-Taihulight and TianHe-2 supercomputer.
\new{
On Sunway-TaihuLight, both MISA-MD and LAMMPS were compiled using ``-O2'' optimization flag by sw5gcc/sw5g++(a customized C/C++ compiler based on gcc version 5.3.0).
It is worth noting that, CPEs acceleration of MISA-MD is disabled for principle of fairness because of the lack of EAM calculation acceleration on CPEs of LAMMPS.
On TianHe-2, both the programs were also compiled using ``-O2'' optimization flag by the same compiler(gcc 5.3.0).
}

By giving varies of atoms number and involving 64, 128, 256, 512 and 1024 CPU cores respectively for both MISA-MD and LAMMPS,
and recording execution time of both programs, the results of computational performance are obtained.
The test cases on Tianhe-2 supercomputer use a $512 \times 512 \times 512$ box containing $2.68 \times 10^8$ atoms for varies of CPU cores.
While for test cases on Sunway-Taihulight, simulation box of $256 \times 256 \times 256$ containing $3.36 \times 10^7$ atoms is introduced.

The performance results for both MISA-MD code and LAMMPS code on \change{Intel} and sunway platform are shown in \cref{fig:md_perf_compare}.
\new{On Sunway platform, compared with LAMMPS package,
computational performance of MISA-MD is increased by 19.02\% to 132.56\% on Sunway-Taihulight.
On \change{Intel} platform of TianHe 2, the performance increment of MISA-MD is from 10.46\% to 22.67\% compared with LAMMPS packages.
}
Results indicate that, \textbf{
    MISA-MD has a better performance than LAMMPS for all performance test cases above.
 }

In LAMMPS, the neighbor list is updated every few timesteps and its cost can be quite expensive.
While in MISA-MD, hash index can be updated in each timestep,
but the time consuming of hash index updating is much less than neighbor list updating in LAMMPS
if the system does not change very frequently.
In cascade collision simulation of metal materials, position of most atoms does not change very frequently.
Thus, in hash index updating stage, most atoms' hash index does not need to be updated,
which can lead a better performance of MISA-MD than LAMMPS.
We should also point out that simulation of other types of crystal material can also be realistic and operable,
and can achieve a quite respectable performance.

\subsection{Scalability}
In this subsection, we will discuss our study on the scalability of our MISA-MD code on Sunway TaihuLight supercomputer.
We use cascade collision cases of pure $\alpha-$Fe system at temperature of 600 K as the case for the scalability tests.

For the strong scalability, a simulation box with 655 billion ($6.55\times 10^{11}$) atoms is presented
and take the performance of \SI{520000}{} cores (8000 core groups) as baseline.
\cref{fig:strong_scalability} shows the parallel efficiency of this case for strong scalability
when the number of cores(including CPE cores) varies from \SI{520000}{} to \SI{8320000}{}.
We can found that when the number of atoms per core group drops to 1/16 of the number,
the parallel efficiency drops from 100\% to 79.55\%, which is quite great for the parallel efficiency.

The weak scalability performance is shown in \cref{fig:weak_scalability}.
In the weak scalability tests, we initialize the simulation system with $8.39 \times 10^6$ atoms per core group (MPI process)
and take the performance of 1040 cores (16 core groups) as baseline.
It can be seen that there is almost no performance loss as the scale increase.
The parallel efficiency only goes down to 98.97\% when the number of cores is increased to 512 times of the baseline.

Results from both strong scalability and weak scalability indicate that
our MISA-MD code have the ability to expand MD simulation to larger scale with large scale parallelism.
Moreover, we also achieve $3.02 \times 10^{13}$ atoms cascade collision simulation at \SI{131072}{} core groups on Sunway TaihuLight supercomputer.
To our best knowledge, it holds the world record of the largest molecular dynamics simulation currently.

\begin{figure}
    \centering

    \begin{subfigure}[]{0.49\textwidth}
        \begin{adjustbox}{scale=0.75,center}
            \subimport{tikz/}{md-performance.tex}
        \end{adjustbox}
        \caption{Strong scalability for $6.55 \times 10^{11}$ atoms with $x$ axis logarithmic scaled.
        The number above the line is the parallel efficiency.}
        \label{fig:strong_scalability}
    \end{subfigure}

    \qquad
    \begin{subfigure}[]{0.49\textwidth}
        \begin{adjustbox}{scale=0.75,center}
            \subimport{tikz/}{md-performance-weak.tex}
        \end{adjustbox}
        \caption{Weak scalability for $8.39 \times 10^{6}$ atoms per core group (or MPI process) with $x$ axis logarithmic scaled.
        The number above the line is the parallel efficiency.}
        \label{fig:weak_scalability}
    \end{subfigure}
    \caption{Results of strong scalability and weak scalability performance tests.}
    \label{fig:md_performance}
\end{figure}

\section{Conclusion}
In this work, a new molecular dynamics code, MISA-MD,
is developed for simulating material evolution in cascade collision.
As a basic innovation,
a new hash based data structure is proposed for efficient atoms storage and quick neighbor indexing.
To fully utilize the performance of SW26010 processor,
the EAM potential interaction computation is also implementation on Sunway TaihuLight
with some efficient acceleration and optimization strategies,
including a coloring method for avoiding writing conflict on sunway CPEs, double-buffer DMA and data reuse strategies etc.
With the new data structure and accelerating of EAM interaction computation on SW26010 processor,
we can extend MD simulation to larger temporal scale and longer spatial scales.
It is worth noting that, many of our strategies are general and could also be implemented
in other multi-core processors and heterogeneous platforms.
For example, the coloring method for sunway CPEs can be easily migrated
to OpenMP multiple threads program model or CUDA platforms.
With these methods above,
performance of 79.5\% parallel efficiency is obtained in strong scaling simulation on Sunway TaihuLight system,
and simulation of more than thirty-trillion atoms is performed using \SI{131072}{} sunway core groups.
Moreover, experiments show that MISA-MD provides high accuracy of cascade collision simulation
and has less memory overhead and higher performance compared with classical molecular dynamics code LAMMPS.

Moreover, MISA-MD is designed to couple its simulation result with kinetic Monte Carlo (kMC) method\cite{sickafus_introduction_2007}
and Cluster Dynamics (CD) method\cite{terrier_cluster_2017} for further multisale evolution simulation
\cite{caturla_multiscale_2001,molnar_multiscale_2012,wirth_multiscale_2004,fu_multiscale_2004}
of cluster and defects in long temporal scale(seconds to years).

\section*{Acknowledgment}

This article is the results of the research project funded
by The National Natural Science Foundation of China under Grant U1867217.

\section{}

\bibliographystyle{elsarticle/elsarticle-num}

Distributed, Parallel, and Cluster Computing (cs.DC)


Automatic Airspace Sectorisation: A Survey

Abstract: Airspace sectorisation provides a partition of a given airspace into sectors, subject to geometric constraints and workload constraints, so that some cost metric is minimised. We survey the algorithmic aspects of methods for automatic airspace sectorisation, for an intended readership of experts on air traffic management.

\section{Introduction}

Airspace \emph{sectorisation} provides a partition of a given airspace
into a given (or upper-bounded, or minimal) number of control sectors,
subject to geometric constraints and workload constraints, so that
some cost metric is minimised.  We parametrise the concept in
Section~\ref{sect:dim}.

Airspace sectorisation is not to be mixed up with airspace
\emph{configuration}, which provides a schedule for the grouping and
splitting of elementary sectors into control sectors that are suitable
for a given number of available controllers and the expected traffic
structure.  This survey does not cover papers showing how to compute
such \emph{sector opening schemes}, such as the works
of~\cite{Barbosa:CASPT01,Verlhac:EEC05}, even though there is an
overlap between both problems.  Configuration is by definition a
(pre-)tactical action, whereas sectorisation is either strategic or
(pre-)tactical, depending on the inputs, so that a sectorisation model
can in principle be re-used within a configuration model.  However, a
configuration model also has time variables for scheduling purposes
and aims to minimise the total delay over a given time interval, but
these temporal aspects are absent from sectorisation.  Further, a
configuration model includes a transition cost incurred at every
switch between configurations, but static sectorisation does not
consider such a cost.  Hence a configuration model is much harder to
re-use properly for a sectorisation model, as projecting away time
variables and re-configuration costs most likely enables a better
model.

This paper is a survey of automatic airspace sectorisation methods.
To the best of our knowledge, there is only one prior survey, namely
\cite{Zelinski:DASC11}, which actually replaces two prior surveys by
the first author of that survey.  To distinguish our survey from that
survey, our survey has the following caveats:
\begin{itemize}
\item Our survey is intended for experts on air traffic control (ATC)
  and air traffic management (ATM) and is thus not self-contained:
  technical jargon (in use at EUROCONTROL\ for instance) as well as
  the rationales behind existing procedures and investigated research
  topics are assumed to be known by the reader.  For definitions of
  concepts such as elementary sector, control sector, air traffic
  control centre (ATCC) = area control centre (ACC), functional
  airspace block (FAB), ATC functional block (AFB), sector capacity,
  sector load, etc, see~\cite{ASTRA:KER:survey} for instance.
\item Our survey is written by experts on computing science: no
  evaluation of the realism of the combination of inputs, parameters,
  outputs, and experiments of automatic airspace sectorisation tools
  is made, as that is the realm of ATM experts.  See
  \cite{Zelinski:DASC11} for a comparison of six of the methods
  discussed in our survey.
\item Our survey is only about the algorithmic aspects of airspace
  sectorisation tools: for instance, we discuss no papers giving only
  rationales for the constraints and objective functions that can be
  used in sectorisation.
\end{itemize}
We do not claim that our survey is complete.  Indeed, the literature
is growing rapidly nowadays.  However, many papers are archived at
pay-sites that we (as academic computer scientists from a university
without a transportation research centre) have no free access to, so
it takes some effort to get hold of these papers.  Also, some
conferences seem to accept papers that are almost devoid of the
technical details that would in principle allow the reader to
reproduce the results, so it is hard to compare such papers to the
others.  Whenever an author (team) has published multiple papers, we
examine the most recent publication (that we could find).

The rest of this survey is organised as follows.  In
Section~\ref{sect:dim}, we introduce the classification criteria used
in the survey proper, which is in Section~\ref{sect:survey}.  In
Section~\ref{sect:concl}, we conclude.

\section{Airspace Sectorisation: Classification Criteria}
\label{sect:dim}

Our survey in Section~\ref{sect:survey} of the literature on airspace
sectorisation is structured around the following classification
criteria.

\paragraph{Approach.}
We distinguish between two approaches to airspace sectorisation.
\begin{itemize}
\item In a \emph{graph-based} model, a graph is constructed whose
  vertices represent the intersections of the existing trajectories,
  and whose edges thus represent segments of the existing
  trajectories.  The core problem of sectorisation is then essentially
  the NP-complete combinatorial problem of graph
  partitioning~\cite{GareyJohnson:NP}.  A graph partition does not
  define the sector boundaries, so actual sectors have to be
  constructed from the resulting vertex sets in a geometric
  post-processing step.
\item In a \emph{region-based} model, the airspace is initially
  partitioned into some kind of regions that are smaller than the
  targeted sectors, so that the combinatorial problem of partitioning
  these regions in principle needs no geometric post-processing step.
\end{itemize}

\paragraph{Frequency.}
Airspace sectorisation can be invoked with different frequencies:
\begin{itemize}
\item \emph{Static}: Sectorisation is strategic or pre-tactical.
\item \emph{Dynamic}: Sectorisation is tactical, but occurs at
  pre-determined times (so as to be different from configuration).
\end{itemize}

\paragraph{Input Granularity.}
Region-based models can start from regions of (any combination of)
different granularities:
\begin{itemize}
\item A mesh of blocks of the same size and shape.
\item ATC functional blocks (AFBs).
\item Elementary sectors, namely the ones of the existing
  sectorisation.  This leads to a further
  distinction~\cite{Zelinski:DASC11}:
  \begin{itemize}
  \item \emph{Base-line}: The output sectorisation should be
    reasonably close to the input one, usually because the
    sectorisation is dynamic and bears a transition cost.
  \item \emph{Free-form}: The output sectorisation can be arbitrarily
    far from the input one.
  \end{itemize}
\item Control sectors.
\item Area of specialisation (AOS).
\item Air traffic control centre (ATCC).
\end{itemize}

\paragraph{Output Granularity.}
An airspace sectorisation can be computed at different levels of
granularity:
\begin{itemize}
\item Elementary sectors.
\item Control sectors.
\item Functional airspace blocks (FABs).
\item Area of specialisation (AOS).
\item Air traffic control centre (ATCC).
\end{itemize}

\paragraph{Dimensionality.}
An airspace sectorisation can be computed in different (numbers of)
dimensions:
\begin{itemize}
\item 2D: The sectorisation is only defined (and tested) in two
  dimensions (longitude and latitude), with the generalisation to 3D
  being considered to be straightforward.
\item 2.5D: The sectorisation is only computed in two dimensions,
  because layers of the 3D airspace are considered to be independent.
\item 3rd D: The sectorisation preserves the 2D boundaries of the
  input regions but can readjust them in the third dimension
  (altitude).
\item 3D: The sectorisation is computed in three dimensions.
\end{itemize}

\paragraph{Constraints.}
Airspace sectorisation aims at satisfying some constraints.  The
following constraints have been found in the literature, so that a
subset thereof is chosen for a given tool:
\begin{itemize}
\item \emph{Balanced workload}: The workload of each sector must be
  within some given imbalance factor of the average across all
  sectors.
\item \emph{Bounded workload}: The workload of each sector must be
  below some upper bound.
\item \emph{Balanced size}: The size of each sector must be within
  some given imbalance factor of the average across all sectors.
\item \emph{Minimum dwell time}: Every flight entering a resulting
  sector must stay within it for a given minimum amount of time (say
  two minutes), so that the coordination work pays off and that
  conflict management is possible.
\item \emph{Minimum distance}: Each existing trajectory must be inside
  each resulting sector by a minimum distance (say ten nautical
  miles), so that conflict management is entirely local to sectors.
\item \emph{Convexity} of the resulting sectors.  Convexity can be in
  the usual \emph{geometric} sense, or \emph{trajectory-based} (no
  flight enters the same sector more than once), or more complex.
\item \emph{Connectedness}: A sector must be a contiguous portion of
  airspace and can thus not be fragmented into a union of unconnected
  portions of airspace.
\item \emph{Compactness}: A sector must have a geometric shape that is
  easy to keep in mind.
\item \emph{Non-jagged boundaries}: A sector must have a boundary that
  is not too jagged.
\end{itemize}
For each sector, there are three kinds of workload: the
\emph{monitoring workload}, the \emph{conflict workload}, and the
\emph{coordination workload}; the first two workloads occur inside the
sector, and the third one between the sector and an adjacent sector.
The quantitative definition of workload varies strongly between the
surveyed papers.  In the following, the word ``workload'' refers to
the sum of all three workload terms, but we will specify the actually
used workload terms whenever they can be inferred from a paper.

\paragraph{Constraint Types.}
Airspace sectorisation constraints can be of different types:
\begin{itemize}
\item A \emph{hard} constraint must be satisfied, whereas a
  \emph{soft} constraint can be violated, although its satisfaction
  earns a numeric bonus.
\item An \emph{explicit} constraint is part of the model, whereas an
  \emph{implicit} constraint is enforced by side effect, either
  because it is logically \emph{implied} by the explicit constraints
  or because its satisfaction is an \emph{invariant} of the solution
  process.
\end{itemize}

\paragraph{Cost Function.}
Airspace sectorisation aims at minimising some cost.  The following
costs have been found in the literature, so that a subset thereof is
combined into the cost function for a given tool, the subset being
empty if sectorisation is not seen as an optimisation problem:
\begin{itemize}
\item \emph{Coordination cost}: The cost of the total coordination
  workload between the resulting sectors must be minimised.
\item \emph{Transition cost}: The cost of switching from the old
  sectorisation to the new one must be minimised.
\item \emph{Workload imbalance}: The imbalance between the workload of
  the resulting sectors must be minimised.
\item \emph{Number of sectors}: The number of sectors must be
  minimised.
\item \emph{Entry points}: The total number of entry points into the
  resulting sectors must be minimised.
\item If any of the constraints above is soft, then there is the
  additional cost of minimising the number of violations of soft
  constraints.
\end{itemize}
A workload cost results from applying a function to the workload to
which it pertains; if this is the identity function, then we talk of
coordination workload rather than coordination cost.

\paragraph{Technology.}
An airspace sectorisation model can be implemented using (any
combination of) different algorithm design methodologies or
optimisation technologies:
\begin{itemize}
\item Stochastic local search (SLS), see \cite{Hoos:SLS} for instance.
\item Constraint programming (CP), see \cite{constraintshandbook} for
  instance.
\item Mathematical programming (MP), such as integer programming (IP)
  and mixed integer programming (MIP).
\item Global optimisation (GO).
\item Evolutionary algorithms (EA).
\item Computational geometry.
\item Ad hoc algorithm design.
\item etc.
\end{itemize}
Hybrid optimisation technologies are becoming increasingly powerful,
witness the hybridisation of CP with SLS \cite{Comet}, called
constraint-based local search (CBLS), and the hybridisation of CP with
MP and GO \cite{Hooker:integrated}.

\paragraph{Test Scale.}
An airspace sectorisation tool can be tested at different scales:
\begin{itemize}
\item \emph{Continental}: A (large fraction of an) entire continent is
  sectorised, such as the European Civil Aviation Conference (ECAC)
  area, or its core countries along the London-Frankfurt-Rome axis.
\item \emph{AOS}: An existing area of specialisation is (re-)sectorised.
\item \emph{ATCC}: An (existing) ATCC is (re-)sectorised.
\end{itemize}

\paragraph{Test Data.}
An automated airspace sectorisation tool can be tested on different
kinds of data:
\begin{itemize}
\item \emph{Historical data} stem from archives of actual flight data.
\item \emph{Extrapolated data} is computed from historical data
  according to some forecast of future flight patterns and volumes.
\item \emph{Artificial data} is generated randomly according to some
  (ideally realistic) model of flight patterns and volumes.
\item \emph{Simulated data} results from fast-time simulations of
  historical or extrapolated flight schedules.
\end{itemize}

\section{Survey}
\label{sect:survey}

In the following, we survey the (in our opinion) most important
papers, proceeding in chronological order, breaking ties by alphabetic
order on the surname of the first author.  Unless otherwise indicated,
each constraint is hard and explicit.  A word flagged with a question
mark (`?') means that we think it is correct but found no explicit
confirmation for it in the discussed paper.

\subsection{\cite{Delahaye:ICEC98}}

\begin{center}
\begin{tabular}{|l|l|}
  \hline
  Criterion & Value \\
  \hline\hline
  Approach & graph-based \\ \hline
  Frequency & (any) \\ \hline
  Input granularity & (not applicable) \\ \hline
  Output granularity & (any) \\ \hline
  Dimensionality & 3D \\ \hline
  Constraints & \parbox{11.5cm}{balanced workload, minimum
    dwell time, minimum distance, trajectory-based convexity,
    connectedness} \\ \hline
  Cost function & minimal coordination workload \\ \hline
  Technology & EA: genetic algorithm \\ \hline
  Test scale & ATCC \\ \hline
  Test data & artificial \\ \hline
\end{tabular}
\end{center}
This is one of the oldest and most cited lines of research on
sectorisation.  No quantitative definition is given of the workload of
a sector.  In the post-processing to the graph partitioning, 3D
Vorono\"i diagrams are used to define precisely the 3D borders of
sectors.  Problem instances of up to $400$ vertices that are to be
partitioned into up to $100$ sectors are solved in reasonable time.

\subsection{\cite{Yousefi:ATIO04}}

\begin{center}
\begin{tabular}{|l|l|}
  \hline
  Criterion & Value \\
  \hline\hline
  Approach & region-based \\ \hline
  Frequency & any \\ \hline
  Input granularity & hexagonal mesh, each side of a hexagon
    being $24$ NM \\ \hline
  Output granularity & elementary sectors \\ \hline
  Dimensionality & 2.5D \\ \hline
  Constraints & \parbox{11.5cm}{balanced workload, convexity (soft),
    reasonable average dwell time, connectedness, compactness (soft)} \\ \hline
  Cost function & minimal variation of workload among sectors \\ \hline
  Technology & MIP: facility location problem (number of sectors is
    not fixed) \\ \hline
  Test scale & continental: USA; initial experiments are with an
    ATCC \\ \hline
  Test data & extrapolated: TAAM simulation of one day of traffic \\ \hline
\end{tabular}
\end{center}
Workload is the sum of four components: the horizontal movement
workload, the conflict detection and resolution workload, the
coordination workload, and the altitude-change workload.  The
\emph{horizontal movement workload} is determined by the number of
aircraft in a sector and the average flight time.  The \emph{conflict
  detection and resolution workload} is based on conflict detection
using the type of conflict and the conflict severity.  The
\emph{coordination workload} is determined by the type of coordination
action including voice call, clearance issue, inter-facility transfer,
and tower transfer.  The \emph{altitude-change workload} is determined
by the type of sector altitude clearance requested.  No details are
given on how to compute the workload, but the reader is referred to
\cite{Yousefi:ATM03}, where more details are given, but the
definitions still depend on unspecified adjustment factors.  The tool,
TAAM, used to simulate the traffic produces workload estimates based
on a model of control workload.  The airspace is divided into three
layers: FL0 -- FL210, FL210 -- FL310, plus FL310 and above.
No details are given of the constraints and cost function, nor is
there any definition of what it means to minimise the variation of
workload among the sectors.

\subsection{\cite{TranDac:RAIRO05}}

\begin{center}
\begin{tabular}{|l|l|}
  \hline
  Criterion & Value \\
  \hline\hline
  Approach & graph-based \\ \hline
  Frequency & (any) \\ \hline
  Input granularity & (not applicable) \\ \hline
  Output granularity & (any) \\ \hline
  Dimensionality & 2D \\ \hline
  Constraints & \parbox{11.5cm}{balanced monitoring + conflict
    workload, minimum dwell time, minimum distance,
    trajectory-based convexity, connectedness} \\ \hline
  Cost function & minimal coordination workload \\ \hline
  Technology & hybrid of CP and SLS \\ \hline
  Test scale & ATCC \\ \hline
  Test data & random \\ \hline
\end{tabular}
\end{center}
Building on the model of \cite{Delahaye:ICEC98} (except that the
coordination workload is not considered within the balance
constraint), this work develops a CP model (rather than an EA one),
introducing propagators for new global constraints and introducing new
branching heuristics.  The monitoring and conflict workload of a
sector is the number of aircraft entering the sector.  The
coordination workload of a sector is the number of aircraft leaving
the sector for an adjacent one.  To solve large instances, the problem
is solved with a hybrid algorithm obtaining first a good solution with
an SLS heuristic and then locally improving the solution with an exact
CP formulation on small subsets of the sectors.  Geometrical sectors
are then built with triangulation techniques similar to the ones used
by \cite{Delahaye:ICEC98}.  Problem instances of up to $1000$ vertices
that are to be partitioned into up to $80$ sectors are solved in
reasonable time.

\subsection{\cite{Bichot:ATM07}}

\begin{center}
\begin{tabular}{|l|l|}
  \hline
  Criterion & Value \\
  \hline\hline
  Approach & region-based \\ \hline
  Frequency & static \\ \hline
  Input granularity & elementary sectors \\ \hline
  Output granularity & ATCCs or FABs \\ \hline
  Dimensionality & 3D \\ \hline
  Constraints & \parbox{11.5cm}{balanced monitoring + conflict
    workload (within a factor of $2$), balanced size (within a
    factor of $2$)} \\ \hline
  Cost function & minimal coordination workload \\ \hline
  Technology & SLS: fusion-fission metaheuristic \\ \hline
  Test scale & continental: $11$ core ECAC countries of Europe \\ \hline
  Test data & historical \\ \hline
\end{tabular}
\end{center}
This work\footnote{This paper is classified by \cite{Zelinski:DASC11}
  as work on configuration, but we disagree and include it in our
  survey.} tackles sectorisation at a larger granularity, namely the
design of ATCCs or FABs from the existing elementary sectors (along
mostly national boundaries), so as to enforce regional cooperation in
the seamless management and control of traffic flows in the ECAC zone.
The monitoring and conflict workload of a sector is the daily number
of aircraft entering the sector.  The coordination workload of a
sector is the daily number of aircraft leaving the sector for an
adjacent one.  A novel meta-heuristic, called \emph{fusion-fission}
and inspired by the corresponding phenomena in nuclear physics (but
described in detail only in a French-medium journal), is shown to be
particularly well-suited to solve this problem.  It seems to
outperform consistently very powerful graph partitioning libraries and
outputs FABs that much improve on the coordination workload and
imbalance of the current ATCC partition of ECAC.

\subsection{\cite{Conker:ATM07}}

\begin{center}
\begin{tabular}{|l|l|}
  \hline
  Criterion & Value \\
  \hline\hline
  Approach & region-based \\ \hline
  Frequency & static \\ \hline
  Input granularity & square mesh, dimension not defined \\ \hline
  Output granularity & elementary sectors \\ \hline
  Dimensionality & 2D(?) \\ \hline
  Constraints & equal complexity density across sectors (soft) \\ \hline
  Cost function & (none) \\ \hline
  Technology & ad hoc algorithm: $k$-means clustering, with an SLS heuristic \\ \hline
  Test scale & continental: USA, west of the Mississippi \\ \hline
  Test data & \parbox{11.5cm}{extrapolated: by simulation of traffic
    patterns based on predicted traffic demands} \\ \hline
\end{tabular}
\end{center}
A combination of two existing tools
\cite{bhadra2005future,wanke2004progressive} and specialised
algorithms is proposed.  The aim is to give a tool chain that assists
the semi-automated design of new sectors.  The workload is defined in
terms of a complexity density metric that is not defined in any
detail.  For the full definition of complexity density the reader is
referred to an unpublished MITRE Corporation memo.  The algorithmic
phase has three components: a modification of $k$-means clustering
\cite{lloyd:k-means:1982} to produce an initial sectorisation; a
hand-crafted SLS heuristic to improve the workload balance; and
straightening of the sector boundaries.  Many of the technical details
of the algorithms are missing.  Much of the effort is dedicated to
tool support to allow human experts to assess the given sectorisation.
Two tools are described: \emph{airspaceAnalyzer}, which simulates air
traffic controllers to assess the workload balance of the new
sectorisation; and \emph{sectorEvaluator}, which allows experts to
assess the quality of a given sectorisation and modify its sectors.

\subsection{\cite{Martinez:GNC07}}

\begin{center}
\begin{tabular}{|l|l|}
  \hline
  Criterion & Value \\
  \hline\hline
  Approach & combination of graph-based and region-based \\ \hline
  Frequency & static \\ \hline
  Input granularity & (not applicable) \\ \hline
  Output granularity & control sectors \\ \hline
  Dimensionality & 2D \\ \hline
  Constraints & bounded workload \\ \hline
  Cost function & minimal number of sectors \\ \hline
  Technology & ad hoc algorithm based on spectral clustering \\ \hline
  Test scale & continental: USA \\ \hline
  Test data & historical: ASDI (aircraft situation display to industry) data \\ \hline
\end{tabular}
\end{center}
Definitions of workload are not considered in any detail.  Instead
workload balancing is done in terms of the peak traffic count in a
sector based on an unspecified time interval.  The constraints and the
cost functions are defined by ad hoc algorithms.  The algorithm is in
three stages: first, flight data is used to construct a network of
flight flows that correspond to frequently flown routes; then spatial
cells are assigned to nodes of the flow network based on a nearest
neighbour, where each node in the flow is weighted by the number of
flights passing through that node; finally, the flow network is
partitioned into smaller and smaller sub-graphs until all sub-graphs
have a workload cost below a certain threshold.  Since the
partitioning problem is NP-hard, a heuristic based on spectral
techniques \cite{simon1991partitioning} from graph theory is used.

\subsection{\cite{Brinton:ATIO09}}

\begin{center}
\begin{tabular}{|l|l|}
  \hline
  Criterion & Value \\
  \hline\hline
  Approach & region-based \\ \hline
  Frequency & static \\ \hline
  Input granularity & square mesh, each side of a square being 1NM \\ \hline
  Output granularity & elementary sectors \\ \hline
  Dimensionality & 2D \\ \hline
  Constraints & non-jagged boundaries (soft), bounded workload \\ \hline
  Cost function & minimal number of sectors \\ \hline
  Technology & hybrid: computational geometry and SLS \\ \hline
  Test scale & continental USA \\ \hline
  Test data & not specified \\ \hline
\end{tabular}
\end{center}
The definition of workload is based on dynamic density, of which there
are multiple definitions in \cite{Kopardekar:DASC02}, but the one used
in this study is not specified.  The algorithm has three stages: an
initial clustering of flight tracks to produce an initial air-space
partition; a grid-based approach, where grid cells can have zero, one,
or many clusters, is used to grow cells containing clusters to
candidate sectors; and finally an SLS algorithm is used to simplify or
straighten the sector boundaries and to combine candidate sectors
while keeping the peak dynamic density below a certain limit.

\subsection{\cite{Drew:ATIO09}}

\begin{center}
\begin{tabular}{|l|l|}
  \hline
  Criterion & Value \\
  \hline\hline
  Approach & region-based \\ \hline
  Frequency & dynamic \\ \hline
  Input granularity & elementary sectors: free-form \\ \hline
  Output granularity & control sectors \\ \hline
  Dimensionality & 3D \\ \hline
  Constraints & bounded workload, connectedness \\ \hline
  Cost function & minimal number of sectors \\ \hline
  Technology & MIP \\ \hline
  Test scale & ATCC: Cleveland high sectors \\ \hline
  Test data & historical \\ \hline
\end{tabular}
\end{center}
The workload of a sector is the maximum number of aircraft
simultaneously present in the sector over a $15$-minute interval; it
has a sector-specific upper bound called the monitor alert parameter
(MAP).  The absence of a shape constraint, such as compactness, is
noted to lead to convoluted control sectors.  The presence of
symmetries in the solution space is noted, but no way is proposed to
exploit them.

\subsection{\cite{Leiden:ATIO09}}

\begin{center}
\begin{tabular}{|l|l|l|}
  \hline
  Criterion & \multicolumn{2}{l|}{Value} \\
  \hline\hline
  Approach & \multicolumn{2}{l|}{region-based} \\ \hline
  Frequency & \multicolumn{2}{l|}{dynamic: every $15$ minutes to $24$ hours} \\ \hline
  Input granularity & \multicolumn{2}{l|}{one AOS or ATCC: base-line} \\ \hline
  Output granularity & \multicolumn{2}{l|}{control sectors} \\ \hline
  Dimensionality & \multicolumn{2}{l|}{3rd D} \\ \hline
  Constraints (shared) & \multicolumn{2}{l|}{connectedness,
    lower-bounded height ($2$ FL)} \\ \hline
  Constraints (specific) & \parbox{5.1cm}{balanced workload \\
    (fixed number of sectors)}
  & \parbox{5.75cm}{(none other)} \\ \hline
  Cost function & \parbox{5.1cm}{(none)}
  & \parbox{5.75cm}{minimal number of sectors} \\ \hline
  Technology & \multicolumn{2}{l|}{ad hoc algorithm design: greedy algorithm} \\ \hline
  Test scale & \multicolumn{2}{l|}{AOS or ATCC: USA, above FL$240$} \\ \hline
  Test data & \multicolumn{2}{l|}{historical} \\ \hline
\end{tabular}
\end{center}
The workload of a sector is the number of aircraft simultaneously
present in the sector over an unspecified time interval; it has a
sector-independent upper bound called the monitor alert parameter
(MAP).  If workload exceeds the MAP, then a horizontal splitting of
the sector must be performed, but how to do this has not been
investigated yet.  The algorithm works in two modes, as indicated in
the table above.  Either way, the algorithm picks the best solution
from a greedy bottom-up phase and a greedy top-down phase through the
considered airspace.  The transition cost between two sectorisations
is the number of aircraft that must be transferred into a new sector;
this metric is used for evaluating computed sectorisations, but
currently not while computing them.  The experimental evaluation
confirms the expectation that capacity increases with the frequency of
re-sectorisation, but also concludes that the transition cost must be
built into the constraints or cost function.

\subsection{\cite{Li:ATIO09}}

\begin{center}
\begin{tabular}{|l|l|}
  \hline
  Criterion & Value \\
  \hline\hline
  Approach & graph-based \\ \hline
  Frequency & static \\ \hline
  Input granularity & (not applicable) \\ \hline
  Output granularity & control sectors \\ \hline
  Dimensionality & 3D \\ \hline
  Constraints & \parbox{11.5cm}{balanced workload, bounded
    workload, minimum distance, connectedness, compactness} \\ \hline
  Cost function & (none) \\ \hline
  Technology & \parbox{11.5cm}{ad hoc algorithm: spectral clustering,
    convex hull, shortest path (number of sectors is not fixed)} \\ \hline
  Test scale & ATCC: Atlanta \\ \hline
  Test data & historical \\ \hline
\end{tabular}
\end{center}
The workload of a sector is a weighted sum of its monitoring workload
(presumably including the conflict workload) and coordination
workload, though it is not made clear how these terms are quantified.
Special care is taken in the geometric post-processing step to obtain
sectors with smooth boundaries and good shapes.  Excellent results are
obtained, in the sense that the current sectorisation of Atlanta is
outperformed on almost all metrics in both the average case and worst
case.

\subsection{\cite{Bloem:ICAS10}}

\begin{center}
\begin{tabular}{|l|l|}
  \hline
  Criterion & Value \\
  \hline\hline
  Approach & region-based \\ \hline
  Frequency & dynamic \\ \hline
  Input granularity & elementary sectors: base-line \\ \hline
  Output granularity & control sectors \\ \hline
  Dimensionality & 3D \\ \hline
  Constraints & bounded workload (soft), connectedness,
    convexity (soft) \\ \hline
  Cost function & minimal workload cost + transition cost \\ \hline
  Technology & approximate dynamic programming \\ \hline
  Test scale & AOS and ATCC: Cleveland super-high sectors \\ \hline
  Test data & historical \\ \hline
\end{tabular}
\end{center}
The workload of a sector is the maximum number of aircraft in the
sector during the considered time step divided by a sector-specific
upper bound called the monitor alert parameter (MAP).  The transition
cost between two sectorisations is the number of new control sectors
compared to the previous time step.  The uncertainty of trajectory
prediction is explicitly taken into account.  Excellent results are
obtained, in the sense that the current sectorisation of Cleveland is
outperformed.

\subsection{\cite{Gianazza:AI10}}

\begin{center}
\begin{tabular}{|l|l|}
  \hline
  Criterion & Value \\
  \hline\hline
  Approach & region-based \\ \hline
  Frequency & any \\ \hline
  Input granularity & elementary sectors \\ \hline
  Output granularity & control sectors \\ \hline
  Dimensionality & (not applicable) \\ \hline
  Constraints & connectedness \\ \hline
  Cost function & \parbox{11.5cm}{minimal number of sectors +
    workload predictions obtained from the neural network} \\ \hline
  Technology & complete search via branch-and-bound \\ \hline
  Test scale & ATCC: five French ATCCs \\ \hline
  Test data & historical data \\ \hline
\end{tabular}
\end{center}
The workload of a sector is defined using a neural network trained on
historical traffic data and sectorisations.  The work considers
combinations of elementary airspace modules to build an optimal
airspace partition.  The work differs from other papers in this survey
in that it uses a neural network to predict workloads of
configurations and tree search together with branch-and-bound
techniques to explore intelligently all possible combinations of
elementary sectors in order to find the optimal combination.

\subsection{\cite{Sabhnani:ATIO10}}

\begin{center}
\begin{tabular}{|l|l|}
  \hline
  Criterion & Value \\
  \hline\hline
  Approach & region-based and graph-based \\ \hline
  Frequency & dynamic \\ \hline
  Input granularity & elementary sectors: free-form \\ \hline
  Output granularity & control sectors \\ \hline
  Dimensionality & 2D with extensions to 2.5D \\ \hline
  Constraints & \parbox{11.5cm}{balanced monitoring + conflict
    workload, compactness, minimum distance, convexity, and two other
    operational constraints (see text below)} \\ \hline
  Cost function & minimal coordination workload \\ \hline
  Technology & hybrid: computational geometry, graph theory, and MIP \\ \hline
  Test scale & ATCC \\ \hline
  Test data & simulated data \\ \hline
\end{tabular}
\end{center}
The focus is to extend previous work \cite{Mitchell:GNC08} to build
sectors that satisfy controllers (operational criteria).  It is argued
that while workload balance is an important goal, the validity of the
sector shapes is also important.  The major operational criteria that
are addressed are that: standard flows should cross sector boundaries
(almost) orthogonally; critical merge-and-conflict pairs should remain
sufficiently inside sector boundaries (minimum distance); no more than
three sectors should come together at the same point; and the shape of
the sector show be more or less convex.  The definition of workload is
as in \cite{Mitchell:GNC08}, where three possible metrics are
considered: \emph{peak workload}, which is the maximum number of
aircraft simultaneously in a sector; \emph{average workload}, which
the average number of aircraft present in a sector; and
\emph{coordination workload}, which is the number of instances during
an unspecified time interval where an aircraft crosses the boundary of
a sector.

\subsection{\cite{Xue:GNC10}}

\begin{center}
\begin{tabular}{|l|l|}
  \hline
  Criterion & Value \\
  \hline\hline
  Approach & region-based \\ \hline
  Frequency & static \\ \hline
  Input granularity & elementary sectors: free-form \\ \hline
  Output granularity & control sectors \\ \hline
  Dimensionality & 3D \\ \hline
  Constraints & (all in the cost function) \\ \hline
  Cost function & \parbox{11.5cm}{maximal average dwell time; maximal distance of
    intersection points from sector boundaries; maximal distance of
    dominant flows from sector boundaries; minimal number of
    flights with short dwell time; minimal variance of sector peak
    flight counts} \\ \hline
  Technology & hybrid: computational geometry and EA: genetic algorithm \\ \hline
  Test scale & ATCC \\ \hline
  Test data & simulated data \\ \hline
\end{tabular}
\end{center}
There is no explicit definition of workload.  The algorithms use a
mixture of computational geometry (Vorono\"i diagrams) and genetic
algorithms with a post-processing step using iterative deepening
search to improve the resulting sector designs.

\subsection{\cite{Jaegare:MSc11}}

\begin{center}
\begin{tabular}{|l|l|}
  \hline
  Criterion & Value \\
  \hline\hline
  Approach & region-based \\ \hline
  Frequency & (any) \\ \hline
  Input granularity & hexagonal or square mesh (arbitrary size) and AFBs \\ \hline
  Output granularity & elementary sectors \\ \hline
  Dimensionality & 3D \\ \hline
  Constraints & \parbox{11.5cm}{balanced monitoring + conflict
    workload, minimum dwell time, minimum distance,
    trajectory-based convexity} \\ \hline
  Cost function & minimal number of entry points \\ \hline
  Technology & CP \\ \hline
  Test scale & ATCC: Europe \\ \hline
  Test data & extrapolated, by ASTAAC \\ \hline
\end{tabular}
\end{center}
This work is close in spirit to \cite{TranDac:RAIRO05}, but takes a
region-based approach (rather than a graph-based one).  It also takes a
CP approach, but without hybridisation, and also introduces
propagators for new global constraints.
The monitoring and conflict workload of a sector is the number of
aircraft entering the sector.  The coordination workload of a sector
is the number of aircraft leaving the sector for an adjacent one.
The data for the experiments is provided by the \emph{Arithmetic
  Simulation Tool for ATFCM and Advanced Concept} (ASTAAC) tool of the
EUROCONTROL\ Experimental Centre.  This tool actually pre-clusters
some regions into AFBs according to additional constraints (sufficient
distance from potential conflicts and trajectories to sector
boundaries).
Experiments were run on up to a few hundred thousand small regions, to
be partitioned into five sectors.  In comparable run-times on the same
machine, the constraint program produces much better sectorisations
than \emph{NEVAC Sector Builder}, which is a greedy algorithm that
comes with ASTAAC (but is unpublished) and considers the
trajectory-based-convexity and minimum-dwell-time constraints to be
soft and yet does not systematically yield fewer sector entry points.
Since two hard constraints, connectedness and compactness, were not
implemented yet, the resulting sectors are sometimes disconnected or
of highly irregular shapes, so that current air traffic controllers
would be very uncomfortable in working with them.
The work is currently being revisited with us (who supervised this
thesis work), with the following targeted profile:
\begin{center}
\begin{tabular}{|l|l|}
  \hline
  Criterion & Value \\
  \hline\hline
  Approach & region-based \\ \hline
  Frequency & (any) \\ \hline
  Input granularity & hexagonal or square mesh (arbitrary size) and AFBs \\ \hline
  Output granularity & elementary sectors \\ \hline
  Dimensionality & 3D \\ \hline
  Constraints & \parbox{11.5cm}{balanced monitoring + conflict
    workload (soft), minimum dwell time (soft), trajectory-based
    convexity (soft), compactness (soft), connectedness} \\ \hline
  Cost function & minimal violation of soft constraints \\ \hline
  Technology & SLS: CBLS \\ \hline
  Test scale & continental: Europe \\ \hline
  Test data & extrapolated, by ASTAAC \\ \hline
\end{tabular}
\end{center}

\subsection{\cite{Kulkarni:ICNS11}}

\begin{center}
\begin{tabular}{|l|l|}
  \hline
  Criterion & Value \\
  \hline\hline
  Approach & region-based \\ \hline
  Frequency & static \\ \hline
  Input granularity & hexagonal mesh, each side of a hexagon being $24$ NM \\ \hline
  Output granularity & elementary sectors \\ \hline
  Dimensionality & 3D \\ \hline
  Constraints & balanced workload, compactness(?) \\ \hline
  Cost function & (none) \\ \hline
  Technology & ad hoc algorithms: knapsack, set covering, etc. \\ \hline
  Test scale & continental: USA, above FL240 \\ \hline
  Test data & extrapolated, by PNP \\ \hline
\end{tabular}
\end{center}
The paper completely lacks definitions and algorithmic details.  It is
claimed (partly in the title) that approximate dynamic programming as
well as control theory are involved, but it is not apparent where.

\section{Conclusion}
\label{sect:concl}

Sectorisation would benefit from further modelling in order to adapt
better to the design of FABs, taking into account more realistic
operational constraints, such as compatibility constraints between
upper and lower airspace, and those of \cite{Sabhnani:ATIO10} and
\cite{Xue:GNC10}.  The implementation of airspace optimisations that
alter the shape of basic control sectors implies a heavy cost in
controller training, as controllers are highly specialised in the
management of their specific sectors and it takes several years to
qualify them on new sectors.  Moreover, the redesign of ATCCs would
induce a new dispatch of radar data and probably the building of
costly new infrastructures to host them.  Contrary to lighter
optimisations that only concern current control structures without
modifying them, like the optimisation of opening schemes changes in
airspace design have very important transition costs and must be very
carefully planned.

From an algorithmic viewpoint, a clear conclusion of this survey is
that constraints should more often be used in the \emph{process} of
computing a sectorisation, rather than only in evaluating the
\emph{results} of a sectorisation algorithm.  We argue that mature
optimisation technologies, such as CP and MP, should be used more
often.  On the one hand, they offer high-level modelling facilities,
so that one can make \emph{explicit} the constraints and cost
function.  On the other hand, they offer off-the-shelf solving
algorithms that operate directly on such models.  In other words,
there is a clean separation of concerns between the declarative aspect
of modelling and the procedural aspect of solving, and highly tuned
algorithms can be re-used.  Such a plug-and-play paradigm allows a
much nimbler prototyping exploration of the algorithm design space,
which is crucial in an area such as sectorisation where suitable sets
of constraints and cost functions still have to be identified.

\bibliographystyle{apalike}