Machine Learning Testing in an ADAS Case Study Using Simulation-Integrated Bio-Inspired Search-Based Testing

Abstract: This paper presents an extended version of Deeper, a search-based simulation-integrated test solution that generates failure-revealing test scenarios for testing a deep neural network-based lane-keeping system. In the newly proposed version, we utilize a new set of bio-inspired search algorithms, genetic algorithm (GA), $({\mu}+{\lambda})$ and $({\mu},{\lambda})$ evolution strategies (ES), and particle swarm optimization (PSO), that leverage a quality population seed and domain-specific cross-over and mutation operations tailored for the presentation model used for modeling the test scenarios. In order to demonstrate the capabilities of the new test generators within Deeper, we carry out an empirical evaluation and comparison with regard to the results of five participating tools in the cyber-physical systems testing competition at SBST 2021. Our evaluation shows the newly proposed test generators in Deeper not only represent a considerable improvement on the previous version but also prove to be effective and efficient in provoking a considerable number of diverse failure-revealing test scenarios for testing an ML-driven lane-keeping system. They can trigger several failures while promoting test scenario diversity, under a limited test time budget, high target failure severity, and strict speed limit constraints.

\section{Introduction}
\label{D:sec:introduction}
Machine Learning (ML) nowadays is used in a wide range of application areas such as automotive \cite{borg2019safely, borg2021exploring}, health care \cite{topol2019high} and manufacturing \cite{lee2018industrial}. Many of the ML-driven systems in these domains present a high level of autonomy \cite{cummings2020regulating} and meanwhile are subject to rigorous safety requirements \cite{burton2020mind}. In 2018, the European Commission (EC) published a strategy for trustworthy Artificial Intelligence (AI) systems \cite{EuropeanCommission}. In this strategy, AI systems are defined as ``systems that display intelligent behavior by analyzing their environment and taking actions--with some degree of autonomy--to achieve specific goals''. The EC states that a trustworthy AI system must be lawful, ethical, and robust. Self-driving cars are examples of safety-critical AI systems, which leverage various ML techniques such as Deep Neural Networks (DNN), machine vision, and sensor data fusion. Meanwhile, in the context of automotive software engineering, there is always a set of strict safety requirements to meet.

The quality assurance methodology for AI systems \cite{hawkins2021AMLASguidance} is quite different from the conventional software systems, since the included ML components in those systems are not explicitly programmed, they are intended to learn from data and experience instead---called \textit{Software 2.0} \cite{Software2.0}. In addition, in AI systems, a part of the requirements is mainly seen as encoded implicitly in the data and the challenge of under-specificity is common in requirements definitions. However, it is still highly expected to assure the ability of the AI system to control the risk of hazardous events in particular in safety-critical domains. In this regard, Hawkins et al. \cite{hawkins2021AMLASguidance} introduced a methodology for assurance of ML in autonomous systems, called AMLAS. It presents a systematic process for integrating safety assurance into the development of ML systems and introduces verification procedures at different stages, e.g., learning model verification and system-level (integration) verification that happens after integrating the ML model into the system.

There is also a vigorous need for integration Verification and Validation (V\&V) of ML models deployed in self-driving cars to make sure that they are safe and dependable. Many of the failures basically emerge in the interplay between software containing ML components, hardware, and remote sensing devices, e.g., sensors, cameras, RADAR, and LiDAR technologies. Hardware-In-the-Loop (HIL), simulation-based and field testing are common approaches for system-level verification of deployed ML models \cite{hawkins2021AMLASguidance}. System-level testing mainly targets defining a set of operational scenarios that could lead to failures. In this regard, in the ISO/PAS 21448 Safety of the Intended Function (SOTIF) standard \cite{SOTIF}---which addresses complementary aspects of functional safety in ISO 26262 \cite{ISO26262}---simulation-based testing has been considered a proficient approach and a proper complementary solution to the on-road testing. Testing on real-world roads is costly, does not scale to cover all the needed scenarios, and in addition, it is dangerous to create and execute critical scenarios. The use of virtual prototyping allows testing and verification at the early stages of the development and offers the possibility of efficient and effective testing. It can capture the whole of the operational environment to a great extent using thoroughgoing physics-based simulators \cite{borg2020digital}. Recently, a growing number of commercial and open-source simulators have been developed to support the need for realistic simulation of self-driving cars \cite{BeamNG, dosovitskiy2017carla, rong2020lgsvl}---we refer interested readers to a review by Rosique et al.~\cite{rosique2019systematic}.

\textbf{\textit{Research Challenge.}} In this study, we target an Advanced Driver-Assistance System (ADAS) that provides lane-keeping assistance. Effective and efficient system-level testing in simulation environments requires sophisticated approaches to generate critical test scenarios. The critical test scenarios are those that break or are close to break the safety requirements of the ADAS under test, which hence result in \textit{safety violations}. Generating effective test scenarios involves sampling from a large and complex set of test inputs. Several authors have shown the potential of search-based software test generation techniques to address this challenge. Various system-level testing techniques using different search-based testing approaches \cite{abdessalem2018learnable, gambi2019automatically, abdessalem2018testing, Moghadam2021Deeper, ebadi2021efficient, GABezier} for different types of ADAS, relying on simulators, have been proposed in recent years.

\textbf{\textit{Research Contribution.}}
In this paper, we present a bio-inspired computation-driven test generator, called {Deeper}, for effective and efficient generation of failure-revealing test scenarios to test a Deep Neural Network (DNN)-based lane-keeping system in the BeamNG driving simulator. The test subject is BeamNG.AI, the built-in ML-driven driving agent in the BeamNG simulator. In this study, a failure is defined in terms of episodes in which the ego car---driven by the BeamNG.AI agent---drives partially outside the lane w.r.t a certain tolerance threshold. The tolerance threshold determines the percentage of the car’s bounding box needed to be outside the lane to be regarded as a failure.

{Deeper} in its current version in this paper benefits from the genetic algorithm (GA), $(\mu + \lambda)$ and $(\mu , \lambda)$ evolution strategies (ES), and the particle swarm optimization (PSO) to generate failure-revealing test scenarios, which are test roads in our study. The problem is basically regarded as an optimization problem, and in order to generate the test scenarios that are of interest, we evaluate the quality of the test scenarios using a \textit{fitness (objective) function} that guides the search process to maximize the detected distance of the car from the center of the lane during driving of the car on the lane. The initial version of {Deeper}\cite{Moghadam2021Deeper} contained a test generator based on NSGA-II. In this paper, we extend {Deeper} with four additional test generators based on GA, $(\mu + \lambda)$ and $(\mu , \lambda)$ ESs, and PSO. In the newly proposed test generators, we leverage an initial quality population seed to boost the search process, and also propose and develop domain-specific cross-over and mutation operations tailored for the presentation model used for modeling the test scenarios in the search algorithms. We rely on the presentation model \cite{riccio2020model} used by DeepJanus \cite{riccio2020model} based on Catmull-Rom cubic splines \cite{catmull1974class}.

\textit{Empirical evaluation.} In order to carry out an empirical evaluation, we use the setup provided by the cyber-physical systems (CPS) testing competition\footnote{Available at \url{https://github.com/se2p/tool-competition-av}.} at the IEEE/ACM 14\textsuperscript{th} International Workshop on Search-Based Software Testing (SBST). Our experiments are designed to answer three main research questions which are as follows:\\
\textbf{RQ1:} How capable are these test generators to trigger failures?\\
\textbf{RQ2:} How diverse are the generated failure-revealing test scenarios?\\
\textbf{RQ3:} How effectively and efficiently do the test generators perform? I.e., given a certain test budget, how many test scenarios are generated, what proportion of the scenarios is valid, and finally what proportion of the valid test scenarios leads to triggering failures? \\
We provide a comparative analysis on the performance of the proposed bio-inspired test generators in {Deeper} and five counterpart tools all integrated into the BeamNG simulator. In this regard, we compare the results of the proposed test generators in {Deeper} with five other test generator tools, Frenetic \cite{Frenetic}, GABExploit and GABExplore \cite{GABezier}, Swat \cite{SWAT}, and also the earlier version of the {Deeper} (based on NSGA-II) \cite{Moghadam2021Deeper}---all participating tools in the CPS competition at SBST 2021. In order to do a fair comparison, we consider the same experimental evaluation procedures as the original CPS tool competition.
Our experimental results show that first, the newly proposed test generators in {Deeper} present a considerable improvement on the previous version, second, they perform as effective and efficient test generators that can provoke a considerable number of diverse failure-revealing test scenarios w.r.t different target failure severity (i.e., in terms of tolerance threshold), available test budget, and driving style constraints (e.g, setting speed limits). For instance, in terms of the number of triggered failures within a given test time budget and with less strict driving constraints, the $(\mu + \lambda)$ ES-driven test generator in {Deeper} considerably outperforms other tools while keeping the level of promoted failure diversity quite close to the counterpart tool with the highest number of detected failures in the competition. Meanwhile, as a distinctive feature, none of the newly proposed test generators leaves the experiment without triggering any failures, and in particular, they act as more reliable test generators than most of the other tools for provoking diverse failures under a limited test budget and strict constraints. With respect to the test effectiveness and efficiency, {Deeper} $(\mu + \lambda)$ ES-, PSO-, and GA-driven result in high effectiveness in terms of the ratio of the number of detected failures to the generated valid test scenarios.

The rest of this paper is organized as follows: Section \ref{D:Sec::background} presents background information on random search techniques including evolutionary and swarm intelligence techniques. Section \ref{D:Sec::Deeper} presents the problem formulation and the technical details of our proposed test generators in {Deeper}. Section~\ref{D:sec::EmpiricalEvaluation} elaborates on the empirical evaluation, including the research method, experiments setup. Section \ref{D:Sec::Result&Discussion} discusses the results, answers to the RQs, and the threats to the validity of the results. Section \ref{D:Sec::RelatedWork} provides an overview of the related work, and finally, Section \ref{D:sec::summary} concludes the paper with our findings and the potential research directions for future work.

\section{Background} \label{D:Sec::background}
Evolutionary and swarm intelligence algorithms are two main classes of random search techniques, which are widely used in many different optimization problems. Genetic algorithms (GA), genetic programming (GP), differential evolution (DE), and evolution strategies (ES) are the main categories inside the family of evolutionary algorithms (EAs). Particle swarm optimization (PSO) is one of the primary representatives of swarm intelligence algorithms \cite{slowik2020evolutionary}.

\textbf{Genetic algorithms} is one of the most common nature-inspired optimization techniques. It starts with a random population of individuals---each called a chromosome---representing a potential solution for the problem. The objectives to be optimized in the problem are defined in an \textit{objective function} and the quality of the solutions is measured via this function. It shows how "well'' each solution satisfies the objective. The quality of each individual, which is also referred to as ``fitness'', is a main factor during the evolution process. At each generation, a new population is formed based on the selected individuals from the previous generation. Three operations are involved in forming the new generation, which are as follows:
\begin{enumerate}
    \item Selection, which mainly identifies highly-valued individuals from the previous generation.
    \item Crossover, which breeds ``child'' individuals by exchanging parts of the ``parent'' individuals. The child individuals (offspring) are formed by selecting genes from each parent individual.
    \item Mutation, which applies small random adjustments to the individuals.
\end{enumerate}

Crossover and mutation operations are applied w.r.t user-set probabilities, and these two operations might be used, either independently or jointly, to create new individuals to form a new population. The resulting individuals are added to the new population. The fitness values are calculated and stored for each individual in this population. This process iterates each generation until stopping criteria are met, e.g., a user-set number of generations or an allowed time budget is exhausted \cite{ebadi2021efficient, slowik2020evolutionary}.

\begin{comment}
\begin{figure}[h]
  \centering
  \includegraphics[width=.95\columnwidth, height = 3.7cm]{Figures/GA.png}
  \label{Algorithm:GA}
\end{figure}
\end{comment}

\begin{comment}
\begin{algorithm}
\SetAlgoLined
\begin{flushleft}
\caption{GA Algorithm}\label{Algorithm:GA_}
\begin{algorithmic}
\textrm{1. Initialize a population with random individuals}\\
\textrm{2. Evaluate the individuals in the population}\\
\Repeat{meeting the termination criteria}{
        \textrm{3. Select highly-valued individuals using selection operation}\\
        \textrm{4. Create offspring by applying crossover and mutation operation}\\
        \textrm{5. Evaluate the offspring based on the objective function}}
\textrm{6. Return the best individuals in the population}\\
\end{algorithmic}
\end{flushleft}
\end{algorithm}
\end{comment}

\textbf{Evolution strategy} is another common class of EAs. It is commonly used in almost all fields of optimization problems including discrete and continuous input spaces. ES also involves applying selection, recombination, and mutation to a population of individuals over various generations to get iteratively evolved solutions. Two canonical versions of ES are  $(\mu/\rho + \lambda)$ and $(\mu/\rho ,\lambda)$ evolution strategies. If $\rho=1$, we have ES cases without recombination, which are denoted by  $(\mu + \lambda)$ and $(\mu,\lambda)$ ESs. In the case of $\rho=1$, the recombination is simply making a copy of the parent. In these notations, $\lambda$ and $\mu$ indicate the size of the offspring and population respectively. The main difference between GA and ES is related to the selection step. In GA, at each iteration, the next generation is formed by selecting highly-valued individuals, while the size of the population is kept fixed \cite{slowik2020evolutionary}. In ES, a temporary population with the size of $\lambda$ is created and the individuals in this temporary population undergo crossover and mutation at user-set probabilities regardless of their fitness values. In $(\mu + \lambda)$ ES, then, both parents and the generated offspring resulting from the temporary population are copied to a selection pool---with size $(\mu + \lambda)$---and a new population with size $\mu$ is formed by selecting the best individuals. While, in $(\mu,\lambda)$ ES, the new generation with size $\mu$ is selected only from the offspring (with size $\lambda$). Therefore, a convergence condition as $\mu < \lambda$ is required to guarantee an optimal solution \cite{beyer2002evolution}.

\begin{comment}
\begin{algorithm}[h]
\SetAlgoLined
\begin{flushleft}
\caption{$(\mu + \lambda)$ ES}\label{Algorithm:mu+lambda_}
\begin{algorithmic}
\textrm{1. Initialize a population P with $\mu$ random individuals}\\
\textrm{2. Evaluate the individuals in the population P}\\
\Repeat{meeting the termination criteria}{
3. Create a temporary population $P_{T}$ with size $\lambda$ by reproduction of individuals from the population P\\
4. Create an offspring by applying crossover and/or mutation on the individuals in the population $P_{T}$\\
5. Evaluate the offspring based on the objective function
6. Select $\mu$ highly-valued individuals using selection operation from the original population P and the offspring
}
\textrm{7. Return the best individuals in the population}\\
\end{algorithmic}
\end{flushleft}
\end{algorithm}
\end{comment}

\textbf{Particle swarm optimization} is one of the most common representative of the swarm intelligence (SI) algorithms, which form a big class of nature-inspired optimization methods alongside the evolutionary algorithms. The SI algorithms present the concept of collective intelligence, which is mainly defined as a  collective behavior in a group of individuals that seem intelligent. SI algorithms have been inspired from collective behavior and self-organizing interactions between living agents in the nature, e.g., ant colonies and honey bees
\cite{mavrovouniotis2017survey}.

PSO is an optimization method simulating the collective behavior of certain types of living species. In PSO, \textit{cooperation} is an important feature of the system as each of the individuals changes its searching pattern based on its own and others' experiences. PSO starts with a swarm of random particles. Each particle has a position and velocity vectors, which are updated w.r.t the local and global best values. The best values get updated at each iteration. In the application of PSO, each particle (individual) represents a potential solution and is often modeled as a vector containing $n$ elements, in which each element represents a variable of the problem that is being optimized. Like GA, PSO searches for the optimal solution through updating solutions and creating subsequent generations, though without using evolution operators \cite{slowik2017nature}. The position (the elements) and velocity of each particle are updated as follows:
\begin{equation} \label{eq:position_update}
P^{t+1} = P^{t} + V^{t+1}
\end{equation}

\begin{equation} \label{eq:Velocity_update}
V^{t+1} = wV^{t} + c_1 r_1 (P^{t}_{best}- P^{t}) + c_2 r_2 (G^{t}_{best} -P^{t})
\end{equation}
where $P^{t}$ and $V^{t}$ are the position and velocity of the particle at iteration $t$, respectively; $P^{t}_{best}$ and $G^{t}_{best}$ are the local best position of the particle and the global best one up to the iteration $t$. The first part of (\ref{eq:Velocity_update}) is perceived as \textit{inertia}, which indicates the tendency of the particle to keep moving in the same direction, while the second part---which reflects a cognitive behavior---indicates the tendency towards the local best position discovered by the particle and the last part---which is the social knowledge---reflects the tendency to follow the best position found so far by other particles.

Therefore, at each iteration, the position of each particle is updated based on its velocity, and the velocity is controlled by the inertia and accelerated stochastically towards the local and global best values. $r_1$ and $r_2$ are random weights from range $(0, 2]$ which adjust the cognitive and social acceleration. In (\ref{eq:Velocity_update}), $w$ is inertia weight, which adjusts the ability of the swarm to change the direction and makes a balance between the level of exploration and exploitation in the search process. A lower $w$ leads to more exploitation of the best solutions found, while a higher value of $w$ facilitates more exploration around the found solutions. $c_1$ and $c_2$ are the acceleration hyperparameters defining to what extent the solutions are influenced by the local best solutions and global best solution. These hyperparameters and the inertia weight could be static or changed dynamically over the iterations. For instance, $w = 0.72984$ and $c1+c2>4$ is a common setup for a static configuration of these parameters \cite{clerc2002particle}.

\begin{comment}
\begin{algorithm}[h]
\SetAlgoLined
\begin{flushleft}
\caption{PSO Algorithm}\label{Algorithm:PSO_}
\begin{algorithmic}
\textrm{1. Initialize a swarm with random particles}\\
\textrm{2. Evaluate the particles of the swarm based on the fitness function }\\
\textrm{3. Select the global best particle w.r.t the fitness value $(G_{best})$}\\
\Repeat{meeting the termination criteria}{
        \For{each particle P in the swarm}{
        {4. Calculate particle's velocity according to eq. (2)}\\
        {5. Update particle's position according to eq. (1)}\\
        {6. Evaluate particle based on the fitness function}
        \IF{fitness value of P is better than the local best of P, ($P_{best}$)}
        \STATE {7. Update $P_{best}$ with $P$}
        \ENDIF
        \IF{fitness value of P is better than the global best, ($G_{best}$)}
        \STATE {8. Update $G_{best}$ with $P$}
        \ENDIF
        }
        }
\textrm{9. Return the best particles in the swarm}
\end{algorithmic}
\end{flushleft}
\end{algorithm}
\end{comment}

\section{Deeper: A Bio-Inspired Simulation-Integrated Testing Framework}\label{D:Sec::Deeper}
This section presents the technical details of {Deeper} and shows how it challenges a DNN-based lane-keeping system in a self-driving car trained and tested in the BeamNG simulator environment \cite{BeamNG}. The subject system is a built-in AI driving agent encompassing a steering angle predictor (ML model) which receives images captured by an onboard camera in the simulation environment. Then, the test inputs (test cases) generated by {Deeper} are defined as scenarios in which the car drives. Our target is to generate diverse test scenarios triggering the misbehavior of the subject system. In this regard, we benefit from bio-inspired search-based techniques to explore the input space and generate highly-valued failure-revealing test scenarios.

\subsection{Test Scenario and failure specification}\label{D:Sec::Test_Scenario_Spec}
Test scenarios are defined as combinations of roads, the environment including e.g., the weather and illumination, and the driving path, i.e., starting and end points and the lane to keep. Hereafter, we consider scenarios involving a single asphalt road surrounded by green grass where the car is to drive on the right lane, and the environment is set to a clear day with the overhead sun (See Fig. \ref{fig:TestScenarioSample}). Therefore, the focus of {Deeper} is to generate diverse roads which trigger failures in the system under test. In this system, failure is defined in terms of an episode, in which the car drives partially outside the lane meaning that $X\%$ of the car’s bounding box gets outside the lane. $X$ is a configurable tolerance threshold for {Deeper}.

\begin{figure}
  \centering
  \includegraphics[width=.60\columnwidth, height = 4.5cm]{TestCaseSample.png}
  \caption{A test scenario executed in the BeamNG simulator.}
  \label{fig:TestScenarioSample}
\end{figure}

In the test scenarios, the road composes two fixed-width lanes with a yellow center line and two white lines separating the lanes from the non-drivable area. In BeamNG, each road is mainly described by a set of points that are used by the simulation engine to render the road. The simulation engine accomplishes the rendering by interpolating the points and creating a sequence of polygons on the points---provided by the SBST competition setup \footnote{\url{https://github.com/se2p/tool-competition-av.git}.}. Notably, not every sequence of road points results in valid roads, so each sequence of points is also validated against some initial geometrical constraints related to the road polygons and some other domain-specific constraints. The main constraints are: 1) the start and end points of the road shall be different, 2) the road shall be completely contained in the map used in the simulation, 3) the road shall not self-intersect, and 4) the road shall not contain too sharp turns that force the vehicle to invade the opposite lane. To assure the satisfaction of these constraints, {Deeper} validates the generated roads before getting executed and consequently, the invalid roads are not counted as failed test scenarios.

\textit{Road Representation Model:} In order to convert the abstract road model into a proper set of points that can be rendered by the simulation engine---as candidate solutions in the test generator---we rely on the representation model used by {DeepJanus} \cite{riccio2020model} based on Catmull-Rom cubic splines \cite{catmull1974class}. Therefore, each road is represented by two sets of points, \textit{control points} and \textit{sample points}. First, control points are provided as an input specification for a candidate road. Second, sample points are calculated using the Catmull-Rom calculation algorithm. Third, the simulation engine uses the sample points, if they are valid, to render the road. Fig. \ref{fig:ModeloftheRoad} shows the representation model of a road in terms of control and sample points (\ref{fig:CP} and \ref{fig:SP}) and the corresponding rendered road in the simulation (\ref{fig:RenderedRoad}).

\begin{eqnarray}
CP= \langle C_1, C_2, \cdots, C_m \rangle  \ , \  &{R_i}_{min} \leq C_i \leq {R_i}_{max}\\
&{R_i}_{min}, \ {R_i}_{max} \in R
\end{eqnarray}

\begin{eqnarray}
SP= \langle S_1, S_2, \cdots, S_n \rangle  \ , \  &SP = Catmull\_Rom\_Spline (CP)\\
&{R_i}_{min} \leq S_i \leq {R_i}_{max}\\
&{R_i}_{min}, \ {R_i}_{max} \in R
\end{eqnarray}

\begin{figure}[h]
  \centering
  \subfloat[Control Points.]{
       \setlength{\fboxrule}{0.0pt} \framebox{\includegraphics[width=0.4\textwidth, height =4.8cm]{CP.png}} \label{fig:CP}}
   \subfloat[Calculated Sample Points.]{
        \setlength{\fboxrule}{0.0pt}\framebox{\includegraphics[width=0.4\textwidth,  height =4.8cm]{SP.png}} \label{fig:SP} }\\
    \subfloat[The corresponding road in the simulation.]{
        \setlength{\fboxrule}{0.0pt}\framebox{\includegraphics[width=0.35\textwidth,  height =4.5cm]{RenderedRoad.png}\vspace{0.5cm}} \label{fig:RenderedRoad}}
  \caption{The representation model of a road}
  \label{fig:ModeloftheRoad}
\end{figure}

\begin{comment}
\begin{figure}
  \centering
 \begin{subfigure}[b]{0.32\textwidth}
         \centering
         \includegraphics[width=\textwidth, height =5cm]{paperD/CP.png}
         \caption{Control Points}
         \label{fig:CP}
     \end{subfigure}
     \begin{subfigure}[b]{0.32\textwidth}
         \centering
         \includegraphics[width=\textwidth,  height =5cm]{paperD/SP.png}
         \caption{Calculated Sample Points }
         \label{fig:SP}
     \end{subfigure}
     \begin{subfigure}[b]{0.3\textwidth}
         \centering
         \includegraphics[width=\textwidth,  height =4cm]{paperD/RenderedRoad.png}
         \vspace{0.0cm}
         \caption{The corresponding road in the simulation}
         \label{fig:RenderedRoad}
     \end{subfigure}
        \caption{The representation model of a road}
        \label{fig:ModeloftheRoad}
\end{figure}
\end{comment}

\subsection{Fitness Function}\label{D:Sec::Fitness_Function}
As indicated above, the focus is the generation of diverse failure-revealing test scenarios w.r.t the intended tolerance threshold. The competition setup elaborates the positional data and detects the episodes in which the car breaks out of the lane bounds, i.e., out of bound episodes (OBE). It computes the distance of the car from the center of the lane in those OBE episodes and reports an OBE failure \cite{gambi2019asfault} each time that the car drives outside the lane if the percentage of the area of the car that is outside the lane is bigger than the intended threshold (referred to as $X$ in Section~\ref{D:Sec::Test_Scenario_Spec}).

The problem is regarded as an optimization problem, and in order to generate valuable test scenarios leading us to meet the target, we evaluate the quality of the test scenarios using a \textit{fitness (objective) function}. In this regard, for each test scenario, the main objective of interest to be optimized is the maximum detected distance of the car from the center of the lane during driving of the car on the lane. So, more accurately the fitness function that we want to minimize is as follows:

\begin{equation}
   Fitness\_Function = (lane\_width)/2 - d(center\_polyline, car\_position)
\end{equation}
where $lane\_width$ is the width of the lane and $d(spin\_polyline, car\_position)$ indicates the distance of the car position from the central polyline (center) of the lane.

\subsection{Bio-Inspired Search Algorithms}\label{D:Sec::Search_Algorithms}
We are interested in sampling from the space of possible test scenarios in an effective way to generate those that lead to the emergence of failures. This can be achieved by using an optimization algorithm to guide the search by the fitness function. Therefore, in order to find the solutions of interest, we use bio-inspired search-based algorithms, i.e., GA, $(\mu + \lambda)$ and $(\mu , \lambda)$ ESs, and PSO, guided by the fitness function introduced in Sec.\ref{D:Sec::Fitness_Function}. These search algorithms are mainly modeled on the basis of population evolution over time and they usually get started with the creation of a random population of solutions. In {Deeper}, we leverage an initial quality population seed to boost the search process regarding the fact that the search is done at a fixed test budget. Throughout the development, the impact of different initial population seeds was investigated. For instance, starting from an initial random population seed was not quite effective to lead the search to find the failure-revealing solutions w.r.t high tolerance thresholds within a reasonable test budget. The quality population seed used in the current search algorithms of {Deeper} is a mix of some valid random solutions and some extracted randomly from the solutions generated by a 5-hour execution of the first version of {Deeper} (based on NSGA-II \cite{deb2002fast}) that start from an initial \textit{random} population seed. Then, the generated solutions which could cause OBEs w.r.t a tolerance threshold $\tau \geq 0.5$ are considered to be used to form the quality population seed.

\subsubsection{{Genetic Algorithm}} \label{D:Sec::Search_Algorithms_GA} The GA-driven test generator in {Deeper} starts with forming an initial population by sampling from the quality population seed. Over various generations, new populations of test scenarios are formed through applying crossover and mutation operations to the best ones selected from the previous generations.

\textit{Selection:} We use \textit{tournament selection} for identifying the promising test scenarios, which have a high probability to lead to failures and safety violations. In tournament selection, a subset of the population is selected at random in each tournament and the best test scenario of the subset is picked. The number of individuals participating in each tournament indicates the size of the tournament.

\textit{Crossover:} We develop a domain-specific \textit{one-point crossover} operation to create new test scenarios, i.e., new test roads, from the ones selected from the previous generation. The proposed crossover operation performs the segment exchange at the sets of \textit{control points}, which means that a random point is selected as the crossover point in the sets of control points in the parent roads, then the parts of the sets beyond the crossover point are swapped between the parents, and accordingly, two new sets of control points for two child roads are formed. The corresponding sample points for the generated child roads are calculated using the Catmull-Rom calculation algorithm (See Fig. \ref{fig:Crossover}).

\begin{figure}[h]
  \centering
    \subfloat[Parent Road 1: Control Points.]{
       \setlength{\fboxrule}{0.0pt}\framebox{\includegraphics[width=0.3\textwidth, height =3.5cm]{Road_1_.png}} \label{fig:Road1}}
   \subfloat[Child Road 1: Control Points.]{
       \setlength{\fboxrule}{0.0pt} \framebox{\includegraphics[width=0.3\textwidth,  height =3.5cm]{Road_3_.png}} \label{fig:Road3_Control_points}}
   \subfloat[Child Road 1: Sample Points.]{
       \setlength{\fboxrule}{0.0pt}\framebox{\includegraphics[width=0.3\textwidth,  height =3.5cm]{Road_3_sample_points.png}} \label{fig:Road3_Sample_points}}
   \newline
    \subfloat[Parent Road 2: Control Points.]{
       \setlength{\fboxrule}{0.0pt} \framebox{\includegraphics[width=0.3\textwidth, height =3.5cm]{Road_2_.png}} \label{fig:Road2}}
   \subfloat[Child Road 2: Control Points.]{
       \setlength{\fboxrule}{0.0pt} \framebox{\includegraphics[width=0.3\textwidth,  height =3.5cm]{Road_4_.png}} \label{fig:Road4_Control_points}}
   \subfloat[Child Road 2: Sample Points.]{
       \setlength{\fboxrule}{0.0pt} \framebox{\includegraphics[width=0.3\textwidth,  height =3.5cm]{Road_4_sample_points.png}} \label{fig:Road4_Sample_points}}
\caption{Crossover operation on two sample roads}
      \label{fig:Crossover}
\end{figure}

\begin{comment}
\begin{figure}[h]
  \centering
   \begin{subfigure}[b]{0.3\textwidth}
         \centering
         \includegraphics[width=\textwidth, height =5cm]{paperD/Road_1_.png}
         \caption{Parent Road 1: Control Points}
         \label{fig:Road1}
     \end{subfigure}

     \begin{subfigure}[b]{0.3\textwidth}
         \centering
         \includegraphics[width=\textwidth,  height =5cm]{paperD/Road_3_.png}
         \caption{Child Road 1: Control Points}
         \label{fig:Road3_Control_points}
     \end{subfigure}

     \begin{subfigure}[b]{0.3\textwidth}
         \centering
         \includegraphics[width=\textwidth,  height =5cm]{paperD/Road_3_sample_points.png}
         \caption{Child Road 1: Sample Points}
         \label{fig:Road3_Sample_points}
     \end{subfigure}

     \newline

     \begin{subfigure}[b]{0.3\textwidth}
         \centering
         \includegraphics[width=\textwidth, height =5cm]{paperD/Road_2_.png}
         \caption{Parent Road 2: Control Points}
         \label{fig:Road2}
     \end{subfigure}

     \begin{subfigure}[b]{0.3\textwidth}
         \centering
         \includegraphics[width=\textwidth,  height =5cm]{paperD/Road_4_.png}
         \caption{Child Road 2: Control Points}
         \label{fig:Road4_Control_points}
     \end{subfigure}

     \begin{subfigure}[b]{0.3\textwidth}
         \centering
         \includegraphics[width=\textwidth,  height =5cm]{paperD/Road_4_sample_points.png}
         \caption{Child Road 2: Sample Points}
         \label{fig:Road4_Sample_points}
     \end{subfigure}
     \caption{Crossover operation on two sample roads}
      \label{fig:Crossover}
\end{figure}
\end{comment}
However, still, these resulting sets of points might not represent valid roads w.r.t the geometrical constraints, so before adding these new test scenarios to the offspring, we also check their validity and let only the resulting valid roads be added to the offspring. If both generated child roads are valid, then both of them will be transferred to the offspring, while if one of them is valid, we keep the valid child and another crossover point is tried to breed the parent roads and generate the second valid child. Likewise, if none of the child roads are identified as valid roads, another crossover point is tried. All in all, in order to generate two valid child roads from a crossover operation, at most five attempts to try with different crossover points are done. If in the end, the attempts do not lead to valid child roads, the whole process will be rolled back and the original parents will be added to the offspring.

\textit{Mutation:} The mutation operation also targets the coordinate values of the control points. We use \textit{Polynomial Bounded mutation}, a bounded mutation operation for real-valued individuals which was used in NSGA-II \cite{deb2002fast}. It features using a polynomial function for the probability distribution and a user-set parameter, $\eta$, presenting the \textit{crowding degree} of the mutation and adjusting the diversity in the resulting mutant. A high value for $\eta$ results in a mutant resembling the original solution, while a low $\eta$ leads to a more divergent mutant from the original.
This domain-specific polynomial bounded mutation operation selects randomly a point---mutation point---in the set of control points and mutates randomly the $x$ or $y$ coordinate of the selected control point (See Fig. \ref{fig:Mutation}). Accordingly, the sample points are re-calculated for the mutated set of control points, and their validity w.r.t the geometrical constraints are checked. In case the mutant does not represent a valid road, another control point is tried. The GA-driven test generator of {Deeper} is configured as presented in Algorithm \ref{Algorithm:GA}.

\begin{figure}[h]
  \centering
    \subfloat[Original road.]{
       \setlength{\fboxrule}{0.0pt} \framebox{\includegraphics[width=0.4\textwidth, height =4.5cm]{Mutation_Original.png}} \label{fig:Original_Road}}
    \subfloat[Mutated road.]{
       \setlength{\fboxrule}{0.0pt} \framebox{\includegraphics[width=0.4\textwidth,  height =4.5cm]{Mutated.png}} \label{fig:Mutated_Road}}
    \caption{Mutation operation on the road}
        \label{fig:Mutation}
\end{figure}

\begin{comment}
\begin{figure}[h]
  \centering
 \begin{subfigure}[b]{0.35\textwidth}
         \centering
         \includegraphics[width=\textwidth, height =5cm]{paperD/Mutation_Original.png}
         \caption{Original road}
         \label{fig:Original_Road}
     \end{subfigure}
     \begin{subfigure}[b]{0.35\textwidth}
         \centering
         \includegraphics[width=\textwidth,  height =5cm]{paperD/Mutated.png}
         \caption{Mutated road }
         \label{fig:Mutated_Road}
     \end{subfigure}
        \caption{Mutation operation on the road}
        \label{fig:Mutation}
\end{figure}
\end{comment}

\begin{algorithm}[h]
\begin{flushleft}
\caption{GA-driven test generator in Deeper}\label{Algorithm:GA}
\begin{algorithmic}
1. Initialize a population of test scenarios (with $size =70$) from the quality population seed\\
\noindent 2. Evaluate the test scenarios through rendering them in the simulation and computing the fitness values\\
\Repeat {reaching the end of the test budget (e.g., given time)}{
{3. Select highly-valued scenarios using tournament selection ($Tournament\ size =3$)}\\
{4. Create offspring by using crossover and mutation operations}\\
{\quad{4.1. Apply the domain-specific crossover operation ($Crossover\ rate = 0.3$)}}\\
{\quad{4.2. Apply the domain-specific polynomial Bounded mutation operation $(Mutation\ rate =0.7)$}}\\
{5. Evaluate the offspring} }
6. Collect the test scenarios revealing OBE failures\\
\end{algorithmic}
\end{flushleft}
\end{algorithm}

\subsubsection{{Evolution Strategies}} The ES-driven test generators in Deeper use two canonical $(\mu + \lambda)$ and $(\mu, \lambda)$ ES algorithms. The ES-driven test generators start with initializing a population with $\mu$ test scenarios sampled from the population seed. The fitness value of each test scenario is calculated through rendering them in the simulation. Next, a temporary population with size $\lambda$ is formed by the reproduction of test scenarios from the original population. This temporary population is used to create offspring by applying the proposed domain-specific crossover or mutation operations (See Section \ref{D:Sec::Search_Algorithms_GA} ) to its individuals according to user-set crossover and mutation probabilities. In $(\mu + \lambda)$ ES, then both parents and the resulting offspring are copied to a selection pool, and a new population with size $\mu$ is created by using tournament selection. While, in $(\mu, \lambda)$ ES-driven test generator, the new population is selected only from the offspring. Algorithm \ref{Algorithm:mu+lambda} presents the procedure of $(\mu + \lambda)$ ES-driven test generator, where $\mu=70$ and $\lambda =30$. The procedure of $(\mu,\lambda)$ ES test generator is almost the same as $(\mu + \lambda)$ ES, and the main difference is in the selection step, where the $\mu$ highly-valued individuals are selected only from the offspring in $(\mu,\lambda)$ ES. However, with regard to the required convergence condition $(\mu < \lambda)$ in this case, then those parameters in $(\mu,\lambda)$ ES test generator are configured as $\mu=70$ and $\lambda =100$ (the size of the main population is fixed at both ES-driven test generators).

\begin{algorithm}[h]
\begin{flushleft}
\caption{$(\mu + \lambda)$ ES-driven test generator in Deeper }\label{Algorithm:mu+lambda}
\begin{algorithmic}
 {1. Initialize a population P with $\mu$ test scenarios sampled from the quality population seed}\\
 {2. Evaluate the test scenarios in the population P through simulation and computing the fitness values}\\
\Repeat {reaching the end of the test budget (e.g., given time)}{
{3. Create a temporary population $P_{T}$ with size $\lambda$ by reproduction of test scenarios from the population P}\\
{4. Create an offspring by applying the crossover or mutation operation to the test scenarios in the population $P_{T}$ (with crossover probability $Cx_P = 0.3$ and mutation probability $Mu_P = 0.7$)}\\
 {\quad 4.1. $choice = Random.random(),\ 0 < choice < 1$}\\
{ \quad 4.2. If $choice < Cx_P$ then}\\
{ \qquad Apply the domain-specific crossover operation}\\
{ \quad else if $choice < Cx_P + Mu_P$}\\
{\qquad Apply the domain-specific polynomial Bounded mutation operation}\\
{5. Evaluate the offspring}\\
{6. Select $\mu$ highly-valued test scenarios using tournament selection ($Tournament\ size =3$) from the original population P and the offspring}}

7. Collect the test scenarios revealing OBE failures
\end{algorithmic}
\end{flushleft}
\end{algorithm}

\subsubsection{{Particle Swarm Optimization}} The PSO-driven test generator in {Deeper} starts with initializing a population of particles from the quality population seed. Each test scenario is modeled as a particle, and the set of control points represents the position vector of the particle that is updated according to Eq. (\ref{eq:position_update}) and (\ref{eq:Velocity_update}) over various generations. After each updating, the corresponding sample points for the updated set of control points are calculated and the validity of the new set of road points is checked against the geometrical constraints. The PSO-driven test generator is configured as presented in Algorithm \ref{Algorithm:PSO} and based on the following setting for the parameters, which is $w=0.8$, $c1=2.0$, and $c2=2.0$.

\section{Empirical Evaluation} \label{D:sec::EmpiricalEvaluation}
We conduct an empirical evaluation of the proposed simulation-integrated bio-inspired test generators in {Deeper}, by running experiments on an experimental setup based on a PC with 64-bit Windows 10 Pro, Intel Core i7-8550U CPU @ 1.80GHz, 16GB RAM, Intel UHD Graphics 620, and BeamNG.tech driving simulator together with the software requirements for running {Deeper}\footnote{See requirements at \url{https://github.com/mahshidhelali/Deeper_ADAS_Test_Generator.git}}.

\begin{algorithm}[h]
\begin{flushleft}
\caption{PSO-driven test generator in {Deeper}}\label{Algorithm:PSO}
\begin{algorithmic}
{1. Initialize a swarm of test scenario particles (with $size=70$) from the quality population seed}\\
{2. Evaluate the particles of the swarm through simulation and computing the fitness values}\\
{3. Select the global best particle w.r.t the fitness value $(G_{best})$}\\
\Repeat {reaching the end of the test budget (e.g., given time)}{
        \For {each test scenario particle P in the swarm} {
        {4. Calculate the particle's velocity according to eq. (2)}\\
        {5. Update particle's position according to eq. (1)}\\
        {6. Evaluate the particle based on the fitness function}\\
        \If {fitness value of P is better than the local best of P, ($P_{best}$),}{
        {7. Update $P_{best}$ with $P$}
        }
        \If {fitness value of P is better than the global best, ($G_{best}$),}{
        {8. Update $G_{best}$ with $P$}
        } }

}
9. Collect the test scenario particles revealing OBE failures
\end{algorithmic}
\end{flushleft}
\end{algorithm}

\textit{Test Subject:} The system under test is BeamNG's
built-in driving agent, BeamNG.AI. It is an autonomous agent utilizing optimization techniques to plan the driving trajectory according to the speed limit while keeping the ego-car inside the road lane. It is equipped with a DL-based lane-keeping ADAS. The DL-based lane-keeping system learns a mapping from the input of the onboard camera in the simulated environment to the steering angle. It is based on the {DAVE-2} architecture including a normalization layer, five convolutional layers followed by three fully connected layers \cite{bojarski2016end}. This test subject has been used in previous research and also in the SBST 2021 cyber-physical tool competition for evaluating test scenario generators \cite{gambi2019asfault, riccio2020model, panichella2021sbst}, and moreover does not require manual training, which can mitigate the threats to the validity of the results \cite{panichella2021sbst}.

\subsection{Research Method} \label{D:sec::Research Method}
We design and implement a set of experiments to answer the research questions:
\begin{enumerate}
    \item RQ1: How capable are these test generators to trigger failures?
    \item RQ2: How diverse are the generated failure-revealing test scenarios?
    \item RQ3: How effectively and efficiently do the test generators perform, i.e., given a certain test budget how many test scenarios are generated, what proportion of the scenarios is valid, and what proportion of the valid test scenarios leads to triggering failures?
\end{enumerate}

The experiments are simulation scenarios generated by a Python test scenario generator and executed by the simulation engine. BeamNG.AI is the autonomous driving agent controlling the ego car in the simulation (Fig. \ref{fig:Experimental_Setup}). In order to provide quantitative answers to the RQs, we use the following quality criteria to assess the bio-inspired test scenario generators in {Deeper}:
\begin{itemize}
    \item Detected Failures: The number of generated test scenarios that lead to failures, w.r.t the given tolerance threshold.
    \item Failure Diversity: The dissimilarity between the test scenarios that lead to the failures. Generating diverse failure-revealing test scenarios is of interest, since triggering the same failures multiple times results in wasting the test budget, e.g., computation resources. In order to measure the failure diversity, we rely on a two-step strategy adopted by the SBST 2021 tool competition. It extracts, first, the road segments related to the failures, then computes the \textit{sparseness}, which is considered as the average of the maximum Levenshtein distance \cite{levenshtein1966binary} between those road segments.\\
    The failure-related road segments are referred to as the parts of the road 30 meters before the OBE and 30 meters after it, and accordingly, the sparseness is calculated as follows:
    \begin{equation}
     Sparseness = \frac{\sum_{i\in OBEs}\max_{j\in OBEs} Lev\_dist(i, j)}{|OBEs|}
    \end{equation}
    where $Lev\_dist(i, j)$ indicates the weighted Levenshtein distance between the road segments.
    \item Test generation efficiency and effectiveness: It indicates how the test generator uses the given test budget to generate the test scenarios, in particular how many test scenarios are generated in total, what fraction of them are valid, and what fraction of the valid ones triggers failures.
\end{itemize}

\begin{figure}
  \centering
  \includegraphics[width=.75\columnwidth, height =5.5cm]{Experimental_Setup.png}
  \caption{An overview of the experimental setup}
  \label{fig:Experimental_Setup}
\end{figure}

\textit{Experiments:} We design two sets of experiments as implemented in the SBST 2021 CPS tool competition. In order to provide a comparative analysis, we compare the results of the proposed test generators in {Deeper} with the presented test generators in the tool competition, i.e., Frenetic \cite{Frenetic}, GABExploit and GABExplore \cite{GABezier}, Swat \cite{SWAT}, and also the earlier version of {Deeper} \cite{Moghadam2021Deeper}. We run the test generators on the test subject based on the same two experiment configurations as in the competition, which are shown in table \ref{table:Experiment_configuration}. The \textit{SET1} of experiments provides a 5-hour test generation budget, meanwhile sets the failing tolerance threshold to a high value, $0.95$, and does not consider any speed limit. This experiment configuration might lead to a more careless style of driving. The \textit{SET2} of the experiments allocates a shorter time budget for the test generation and considers a lower tolerance threshold, $0.85$, while imposing a speed limit of 70 km/h---promoting a more careful driving style. To ensure a fair comparison, we run each tool the same number of times on the same dedicated machine. We run each test generator $5$ times in experiment configuration SET1 and $10$ times in SET2 and report distributions of the results.

\begin{table}[h]
\centering
\caption{Experiment configurations}
\begin{tabular}{ |m{2.2cm}|m{1.7cm}|m{1.7cm}|m{1.7cm}|m{1.7cm}|}
\hline
\textbf{Name} & \textbf{Test Budget (h)} & \textbf{Map Size (m\textsuperscript{2})} & \textbf{Speed Limit (Km/h)} & \textbf{Failing Tolerance Threshold ($\%$)}\\
\hline
SET1 (careless driving) & 5h  & $200\times200$  & None & 0.95 \\
\hline
SET2 (cautious driving) & 2h  & $200\times200$  & 70 Km/h & 0.85 \\
\hline
\end{tabular}
\label{table:Experiment_configuration}
\end{table}

\section{Results and Discussion}\label{D:Sec::Result&Discussion}
This sections reports results corresponding to the three RQs.

\subsection{Detected Failures (RQ1)}
Figure \ref{fig:Exp_1_DetectedFailures} reports the number of triggered failures by each of the tools in experiment configuration SET1. In this regard, $(\mu+\lambda)$ ES-driven test generator in {Deeper} could successfully trigger at least 2X more OBEs than the highest record---held by GABExploit---in the competition. At the same time, {Deeper} $(\mu+\lambda)$ ES showed more consistent performance over the runs, i.e., with lower standard deviation, compared to GABExploit, which revealed highly different behavior across the runs (e.g., returning over 100 OBEs in some runs, but failed to trigger any failure in other runs \cite{panichella2021sbst}). PSO- and GA-driven test generators in {Deeper}, in half of the cases and also on average, triggered higher numbers of failures than the competition's test generators---except for GABExploit, which showed comparable results. Moreover, the PSO- and GA-driven approaches were able to trigger the failures in which the ego-car invades the opposite lane of the road---a type of failure that has been typically considered difficult to trigger in most of the test generator tools \cite{panichella2021sbst}. Lastly, in SET1, {Deeper} $(\mu,\lambda)$ ES clearly gave a better performance, in terms of the number of detected failures, than {Deeper} NSGA-II, GABExplore, and Swat.

Similarly, figure \ref{fig:Exp_2_DetectedFailures} presents the number of triggered failures in experiment configuration SET2. With regard to the limited time budget, the speed limit of 70 km/h and the tolerance threshold $0.85$, in contrast to {Deeper} NSGA-II, GABExplore, GABExploit, and Swat, none of the newly proposed test generators in {Deeper} left an experiment without triggering any failure or with a very low number of failures (i.e., less than $3$), which means they are able to detect failures even within a limited test budget and strict constraints such as setting a speed limit (See Table \ref{table:TriggeredFailures_SET2}). It is noted that {Deeper} NSGA-II, GABExplore, and Swat triggered just equal or less than $1$ failure in a considerable number of experiments done based on configuration SET2. Therefore, all the new test generators in {Deeper} outperform {Deeper} NSGA-II, GABExplore, and Swat w.r.t the number of triggered failures within a limited test time and moreover the PSO-driven test generator results in a very comparable number of detected failures to Frenetic. Additionally, in the experiment configuration SET2, the GA-driven test generator is still able to trigger the failures showing the invasion of the car to the opposite lane of the road.

\begin{figure}[h]
  \centering
  \includegraphics[width=.85\columnwidth, height =6.5cm]{Exp_1_Detected_Failures.pdf}
  \caption{Number of detected failures in SET1}
  \label{fig:Exp_1_DetectedFailures}
\end{figure}

\begin{figure}[h]
  \centering
  \includegraphics[width=.85\columnwidth, height =6.5cm]{Exp_2_Detected_Failures.pdf}
  \caption{Number of detected failures in SET2}
  \label{fig:Exp_2_DetectedFailures}
\end{figure}

\begin{table}[h]
\centering
\caption{Triggered failures in SET2}
\begin{tabular}{ |m{2.5cm}|m{0.75cm}|m{0.75cm}|m{0.75cm}|m{0.75cm}|m{0.75cm}|m{0.75cm}|m{0.75cm}|m{0.75cm}|m{0.75cm}|m{0.75cm}|}
\hline
\textbf{Test Generator} & \footnotesize{Exp.1} & \footnotesize{Exp.2} & \footnotesize{Exp.3} & \footnotesize{Exp.4}& \footnotesize{Exp.5} & \footnotesize{Exp.6} & \footnotesize{Exp.7} & \footnotesize{Exp.8} & \footnotesize{Exp.9} & \footnotesize{Exp. 10} \\
\hline
Deeper NSGA-II & \cellcolor{blue!50} 0 & \cellcolor{blue!50} 0 & \cellcolor{blue!50} 0 & \cellcolor{blue!25} 1 & \cellcolor{blue!25} 1 & \cellcolor{blue!50} 0 & \cellcolor{blue!50} 0 & \cellcolor{blue!50} 0 & 3 & \cellcolor{blue!50} 0 \\
\hline
Frenetic & 12 & 8 & 9 & 11 & 19 & 10 & 15 & 20 & 6 & 23 \\
\hline
\small{GABExplore} & \cellcolor{blue!50} 0 & \cellcolor{blue!25} 1 & \cellcolor{blue!25} 1 & \cellcolor{blue!50} 0 & \cellcolor{blue!25} 1 & 3 & \cellcolor{blue!50} 0 & \cellcolor{blue!50} 0 & 3 & \cellcolor{blue!50} 0 \\
\hline
\small GABExploit & 84 & 18 & 126 & 38 & \cellcolor{blue!50}  0 & 11 & 47 & \cellcolor{blue!50} 0 & 28 & 5 \\
\hline
Swat & \cellcolor{blue!50} 0 & \cellcolor{blue!50} 0 & \cellcolor{blue!50} 0 & \cellcolor{blue!50} 0 & 5 & \cellcolor{blue!25} 1 & \cellcolor{blue!25} 1 & 2 & \cellcolor{blue!25} 1 & 2 \\
\hline
Deeper PSO & 54 & 8 & 13 & 13 & 10 & 9 & 10 & 15 & 18 & 12 \\
\hline
Deeper $(\mu,\lambda)$ ES & 8 & 4 & 3 & 5 & 7 & 7 & 3 & 5 & 7 & 9 \\
\hline
Deeper $(\mu+\lambda)$ ES & 29 & 5 & 9 & 4 & 4 & 7 & 19 & 3 & 14 & 5 \\
\hline
Deeper GA & 11 & 7 & 11 & 15 & 14 & 9 & 25 & 9 & 14 & 20 \\
\hline
\end{tabular}
\label{table:TriggeredFailures_SET2}
\end{table}

\subsection{Diversity of Failures (RQ2)}
Figure \ref{fig:Exp_1_DetectedFailures} depicts the diversity of the detected failures in SET1 in terms of the distribution of the failures' sparseness. All the new test generators in {Deeper} resulted in a considerable improvement on failure sparseness compared to the first version, {Deeper} NSGA-II in the competition. Meanwhile, among the new test generators of {Deeper}, the PSO- and GA-driven test generators lead to a higher average sparseness compared to the $(\mu+\lambda)$ and $(\mu,\lambda)$ ES approaches. {Deeper} PSO, $(\mu+\lambda)$, and GA also show a higher sparseness than GABExploit---both on average and in almost half of the cases. However, still the failure diversity---in terms of the special sparseness metric defined by the competition---promoted by the new test generators in {Deeper} are not as high as Frenetic, GABExplore, and Swat. Note that GABExplore and Swat in $20\%$ of the experiments did not report any sparseness figure, since they just triggered one failure in each of those experiments.

Figure \ref{fig:Exp_2_FailureDiversity} shows the distribution of the failures' sparseness in SET2. In the limited test budget and strict driving constraints, all the newly test generators again show a big improvement on promoting failure sparseness in comparison to the first version, {Deeper} NSGA-II. In the meantime, {Deeper} PSO, $(\mu+\lambda)$, and GA promote comparable levels of failure diversity, though more consistent, compared to GABExploit. At the same time, GABExplore and Swat in around $70\%$ of the experiments in SET2 did not provide any sparseness figure, since they triggered just one or zero failure (See Table \ref{table:TriggeredFailures_SET2}).

\begin{figure}[h]
  \centering
  \includegraphics[width=.85\columnwidth, height =6.5cm]{Exp_1_Sparseness.pdf}
  \caption{Failure diversity in terms of sparseness in SET1}
  \label{fig:Exp_1_FailureDiversity}
\end{figure}

\begin{figure}[h]
  \centering
  \includegraphics[width=.85\columnwidth, height =6.5cm]{Exp_2_Sparseness.pdf}
  \caption{Failure diversity in terms of sparseness in SET2}
  \label{fig:Exp_2_FailureDiversity}
\end{figure}

\subsection{Test Effectiveness and Efficiency (RQ3)}
In SBST competition, test effectiveness and efficiency were indicated by how many test scenarios are generated and what proportion of the scenarios is valid, given a certain test budget. They basically show how well the test generator is able to utilize the test budget. Figures \ref{fig:Exp_1_TestEfficiency} and \ref{fig:Exp_2_TestEfficiency} report the average number of total test scenarios, as well as the number of valid and invalid scenarios generated by each tool in SET1 and SET2 respectively. Generally, the new test generators in {Deeper} utilize the test budget more efficiently than the competition tools and generate a higher number of test scenarios within the given test time. In this regard, {Deeper} PSO results in the highest efficiency (e.g., generates more than 650 scenarios on average within 5 hours) among all the test generators in both experimental configurations. Regarding the number of valid test scenarios, all the {Deeper} test generators along with Swat lead to an almost comparable number of valid scenarios. However, with respect to the ratio of the valid test scenarios to the total generated ones---called test effectiveness according to the competition evaluation---Swat, {Deeper} NSGA-II, and GABExploit are the ones showing the highest result.

To answer RQ3 on test effectiveness and efficiency, in addition to the metrics defined and used by the competition, we defined an extra metric,  aggregated test effectiveness called \textit{effectiveness plus}, which indicates what proportion of the valid test scenarios leads to triggering failures. It is defined as the ratio of the triggered failures to the number of valid test scenarios and is intended to present the effectiveness of the test generators w.r.t meeting the target---detecting failures. Figures \ref{fig:Exp_1_Effectiveness_Plus} and \ref{fig:Exp_2_Effectiveness_Plus} report the test effectiveness plus for the test generators in SET1 and SET2 respectively. With respect to the effectiveness plus, {Deeper} $(\mu+\lambda)$ ES-driven test generator results in the highest target-based effectiveness in SET1 and then {Deeper} PSO and GA are the next most effective tools. In SET2, GABExploit shows the highest effectiveness plus, while Frenetic, {Deeper} PSO and GA are the next most effective ones. It is worth noting that as shown in Fig. \ref{fig:Exp_1_Effectiveness_Plus} and \ref{fig:Exp_2_Effectiveness_Plus} both {Deeper} PSO- and GA-driven test generators keep their effectiveness in generating failure-revealing scenarios in both experimental conditions of SET1 and SET2, which means they are effective test generators even within a limited test budget and strict driving constraints.

\begin{figure}
  \centering
  \includegraphics[width=.85\columnwidth, height =6.5cm]{Exp_1_Test_Efficiency.pdf}
  \caption{Test generation effectiveness and efficiency in SET1}
  \label{fig:Exp_1_TestEfficiency}
\end{figure}

\begin{figure}
  \centering
  \includegraphics[width=.85\columnwidth, height =6.5cm]{Exp_2_Test_Efficiency.pdf}
  \caption{Test generation effectiveness and efficiency in SET2}
  \label{fig:Exp_2_TestEfficiency}
\end{figure}

\begin{figure}[h]
  \centering
  \includegraphics[width=.85\columnwidth, height =6.5cm]{Exp_1_Effectiveness_Plus.pdf}
  \caption{Test generation effectiveness plus in SET1}
  \label{fig:Exp_1_Effectiveness_Plus}
\end{figure}

\begin{figure}[h]
  \centering
  \includegraphics[width=.85\columnwidth, height =6.5cm]{Exp_2_Effectiveness_Plus.pdf}
  \caption{Test generation effectiveness plus in SET2}
  \label{fig:Exp_2_Effectiveness_Plus}
\end{figure}

\subsection{Threats to Validity}
The evaluation of {Deeper} comes with a set of threats to construct, internal, and external validity of the results.\\
\textbf{\textit{Construct validity:}} The choices of the fitness function---the distance of the car position from the center of the lane---and also the metrics used for calculating the sparseness and indicating the diversity of the test scenarios---weighted Levenshtein distance---in this study are domain-specific. However, we have based our choices on the sound metrics adopted by other research works in the literature \cite{panichella2021sbst, gambi2019asfault, riccio2020model}.\\
\textbf{\textit{Internal validity:}} The randomized nature of the used bio-inspired algorithms could be a source of threats to the internal validity of the results. In this regard, we follow the guidelines given by Arcuri and Briand \cite{arcuri2014hitchhiker} for the evaluation and analysis of the results, and mitigate this threat by running the experiments multiple times (e.g., 5 times in SET1 and 10 times in SET2), reporting the distribution and statistics of the results (e.g., using Box plots to show the results), using same algorithm settings, e.g., population size, crossover and mutation probabilities, in the proposed test generators.\\
\textbf{\textit{External validity:}} The choice of the test subject system is a potential threat to the external validity of the results. However, the test system in this study is one of the main and commonly used systems in self-driving cars, and furthermore different ML models with various quality levels (i.e., different accuracy levels) could be deployed within the BeamNG.AI agent, and in this regard, the proposed test scenario generation techniques can still be used. Nonetheless, it still offers one type of DL-based systems in self-driving cars, and further studies are required to address the testing of other DL-driven systems, meanwhile, we also keep the tool open for extensions, for example, to support the execution of test scenarios in other simulators.

\section{Related Work}\label{D:Sec::RelatedWork}
Input data assurance alongside the model and integration testing \cite{hawkins2021AMLASguidance, riccio2020testing} are considered different test levels investigated by various research works for ML systems. Model testing could be regarded as unit testing for ML components and integration testing focuses on the issues emerging after the integration of the ML model into the system.
Regarding access to the test subject, there are black-box and white-box testing approaches analogous to traditional non-ML systems. Black-box testing involves access only to the ML inputs and outputs, while the white-box testing implies access to the internal architecture of the test ML subject, code, hyperparameters, and training/test data. However, Riccio et al. in \cite{riccio2020testing} also introduced another type of ML testing called data-box, which requires access to data plus everything that a black-box test requires.

Test input data that can reveal failures in the test subject is the most common generated test artifact in the literature related to testing of automotive AI systems \cite{riccio2020testing}. Depending on the test level and the test subject, the inputs could be images, for instance, as used in DeepTest \cite{tian2018deeptest}, or test scenario configurations as used in \cite{ebadi2021efficient}. A brief overview of the most common techniques used to generate the test data is as follows:

\textbf{Input data mutation.} This type of mutation involves generating new inputs based on the transformation of the existing input data. For instance, DeepXplore \cite{pei2017deepxplore} uses such input transformations to find the inputs triggering different behaviors between similar autonomous driving DNN models, while also striving to increase the level of neuron coverage. Moreover, in many studies, those transformations are based on metamorphic relations. DeepTest \cite{tian2018deeptest} applies different transformations to a set of seed images with the aim of increasing neuron coverage and uses metamorphic relations to find the erroneous behaviors of different Udacity DNN models for self-driving cars. DeepRoad \cite{zhang2018deeproad} uses a GAN-based metamorphic testing technique to generate input images to test three autonomous driving Udacity DNN models. It defines the metamorphic relations such that the driving behavior in a new synthesized driving scene is expected to be consistent with the one in the corresponding original scene.

\textbf{Test scenario manipulation.} Another major category of the methods to generate test input data is based on the manipulation and augmentation of the test scenarios. Most of the works in this category use search-based techniques to go through the search space of the scenarios to find the failure-revealing or collision-provoking test scenarios. In this regard, simulators as a form of digital twins have played a key role to generate and capture those critical failure-revealing test scenarios. Simulation-based testing can act as an effective complementary solution to field testing, since exhaustive field testing is expensive, meanwhile inefficient, and even dangerous, in some cases. Recently, various high-fidelity simulators such as the ones using physics-based models (e.g., SVL simulator \cite{SVLSimulator}, Pro-SiVIC \cite{ProSivic_belbachir2012simulation}, and PreScan \cite{PreScan}) and the ones based on game engines (e.g., BeamNG.tech \cite{BeamNG} and CARLA \cite{dosovitskiy2017carla}) have considerably contributed to this area by providing the possibility of realistic simulations of functionalities in autonomous driving.

Accordingly, various testing approaches relying on the simulators have been presented in the literature and in this regard, search-based techniques have been frequently used to address the generation of failure-revealing test scenarios. Abdessalem et al. utilize multi-objective search algorithms such as NSGA-II \cite{ben2016testing} along with surrogate models to find critical test scenarios with fewer simulations and then at less computation time for a pedestrian detection system. In a following study \cite{abdessalem2018testing}, they use MOSA \cite{panichella2015reformulating}---a many-objective optimization search algorithm--- along with objectives based on the branch coverage and some failure-based heuristics to detect undesired and failure-revealing feature interaction scenarios for integration testing in a self-driving car. Further, in another study \cite{abdessalem2018testinglearnable}, they leverage a combination of multi-objective optimization algorithms (NSGA-II) and decision tree classifier models---referred to as a learnable evolutionary algorithm---to guide the search-based process of generating critical test scenarios and also to refine a classification model that can characterize the failure-prone regions of the test input space for a pedestrian detection and emergency braking system.
Haq et al. \cite{haq2020automatic} use many-objective search algorithms to generate critical test data resulting in severe mispredictions for a facial key-points detection system in the automotive domain. Ebadi et al. \cite{ebadi2021efficient} benefit from GA along with a flexible data structure to model the test scenarios and a safety-based heuristic for defining the objective function to test the pedestrian detection and emergency braking system of the Baidu Apollo (an autonomous driving platform) within the SVL simulator.

Regarding the impacts of the simulators in this area, Haq et al. \cite{ul2022efficient} provide a comparison between the results of testing DNN-based ADAS using online and offline testing. Their results clearly motivate an increased focus on online testing as it can identify faults that never would be detected in offline settings---whereas the opposite does not appear to be likely. Our current study responds to this call, and motivates our work on systems testing in simulated environments. With regard to a different perspective, Borg et al. \cite{borg2021digital} discuss the consistency between the test results obtained from running the same experiments based on two different simulators and investigate the reproducibility of the results in both simulators. When running the same testing campaign in PreScan and ESI Pro-SiVIC, the authors found notable differences in the test outputs related to revealed safety violations and the dynamics of cars and pedestrians.

\section{Conclusion}\label{D:sec::summary}
{Deeper} in its extended version utilizes a set of bio-inspired algorithms, genetic algorithm (GA), $(\mu + \lambda)$ and $(\mu , \lambda)$ evolution strategies (ES), and particle swarm optimization (PSO), to generate failure-revealing test scenarios for testing a DL-based lane-keeping system. The test subject is an AI agent in BeamNG.tech's driving simulator. The extended {Deeper} contains four new bio-inspired test generators that leverage a quality seed population and domain-specific cross-over and mutation operations tailored for the presentation model used for modeling the test scenarios. Failures are defined as episodes where the ego car drives partially out of the lane w.r.t a certain tolerance threshold. In our empirical evaluation we focused to answer three main questions as first how many failures the test generators can detect, second how much diversity they can promote in the failure-revealing test scenarios, and third how effectively and efficiently they can perform, w.r.t different target failure severity (i.e., in terms of tolerance threshold), available test budget, and driving style constraints (e.g, setting speed limits). Our results show that the newly proposed test generators in {Deeper} present a considerable improvement on the previous version and they are able to act as effective and efficient test generators that provoke a considerable number of diverse failure-revealing test scenarios for testing an ML-driven lane-keeping system. They show considerable effectiveness in meeting the target, i.e., detecting diverse failures, with respect to different target failures intended and constraints imposed. In particular, they act as more reliable test generators than most of the counterpart tools for provoking diverse failures within a limited test budget and with respect to strict constraints.

As some directions for future work, we plan to apply the proposed approaches to testing further types of ML-based lane-keeping systems, i.e., more industrial ones and also in other state-of-the-art simulation platforms. We also plan to extend the approaches by applying machine learning-based techniques such as reinforcement learning or Generative Adversarial Networks (GANs) for empowering the discovery of failure-revealing test scenarios.

\section*{Acknowledgment}
This work has been funded by Vinnova through the ITEA3 European IVVES (\url{https://itea3.org/project/ivves.html}) and H2020-ECSEL European AIDOaRT (\url{https://www.aidoart.eu/}) projects. Furthermore, the project received partially financial support from the SMILE~III project financed by Vinnova, FFI, Fordonsstrategisk forskning och innovation under the grant number: 2019-05871.

\bibliographystyle{IEEEtran}

Software Engineering (cs.SE); Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Neural and Evolutionary Computing (cs.NE)