HUDD: A tool to debug DNNs for safety analysis

Abstract: We present HUDD, a tool that supports safety analysis practices for systems enabled by Deep Neural Networks (DNNs) by automatically identifying the root causes for DNN errors and retraining the DNN. HUDD stands for Heatmap-based Unsupervised Debugging of DNNs, it automatically clusters error-inducing images whose results are due to common subsets of DNN neurons. The intent is for the generated clusters to group error-inducing images having common characteristics, that is, having a common root cause. HUDD identifies root causes by applying a clustering algorithm to matrices (i.e., heatmaps) capturing the relevance of every DNN neuron on the DNN outcome. Also, HUDD retrains DNNs with images that are automatically selected based on their relatedness to the identified image clusters. Our empirical evaluation with DNNs from the automotive domain have shown that HUDD automatically identifies all the distinct root causes of DNN errors, thus supporting safety analysis. Also, our retraining approach has shown to be more effective at improving DNN accuracy than existing approaches. A demo video of HUDD is available atthis https URL.

\section{Conclusion}
\label{sec:conclusion}

We introduced \emph{HUDD}\xspace, a toolset that supports safety analysis practices for DNN-enabled safety-critical systems.
It generates clusters (i.e., root cause clusters, RCCs) containing misclassified input images
sharing a common set of characteristics that are plausible causes of errors.
In addition, \emph{HUDD}\xspace minimizes the effort required to select and label
additional images to be used to augment the training set and improve the DNN.

Empirical evaluation with simulator images show that \emph{HUDD}\xspace generates clusters
with images that provide explanations for DNN errors; further, results with both simulated and real images show how these clusters can be effectively used to select new images for retraining, in a way that is more efficient than existing practices and leading to better DNN accuracy.

\emph{HUDD}\xspace is a software engineering tool to support the development of ML-based systems. Indeed, by helping identify different plausible causes of DNN errors, it supports engineers in specifying solutions to improve the system. For example, (1) RCCs that highlight an incomplete training set suggest further training, whereas (2) RCCs with borderline cases may suggest introducing technical countermeasures. Further, the automated retraining strategy implemented by \emph{HUDD}\xspace, in addition to supporting automated debugging, enables engineers to justify their selection of retraining images according to safety principles (i.e., to show the intent of eliminating root causes of errors).

\balance
\section{Empirical Evaluation}
\label{sec:evaluation}

This section provides an overview of the main findings of an evaluation conducted to address the following research questions~\cite{HUDD:TRel}:
\begin{itemize}
\item[RQ1] Does \emph{HUDD}\xspace enable engineers to identify the root causes of DNN errors?
\item[RQ2] How does \emph{HUDD}\xspace compare to traditional DNN accuracy improvement practices?
\end{itemize}

For our empirical evaluation, we considered six DNNs. A gaze detection system (GD) that determines the gaze direction of a human face.
A drowsiness detection system (OC) that features the same architecture as the gaze detection system, except that the DNN predicts whether eyes are closed.
A head poses detection system (HPD) that classifies the position of a person's head in an image according to nine classes: straight, bottom-left, left, top-left, bottom-right, right, top-right, reclined, looking up.
A facial landmarks detection system (FLD) that identifies the location of the pixels corresponding to 27 face landmarks delimiting seven face elements.
An object detection system (OD) that tries to detect the existence of eyeglasses.
A traffic signs recognition system (TSR) that recognizes the presence of traffic signs in road images.

RQ1 investigates whether HUDD is feasible and generates RCCs with images presenting a common set of characteristics that are plausible causes of DNN errors.

To determine if the RCCs generated by \emph{HUDD}\xspace include images with common characteristics, we relied on images generated using simulators. Since simulator images can be associated with the simulator parameter values used to generate them (e.g., the vertical angle of a person’s head), we could objectively determine if the images in the same cluster present common characteristics. Indeed, a characteristic that is shared between the images in the same RCC shall lead to a lower within-cluster variance, for at least one simulator parameter, compared to the whole test set. Also, by focusing on the parameters showing a high variance reduction (e.g., >50\%), we can objectively determine if the RCCs help engineers spot the root cause of an error. Indeed, if the average value for such parameters is close to a value that likely leads to error-inducing images (e.g., a gaze angle that is borderline between two gaze directions) we can assume that the RCC provides an explanation for the DNN error that can be understood by the engineer inspecting the images.

Our results show that a very high percentage of the clusters (i.e., between 57\% and 100\%) include at least one parameter with 50\% reduction in variance, which means that, for most of the clusters, engineers can identify commonalities among images. Also, all the parameters with high reduction in variance are associated with image characteristics that are plausible causes of errors.

To evaluate if the visual inspection of root cause clusters is practically feasible, we report on the number of clusters generated by HUDD. Precisely, the ratio of error-inducing images that should be visually inspected when relying on HUDD, should provide an indication of the time saved with respect to current practice (i.e., manual inspection of all the error-inducing images). To perform the evaluation, based on our experience, we assumed that engineers visually inspect five images for each root cause cluster. Table~\ref{table:RQ1.1} provides summary data; we can observe that the ratio of error-inducing images that is inspected with \emph{HUDD}\xspace is low, ranging from 1.49\% (GD) to 22.84\% (FLD), with a median of 6.87\%. This suggests that the analysis supported by \emph{HUDD}\xspace saves a great deal of effort with respect to current practice (i.e., manual inspection of all the error-inducing images). A user study concerning the time savings introduced by HUDD is part of our future work

\input{tables/RQ1_1}

RQ2 concerns DNN improvement. We considered four DNNs working with simulator images and two DNNs working with real-world images. We compared HUDD with two baseline approaches: (1) retraining with failing images selected from a subset of the improvement set and (2) retraining with random images. To avoid bias, for all the considered approaches, we label the same number of images. Experiments were repeated ten times. HUDD leads to significantly larger accuracy improvements than baselines,
increasing DNN accuracy up to 30.24\%.
\section{Architecture \& Usage}
\label{sec:implementation}

 \begin{figure}[tb]
 \begin{center}
 \includegraphics[width=\columnwidth]{images/HUDD_tool}
 \caption{Architecture (left) and output structure (right).}
 \label{fig:tool}
 \end{center}
 \end{figure}

Figure~\ref{fig:tool} provides the architecture of the \emph{HUDD}\xspace toolset.
\emph{HUDD}\xspace is implemented in Python with the following dependencies.
\emph{Tensorflow}~\cite{tensorflow} and \emph{PyTorch}\cite{PyTorch} are used for DNN models.
\emph{NumPy}, an array programming library~\cite{numpy}, is used for the manipulation and storage of heatmaps.
\emph{Pandas}, a data analysis tool~\cite{pandas}, is used to compute heatmap distance matrices.
\emph{Scipy}, a library for scientific computing~\cite{SciPy}, is used for the generation of clusters.
\emph{Kneed}, a library that implements the kneedle algorithm~\cite{kneedle}, is used to determine the optimal number of clusters.

The HUDD tool consists of a command line user interface called \emph{HUDD.Helper} and five modules: TestModule, HeatmapModule, ClusterModule, AssignModule, RetrainModule.

To execute HUDD,
the engineer provides to the \emph{HUDD.Helper} the DNN model to be analyzed.
The DNN under analysis shall be stored in the \emph{DNNModels} folder; the datasets shall be provided in the \emph{DataSets} folders \emph{TrainingSet}, \emph{TestSet}, and \emph{ImprovementSet}.
\emph{HUDD}\xspace relies on LRP and thus requires DNN models that integrate the LRP backpropagation function.
We provide an AlexNet model to be used for classification tasks and a KPNet model that can be used for regression tasks.
However, DNN models can be easily extended to integrate LRP-based heatmap generation by following existing guidelines~\cite{ExplainMontavon}.
For example, our AlexNet implementation is an extension of the PyTorch AlexNet.
The HUDD.Helper orchestrates the execution of all the other modules. The intermediate results generated by \emph{HUDD}\xspace are stored within the temporary folder \emph{T}, which is kept to enable further inspection of all the processed data.

The \emph{TestModule} uses the DNN under analysis to process the inputs in the training and test set. Outputs are exported in the files \emph{trainResult.csv} and \emph{testResult.csv}. The former is used to compute training set accuracy, which is used to determine if the improved DNN is better than the original one (\emph{HUDD}\xspace Step 7). The latter is used to determine which are the error-inducing images to be used to generate RCCs (\emph{HUDD}\xspace Step 1).

The \emph{HeatmapModule} generates heatmaps for error-inducing images. For each DNN layer, it stores, in the \emph{Heatmaps} directory, a NumPy file with the heatmaps of all the error-inducing images.

The \emph{ClusterModule}, for each layer, computes the distance matrix and exports it in an \emph{XLSX} file.
Also, it performs hierarchical agglomerative clustering based on the heatmaps generated for each layer and selects the optimal number of clusters.
Finally, for each $K^{th}$ layer, it stores the generated clusters in a directory called \emph{T/ClusterAnalysis/Layer{K}}.
The clusters for the layer showing the best results (layer $X$)  are copied in the parent folder (i.e., \emph{./T/Layer{X}/}).
For each RCC, the \emph{ClusterModule} generates a directory with all the images belonging to the cluster,
which are to be visually inspected by engineers as per HUDD Step 2.

To simplify the inspection of RCCs, the \emph{ClusterModule} also generates a set of animated GIF images, one for each cluster. Each generated GIF image shows all the images belonging to a cluster one after the other. Animated GIFs enable engineers to inspect a large number of images in a few seconds (e.g., we configure our tool to visualize 100 images per minute) thus facilitating the detection of the common characteristics among them.

The \emph{AssignModule} processes the ImprovementSet images and stores the unsafe set in the  folder \emph{UnsafeSet}. Finally, the \emph{RetrainModule} retrains the DNN using the images in the training and unsafe sets.
The retrained DNN model is saved in the DNNModels directory.

Our toolset, case studies, and results are available online~\cite{REPLICABILITY}.
\section{Introduction}
\label{sec:introduction}

Deep Neural Networks (DNNs) are common building blocks for safety-critical cyber-physical systems (e.g., their perception layer), particularly in the automotive sector.
Common examples include Advanced Driver Assistance Systems (ADAS), where DNNs are used to automate emergency braking and lane changing~\cite{TeslaDNN}.
DNN-enabled systems are a key product not only for large companies but also for car component manufacturers~\cite{IEE,ZF}. This is the case of IEE Sensing~\cite{IEE}, our industry partner, who develops in-vehicle monitoring systems such as drowsiness detection and gaze detection systems~\cite{Naqvi2018}.

When DNN-based systems are used in a safety-critical context (e.g., automotive),
developers must comply with
safety standards such as ISO26262~\cite{ISO26262} and ISO/PAS 21448~\cite{SOTIF}.
Such safety standards enforce the identification of the situations in which the system might be unsafe (i.e., outputs leading to safety violations) and the design of countermeasures to put in place (e.g., integrating different types of sensors).
However, since DNNs transform high-dimensional vectors through a large number of activation functions whose parameters are learned during training,
engineers cannot understand the rationale of predictions through manual inspection of DNN code and, consequently, they cannot rely on traditional safety analysis practices. For this reason, safety standards targeting DNN-enabled systems (e.g., ISO/PAS 21448) suggest (1) to rely on accuracy evaluation (i.e., test the DNN using inputs generated by simulators or collected in the field) to identify unsafe scenarios and (2) to rely on the manual inspection of error-inducing images to perform \emph{root cause analysis} (i.e., to understand what are the characteristics of the inputs that lead to a DNN error).

In this context, quantitative targets for accuracy evaluation may be used to demonstrate that unsafe situations are unlikely; however, standards like ISO/PAS 21448 point out that quantitative targets are not sufficient and that engineers remain liable for potentially hazardous scenarios underrepresented in the test set.

Root cause analysis is expected to help reduce such liability risk. Indeed, engineers can retrain the DNN with additional images similar to the ones leading to DNN errors; however, \emph{the manual identification of additional inputs to retrain the DNN is expensive}.
Also, engineers can introduce countermeasures to make the system robust against unsafe conditions (e.g., by relying on both radar and vision to take decisions).
Unfortunately, \emph{the manual identification of unsafe conditions is error-prone}. For example, engineers may overlook unsafe conditions that are underrepresented in the test set.
Indeed, human body simulators may generate head images with an horizontal angle determined based on a uniform distribution, between 160 (head turned right) and 220 degrees (head turned left). As a result, very few images with an angle of 160 or 220 degrees are generated and, though it may be an unsafe condition (i.e., one eye is barely visible and the gaze direction prediction may be inaccurate) experiments based on test sets generated with such simulators may suggest that the DNN is on average very accurate and engineers may not notice such unsafe conditions.

Existing toolsets do not help engineers address the above-mentioned problems.
When inputs are images, existing solutions for root cause analysis generate heatmaps  that use colors to capture the importance of pixels in their contribution to a DNN result~\cite{Selvaraju17,Montavon2019}.
By inspecting the heatmaps generated for a set of erroneous results, a human operator can determine that these heatmaps highlight the same objects, which may suggest the root cause of the problem (e.g., long hair~\cite{Selvaraju17}).
Based on the identified root cause, engineers can then retrain the DNN using additional images with similar characteristics.
Unfortunately, this process is expensive and error-prone because it relies on the visual inspection of many generated heatmaps.

In this paper, we present HUDD, the toolset that automates our methodology for the identification of root causes for DNN errors and DNN retraining~\cite{HUDD:TRel}.
\emph{HUDD}\xspace relies on hierarchical agglomerative clustering~\cite{King:2014} to identify the distinct root causes of DNN errors.
It configures clustering with a specific distance function based on the heatmaps computed for internal DNN layers.
A subset of the images belonging to each cluster is inspected by the engineer who is thus helped in determining unsafe conditions (i.e., commonalities among the images in a same cluster), including the infrequent ones (i.e., clusters with few members).
Further, \emph{HUDD}\xspace relies on the computed clusters to identify new images to be used to retrain the DNN.
Given a potentially large set of unlabeled images, \emph{HUDD}\xspace
selects the subset of images that are more likely to belong to the identified clusters.
These images are assumed to include potentially unsafe conditions and are then used to retrain the DNN.
We performed an empirical evaluation on six DNNs.
Our empirical results show that \emph{HUDD}\xspace can automatically and accurately identify the different root causes of DNN errors.
Also, our results show that the \emph{HUDD}\xspace retraining process improves DNN accuracy up to 30.24 percentage points and is more effective than baseline approaches.

In the remaining sections, we present our methodology, outline the tool, highlight the findings from our evaluation of \emph{HUDD}\xspace with six case studies, and compare with related work.

\section{Related Work}
\label{sec:related}

A number of tools supporting DNN explanation are available nowadays~\cite{ExplainMontavon,huang2018survey}.
However, for explanations concerning DNNs that process images, such frameworks boil down to generating heatmaps
one for every error-inducing input image,
that shall be visually inspected by engineers.
Example frameworks are INNvestigate~\cite{innvestigate} and TorchRay~\cite{TorchRay}.
The cost of the manual inspection of heatmaps is one of the problems addressed by HUDD.

Research on the automated debugging and repair of DNNs is still at very early stages and includes MODE~\cite{Ma2018} and Apricot\cite{Zhang19}.
Similarly to \emph{HUDD}\xspace, MODE automatically identifies the images to be used to retrain a DNN~\cite{Ma2018}.
However, it cannot identify the root causes of DNN errors, which is a major limitation in our context.
Also MODE entails repeated modification and retraining of the DNN under test, which is an expensive endeavor.
Further, no tool implementing MODE is available.
Finally, in our empirical evaluation, we evaluated HUDD with an Object Detection (OD) classifier DNN that has been used for the evaluation of MODE. Despite differences in the DNN architecture used by HUDD and MODE, both models are trained on the same images. While MODE improves the model’s overall accuracy from 83\% to 89\% (+6\%), HUDD improves the model’s overall accuracy from 84\% to 97\% (+13\%).

 Apricot~\cite{Zhang19} repairs DNNs by changing the weights of the DNN model; however, an implementation of Apricot is not available.
HUDD is the first tool for the automated debugging of DNNs that is available for reuse.

\section{The \emph{HUDD}\xspace methodology}
\label{sec:overview}

 \begin{figure}[tb]
 \includegraphics[width=\columnwidth]{images/Approach}
 \caption{Overview of the \emph{HUDD}\xspace methodology.}
 \label{fig:approach}
 \end{figure}

\emph{HUDD}\xspace works in seven steps, depicted in Figure~\ref{fig:approach}.

In \emph{Step 1}, \emph{HUDD}\xspace takes as input the test set images leading to a DNN error (hereafter, error-inducing test set images). A DNN error might be either a wrong output label (for classifier DNNs) or an output loss higher than a given threshold (for regression DNNs).
Step 1 consists of three activities: (1) generate heatmaps for the error-inducing test set images, (2) compute a distance matrix with the distances between every pair of images\footnote{We rely on the Euclidean distance function applied to the heatmap of each image.}, and (3) execute hierarchical agglomerative clustering to group images based on the computed distances.
Step 1
leads to the identification of \emph{root cause clusters (RCCs)}, i.e., clusters of images with a common root cause for the observed DNN errors.

To generate heatmaps, HUDD relies on the Layer-Wise Relevance Propagation (LRP) algorithm~\cite{Montavon2019}.
LRP enables the generation of heatmaps for the internal layers of the DNN (\emph{internal heatmaps}).
An internal heatmap for a layer $k$ consists of a matrix with the relevance scores computed for all the neurons of that layer.

In \emph{Step 2}, engineers inspect the RCCs (typically a subset of the RCC images) to identify unsafe conditions, as required by functional safety analysis. Figure~\ref{fig:ClusteringResults} provides examples of RCCs generated for a DNN that detects the gaze of an eye (Gaze-DNN) and for a DNN that determines the position of a person's head (HPD-DNN).
Gaze-DNN processes images generated with a simulator; HPD-DNN processes real-world images.

The clusters in Figure~\ref{fig:ClusteringResults}
show that
\emph{HUDD}\xspace identifies root causes that are associated with: (1) \emph{borderline cases}
(e.g., the gaze and head pose angle detected by G3, C1, and C2),
(2) \emph{an incomplete training set} (e.g, persons with a face turned left and eyes looking right in C3),
(3) \emph{an incomplete definition of the predicted classes}
(i.e., the middle center gaze detected by cluster G4 and the closed eyes detected by cluster G2) and (4) \emph{limitations in our capacity to control the simulator} (i.e., unlikely face positions detected by cluster G1).
The first two cases are addressed by HUDD retraining procedures (i.e., Steps 4-7); also, borderline cases may help engineers identify countermeasures (e.g., an additional camera with a different camera angle), the identification of countermeasures being part of standard safety analysis practices. Finally, the other causes require that engineers modify the DNN (e.g., to add an output class) or improve the simulator.

In \emph{Step 3}, engineers rely on real-world data or simulation software
to generate a new set of images to retrain and improve the DNN. Step 3 is common practice and entails limited effort (e.g., acquiring field images or configuring a simulator).

In \emph{Step 4}, \emph{HUDD}\xspace automatically identifies the subset of images belonging to the improvement set that are likely to lead to DNN errors, referred to as  \emph{unsafe set}. It is obtained by assigning
the images of the improvement set to the RCCs according to their heatmap-based distance.
Since RCCs characterize only the portion of the input space that is unsafe, images belonging to the improvement set may not belong to any of these clusters.
For this reason, \emph{HUDD}\xspace selects only images that are sufficiently close to cluster members based on test set observations.

In \emph{Step 5}, engineers manually label the images belonging to the unsafe set. Different from traditional practice, which consists of labelling a large set of additional images and select for retraining the ones that lead to DNN errors,  \emph{HUDD}\xspace requires that engineers label only a small subset of the improvement set (i.e., the images that likely lead to DNN errors because they belong to the identified RCCs).

In \emph{Step 6}, to improve the accuracy of the DNN for every root cause observed,
independently from their frequency of appearance,
\emph{HUDD}\xspace balances the unsafe set with bootstrap resampling~\cite{ML:Book}, i.e., it randomly duplicates the images belonging to the cluster until every cluster has the same size.

In \emph{Step 7}, \emph{HUDD}\xspace retrains the DNN model by relying on a training set that consists of the union of the original training set and the balanced labeled unsafe set. HUDD uses the available model to set the initial configuration for the DNN weights. The original training set is retained to avoid reducing the accuracy of the DNN for parts of the input space that do not show any error in the test set.
The retraining process is expected to lead to an improved DNN model compared to that based on the original training set.

By driving the retraining based on the observed DNN errors, HUDD enables engineers to demonstrate that they took measures to improve safety, an important aspect to comply with regulations (e.g., ISO/PAS 21448 highlights the importance to adopt methods that reduce the unsafe portion of the input space~\cite{SOTIF}).

\begin{figure}[tb]
\includegraphics[width=8.4cm]{images/ClusteringResults.pdf}
\caption{Example of clustering results for two DNNs.}
\label{fig:ClusteringResults}
\end{figure}

Steps 2, 3, and 5 are manual steps which are also part of state-of-the-art solutions. But in the case of \emph{HUDD}\xspace, the manual effort required in such steps is much more limited than in existing approaches. With \emph{HUDD}\xspace, in Step 2, engineers inspect a few images per root cause clusters rather than the whole set of images resulting in significant cost savings and effective guidance towards the identification of root causes. Step 3 is common practice and entails limited effort such as buying field images or configuring a simulator. Finally, in Step 5, engineers label only a subset of the improvement set containing images that are likely to be unsafe and can be effectively used for retraining. Without \emph{HUDD}\xspace, engineers would have to label a randomly selected subset of the improvement set, which would contain fewer unsafe images and thus be less effective during retraining.

Software Engineering (cs.SE); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)


Testing DNN Image Classifiers for Confusion & Bias Errors

Abstract: Image classifiers are an important component of today's software, from consumer and business applications to safety-critical domains. The advent of Deep Neural Networks (DNNs) is the key catalyst behind such wide-spread success. However, wide adoption comes with serious concerns about the robustness of software systems dependent on DNNs for image classification, as several severe erroneous behaviors have been reported under sensitive and critical circumstances. We argue that developers need to rigorously test their software's image classifiers and delay deployment until acceptable. We present an approach to testing image classifier robustness based on class property violations.We found that many of the reported erroneous cases in popular DNN image classifiers occur because the trained models confuse one class with another or show biases towards some classes over others. These bugs usually violate some class properties of one or more of those classes. Most DNN testing techniques focus on per-image violations, so fail to detect class-level confusions or biases.We developed a testing technique to automatically detect class-based confusion and bias errors in DNN-driven image classification software. We evaluated our implementation, DeepInspect, on several popular image classifiers with precision up to 100% (avg.~72.6%) for confusion errors, and up to 84.3% (avg.~66.8%) for bias errors. DeepInspect found hundreds of classification mistakes in widely-used models, many exposing errors indicating confusion or bias.

\section{Introduction}
\label{sec:intro}

Image classification has a plethora of applications in software for safety-critical domains such as self-driving cars, medical diagnosis, \hbox{\emph{etc.}}\xspace Even day-to-day consumer software includes image classifiers, such as Google Photo search and Facebook image tagging. Image classification is a well-studied problem in computer vision, where a model is trained to classify an image into single or multiple predefined categories~\cite{kamavisdar2013survey}.
Deep Neural Networks (DNNs) have enabled major breakthroughs in image classification tasks over the past few years,
sometimes even matching human-level accuracy under some conditions~\cite{he2016deep}, which has led to their ubiquity in modern software.

However, in spite of such spectacular success, DNN-based image classification models, like traditional software, are known to have serious bugs.  For example, Google faced backlash in 2015 due to a notorious error in its photo-tagging app, which tagged pictures of dark-skinned people as ``gorillas''~\cite{google_tag}. Analogous to traditional software bugs, the Software Engineering (SE) literature denotes these classification errors as {\em model bugs}~\cite{Ma:2018:MAN:3236024.3236082}, which can arise due to either imperfect model structure or inadequate training data.

At a high-level, these bugs can affect either an {\em individual image}, where a particular image is mis-classified (\hbox{\emph{e.g.,}}\xspace a particular skier is mistaken as a part of a mountain),
or an {\em image class}, where a class of images is more likely to be mis-classified (\hbox{\emph{e.g.,}}\xspace dark-skinned people are more likely to be misclassified), as shown in Table~\ref{tab:example}. The latter bugs are specific to a whole {\em class} of images rather than individual images, implying systematic bugs rather than the DNN equivalent of off-by-one errors. While much effort from the SE literature on Neural Network testing has focused on identifying individual-level violations\textemdash using white-box~\cite{pei2017deepxplore,ma2018deepgauge, kim2019guiding, wang2019adversarial}, grey-box~\cite{tian2017deeptest,Ma:2018:MAN:3236024.3236082}, or concolic testing~\cite{sun2018concolic}, detection of class-level violations remains relatively less explored. This paper focuses on automatically detecting such class-level bugs, so they can be fixed.

\input{tables/example.tex}

After manual investigation of some public reports describing the class-level violations listed in Table~\ref{tab:example}, we determined two root causes: (i) \textbf{Confusion}: The model cannot differentiate one class from another. For example, Google Photos confuses skier and mountain~\cite{google_photo}.  (ii) \textbf{Bias}: The model shows disparate outcomes between two related groups. For example, Zhao \hbox{\emph{et al.}}\xspace in their paper ``Men also like  shopping''~\cite{zhao2017men}, find classification bias in favor of women on activities like shopping, cooking, washing, \hbox{\emph{etc.}}\xspace We further notice that some class-level properties are violated in both kinds of cases. For example, in the case of {\em confusion errors}, the classification error-rate between the objects of two classes, say, skier and mountain, is significantly higher than the overall classification error rate of the model. Similarly, in the bias scenario reported by Zhao \hbox{\emph{et al.}}\xspace, a DNN model should not have different error rates while classifying the gender of a person in the shopping category. Unlike individual image properties, this is a class property affecting all the shopping images with men or women. Any violation of such a property by definition affects the whole class although not necessarily every image in that class, \hbox{\emph{e.g.,}}\xspace a man is more prone to be predicted as a woman when he is shopping, even though some individual images of a man shopping may still be predicted correctly.  Thus, we need a class-level approach to testing image classifier software for confusion and bias errors.

The bugs in a DNN model occur due to sub-optimal interactions between the model structure and the training data~\cite{Ma:2018:MAN:3236024.3236082}.
To capture such interactions, the literature has proposed various metrics primarily based on either neuron activations~\cite{pei2017deepxplore, ma2018deepgauge, kim2019guiding} or feature vectors~\cite{Ma:2018:MAN:3236024.3236082, pmlr-v97-odena19a}.
However, these techniques are primarily targeted at the individual image level. To detect class-level violations, we abstract away such model-data interactions at the class level and analyze the inter-class interactions using that new abstraction. To this end, we propose a metric using neuron activations and a baseline metric using weight vectors of the feature embedding to capture the class abstraction.

For a set of test input images, we compute the probability of activation of a neuron per predicted class.
Thus, for each class, we create a vector of neuron activations where each vector element corresponds to a neuron activation probability. If the distance between the two vectors for two different classes is too close, compared to other class-vector pairs, that means the DNN under test may not effectively distinguish between those two classes.
Motivated by MODE’s technique~\cite{Ma:2018:MAN:3236024.3236082}, we further create a baseline where each class is represented by the corresponding weight vector of the last linear layer of the model under test.

We evaluate our methodology for both single- and multi-label classification models in eight different settings. Our experiments demonstrate that DeepInspect\xspace can efficiently detect both Bias and Confusion errors in popular neural image classifiers.

 We further check whether DeepInspect\xspace can detect such classification errors in state-of-the-art models designed to be robust against norm-bounded adversarial attacks~\cite{NIPS2018_8060}; DeepInspect\xspace finds hundreds of errors proving the need for orthogonal testing strategies to detect such class-level mispredictions.
Unlike some other DNN testing techniques~\cite{tian2017deeptest,pei2017deepxplore,pmlr-v97-odena19a,sun2018concolic}, DeepInspect\xspace does not need to generate additional transformed (synthetic) images to find these errors.
The primary contributions of this paper are:

\begin{itemize}[leftmargin=*]
    \item
    We propose a novel neuron-coverage metric to automatically detect class-level violations (confusion and bias errors) in DNN-based visual recognition models for image classification.
    \item
    We implemented our metric and underlying techniques in DeepInspect\xspace.
    \item
    We evaluated DeepInspect\xspace and found many errors in widely-used DNN models with precision up to 100\% (avg.~72.6\%) for confusion errors and up to 84.3\% (avg.~66.8\%) for bias errors.
\end{itemize}

Our code is available at \url{https://github.com/ARiSE-Lab/DeepInspect}.
The errors reported by DeepInspect\xspace are available at: \url{https://www.ariselab.info/deepinspect}.

\section{DNN Background}
\label{sec:background}

Deep Neural Networks (DNNs) are a popular type of machine learning model loosely inspired by the neural networks of human brains. A DNN model learns the logic to perform a software task from a large set of \emph{training examples}. For example, an image recognition model learns to recognize {\tt cows} through being shown (trained with) many sample images of cows.

A typical "feed-forward" DNN consists of a set of connected computational units, referred as {\em neurons}, that are arranged sequentially in a series of {\em layers}.
The neurons in sequential layers are connected to each other through \textit{edges}.
Each edge has a corresponding weight.
Each neuron applies $\sigma$, a \textit{nonlinear activation function} (\hbox{\emph{e.g.,}}\xspace ReLU~\cite{nair2010rectified}, Sigmoid~\cite{Mitchell:1997:ML:541177}), to the incoming values on its input edges and sends the results on its output edges to the next layer of connected neurons.
For image classification, convolutional neural networks (CNNs)~\cite{lecun1998gradient}, a specific type of DNN,  are typically used. CNNs consist of layers with local spatial connectivity and sets of neurons with shared parameters.

When implementing a DNN application, developers
typically start with a set of annotated experimental data\xspace and divide it into three sets:
(i) training: to construct the DNN model in a supervised setting, meaning the training data is labeled (\hbox{\emph{e.g.,}}\xspace using stochastic gradient descent with gradients computed using back-prop\-agation~\cite{rumelhart1988learning}); (ii) validation: to tune the model's hyper-parameters, basically configuration parameters that can be modified to better fit the expected application workload; and
(iii) evaluation: to evaluate the accuracy of the trained model \hbox{\emph{w.r.t.}}\xspace to its predictions on other annotated data, to determine whether or not it predicts correctly.
Typically, training, validation, and testing data are drawn from the same initial dataset.

For image classification, a DNN can be trained in either of the following two settings:

\noindent
(i)~\textbf{Single-label Classification.}
In a traditional single-label classification problem, each datum is associated with a single label \textit{l} from a set of disjoint labels $L$ where $|L| > 1$. If $|L| = 2$, the classification problem is called a binary classification problem; if $|L| > 2$, it is a multi-class classification problem~\cite{tsoumakas2007multi}. Among some popular image classification datasets, MNIST, CIFAR-10/CIFAR-100~\cite{cifar} and ImageNet~\cite{ILSVRC15} are all single-label,
where each image can be categorized into only one class or outside that class.

\noindent
(ii)~\textbf{Multi-label Classification.}
In a multi-label classification problem, each datum is associated with a set of labels \textit{Y} where $Y \subseteq L$.
COCO\cite{lin2014microsoft} and imSitu\cite{yatskar2016} are popular datasets for multi-label classification. For example, an image from the COCO~ dataset can be labeled as \textit{car, person, traffic light and bicycle}. A multi-label classification model is supposed to predict all of \textit{car, person, traffic light and bicycle} from a single image that shows all of these kinds of objects.

Given any single- or multi-label classification task, DNN classifier software tries to learn the decision boundary between the classes\textemdash all members of a class, say $C_i$, should be categorized identically irrespective of their individual features,
and members of another class, say $C_j$, should not be categorized to $C_i$~\cite{bengio2013representation}. The DNN represents the input image in an embedded space with the feature vector at a certain intermediate layer and uses the layers after as a classifier to classify these representations. The {\em class separation} between two classes estimates how well the DNN has learned to separate each class from the other.
If the embedded distance between two classes is too small compared to other classes, or lower than some pre-defined threshold, we assume that the DNN could not separate them from each other.

\section{Methodology}
\label{sec:method}

We give a detailed technical description of DeepInspect\xspace.
We describe a typical scenario where we envision our tool might be used in the following and design the methodology accordingly.

\smallskip
\noindent
\textbf{Usage Scenario.} Similar to customer testing of post-release software, DeepInspect\xspace works in a real-world
setting where a customer gets a pre-trained model and tests its performance in a sample production scenario before deployment. The customer has white-box access to the model to profile, although all the data in the production system can be {\em unlabeled}.
In the absence of ground truth labels, the classes are defined by the {\em predicted labels}.
These predicted labels are used as class references as DeepInspect\xspace tries to detect confusion and bias errors among the classes.
DeepInspect\xspace tracks the activated neurons per class and reports a potential class-level violation if the class-level activation-patterns are too similar between two classes.
Such reported errors will help customers evaluate how much they can trust the model's results related to the affected classes.
As elaborated in Section~\ref{sec:discussion}, once these errors are reported back to the developers, they can focus their debugging and fixing effort on these classes.
Figure \ref{fig:workflow} shows the DeepInspect\xspace workflow.

\subsection{Definitions}

Before we describe DeepInspect\xspace's methodology in detail, we introduce definitions that we use in the rest of the paper.
The following table shows our notation. \\
\begin{center}
{\footnotesize
\begin{tabular}{ll}
    \toprule
    All neurons set     & $ N = \{N_1,N_2,...\}$ \\

    Activation function & $ out(N, c)$ returns output  \\
         & for neuron $N$ , input $c$. \\
    Activation threshold & $Th$ \\
    \bottomrule
\end{tabular}
}
\end{center}
\ \\
\noindent
    \textit{Neural-Path ($NP$).}
For an input image $c$, we define {\em neural-path} as a sequence of neurons that are activated by $c$.

\noindent
\textit{Neural-Path per Class ($NP_{C}$).}
For a class $C_i$, this metric represents a set consisting of the union of neural-paths activated by all the inputs in $C_i$.

For example, consider a class {\tt cow} containing two images: a brown cow and a black cow. Let's assume they activate two neural-paths: $[N_1,N_2,N_3]$ and $[N_4,N_5,N_3]$.
Thus, the neural-paths for class {\tt cow} would be
$NP_{cow} = \{[N_1,N_2,N_3],[N_4,N_5,N_3]\}$.
$NP_{cow}$ is further represented by a vector
$(N_1^1,N_2^1,N_3^2,N_4^1,N_5^1)$, where the superscripts represent the number of times each neuron is activated by $C_{cow}$.
Thus, each class $C_i$ in a dataset can be expressed with a {\em neuron activation frequency vector}, which captures how the model interacts with  $C_i$.

\smallskip
\noindent
\textbf{\em Neuron Activation Probability}:
Leveraging how {\em frequently} a neuron $N_j$ is activated by all the members from a class $C_i$, this metric estimates the probability of a neuron $N_j$ to be activated by $C_i$. Thus,
we define: $\displaystyle P(N_j ~|~ C_i) =
    \frac{|\{ c_{ik}~|~\forall{c_{ik}} \in C_i,out(N_j,c_{ik}) > Th\}|}{|C_i|}$

\medskip
\noindent
We then construct a $n\times m$ dimensional {\em neuron activation  probability matrix}, $\rho$, ($n$ is the number of neurons and $m$ is the number of classes) with its ij-th entry being $P(N_j ~|~ C_i)$.

{
\setlength{\abovedisplayskip}{0pt}
\setlength{\belowdisplayskip}{0pt}
\vspace{-0.3cm}
\begin{equation}
\scriptstyle
\label{eq:napm}
  \rho=
\begin{blockarray}{cccccc}
 & C_1 & ... & C_i & ... & C_m \\
\begin{block}{c(ccccc)}
  N_1 & p_{11} &   &   &   &  p_{1m} \\
  ... & ...    &   &   &   &  \\
  N_j & p_{j1} &   &...&   & p_{jm} \\
  ... & ...    &   &   &   &  \\
  N_n & p_{n1} &   &   &   &  p_{nm}\\
\end{block}
\end{blockarray}
\end{equation}
\vspace{-0.3cm}
}

This matrix  captures how a model interacts with a set of input data.
The column vectors ($\rho_{\alpha m}$) represent
 the interaction of a class $C_m$ with the model.
Note that, in our setting, $C$s are predicted labels.

Since Neuron Activation Probability Matrix\xspace ($\rho$) is designed to represent each class, it should be able to distinguish between different $C$s.
Next, we use this metric to find two different classes of errors
often found in DNN systems: {\em confusion} and {\em bias} (see~\Cref{tab:example}).

\begin{figure
\centering
\includegraphics[width=1\columnwidth]{figures/workflow-eps-converted-to.pdf}
\caption{\textbf{\small DeepInspect\xspace Workflow
}
\label{fig:workflow}
\end{figure}

\subsection{Finding Confusion Errors}
\label{sec:conf}

In an object classification task, when the model cannot distinguish one object class from another, confusion occurs. For example, as shown in~\Cref{tab:example}, a Google photo app model confuses a skier with the mountain.
Thus, finding confusion errors means checking how well the model can distinguish between objects of different classes. An error happens when the model under test classifies an object with a wrong class, or for multi-label classification task, predicts two classes but only one of them is present in the test image.

We argue that the model makes these errors because during the training process the model has not learned to distinguish well between the two classes, say $a$ and $b$. Therefore, the neurons activated by these objects are similar and the column vectors corresponding to these classes: $\rho_{\alpha a}$  and $\rho_{\alpha b}$
will be very close to each other. Thus, we compute the confusion score between two classes
as the euclidean distance between their two probability vectors:

{
\setlength{\abovedisplayskip}{0pt}
\setlength{\belowdisplayskip}{0pt}
\begin{equation}
\label{eq:conf}
\begin{split}
\scriptstyle
    \textsc{napvd}\xspace(a,b) = \Delta(a, b) = ||\rho_{\alpha a} - \rho_{\alpha b}||_2
\displaystyle
     = \sqrt{\sum_{i = 1}^{n} \left( P(N_i|a) - P(N_i|b)\right) ^{2}}
\end{split}
\end{equation}
}

If the $\Delta$ value is less than some pre-defined threshold (conf\_th) for two pairs of classes, the model will potentially make mistakes in distinguishing one from another, which results in confusion errors.
This $\Delta$ is  called \textsc{napvd}\xspace  (\underline{N}euron \underline{A}ctivation \underline{P}robabiliy \underline{V}ector \underline{D}istance).

\subsection{Finding Bias Errors}
\label{sec:bias}

In an object classification task, bias occurs if the model under test shows disparate outcomes between two related classes. For example,
we find that ResNet-34 pretrained by imSitu dataset, often mis-classifies a man with a baby as {\tt woman}. We observe that in the embedded matrix $\rho$, $\Delta(baby,woman)$
is much smaller than $\Delta(baby,man)$.
Therefore, during testing, whenever the model finds an image with a baby, it is biased towards associating the baby image with a woman. Based on this observation, we propose an inter-class distance based metric to calculate the bias learned by the model. We define the bias between two classes $a$ and $b$  over a third class $c$ as follows:
\begin{equation}
    \label{eq:bias}
  \displaystyle
    bias(a, b, c) := \frac{|\Delta(c, a) - \Delta(c, b)|}{\Delta(c, a) + \Delta(c, b)}
\end{equation}

 If a model treats objects of classes $a$ and $b$ similarly under the presence of a third object class $c$, $a$ and $b$ should have similar distance \hbox{\emph{w.r.t.}}\xspace $c$ in the embedded space $\rho$; thus, the numerator of the above equation will be small. Intuitively, the model's output can be more influenced by the nearer object classes, \hbox{\emph{i.e.}}\xspace if $a$ and $b$ are closer to $c$. Thus, we normalize the disparity between the two distances to increase the influence of closer classes.

 This bias score is used to measure how differently the given model treats two classes in the presence of a third object class.
 An \textbf{average bias} (abbreviated as \textrm{avg\_bias}\xspace) between two objects $a$ and $b$ for all class objects $O$ is defined  as:
\begin{equation}
\label{eq:abgbias}

avg\_bias(a, b) := \frac{1}{|O|-2} \sum_{c\in O, c\neq a,b} bias (a, b, c)
\end{equation}
The above score captures the overall bias of the model between two classes. If the bias score is larger than some pre-defined threshold, we report potential {\em bias errors}.

Note that, even when the two classes $a$ and $b$ are not confused by the model, \hbox{\emph{i.e.}}\xspace $\Delta(a,b) > conf\_th$,  they can still show bias \hbox{\emph{w.r.t.}}\xspace another class, say $c$, if  $\Delta(a,c)$ is very different from $\Delta(b,c)$. Thus, bias and confusion are two separate types of class-level errors that we intend to study in this work.

Using these above equations we develop a novel testing tool, DeepInspect\xspace, to inspect a DNN implementing image classification tasks and look for potential confusion and bias errors. We implemented DeepInspect\xspace in the Pytorch deep learning framework and Python 2.7. All our experiments were run on Ubuntu 18.04.2 with two TITAN Xp GPUs. For all of our experiments, we set the activation threshold $Th$ to be 0.5 for all datasets and models. We discuss why we choose 0.5 as neuron activation threshold and how different thresholds affect our performance in the section \ref{sec:discussion}.

\section{Experimental Design}
\label{sec:experiment}

\subsection{Study Subjects}
\label{sec:subj}

We apply DeepInspect\xspace for both multi-label and single-label DNN-based classifications. Under different settings, DeepInspect\xspace automatically inspects 8 DNN models for 6 datasets.
Table~\ref{tab:subj} summarizes our study subjects. All the models we used are standard, widely-used models for each dataset.
We used pre-trained models as shown in the Table for all settings except for COCO~ with gender. For COCO~ with gender model, we used the gender labels from \cite{zhao2017men} and trained the model in the same way as \cite{zhao2017men}. imSitu model is a pre-trained ResNet-34 model \cite{yatskar2016}. There are in total 11,538 entities and 1,788 roles in the imSitu dataset. When inspecting a model trained using imSitu, we only considered the top 100 frequent entities or roles in the test dataset.

\input{tables/subject.tex}

Among the 8 DNN models, three  are pre-trained relatively more robust models that are trained using adversarial images along with regular images. These models are pre-trained by provably robust training approach proposed by~\cite{NIPS2018_8060}. Three models with different network structures are trained using the CIFAR10 dataset~\cite{NIPS2018_8060}.

\subsection{Constructing Ground Truth (GT) Errors}
\label{sec:gt}

To collect the ground truth for evaluating DeepInspect\xspace,
 we refer to the test images misclassified by a given model. We then aggregate these misclassified image instances by their real and predicted class-labels and estimate pair-wise confusion/bias.

\subsubsection{GT of Confusion Errors}
\label{sec:gt_conf}
Confusion occurs when a DNN often makes mistakes in disambiguating
members of two different classes. In particular, if a DNN is confused between two classes, the classification error rate is higher between those two classes than between the rest of the class-pairs. Based on this, we define two types of confusion errors for single-label classification and multi-label classification separately:

\textit{Type1 confusions\xspace}: In single-label classification, Type1 confusion occurs when an object of class $x$ (\hbox{\emph{e.g.,}}\xspace {\tt violin}) is misclassified to another class $y$ (\hbox{\emph{e.g.,}}\xspace {\tt cello}).
For all the objects of class $x$ and $y$, it can be quantified as: $\typeaconf(x, y) = \mean(\prob(x| y), \prob(y| x))$ \textemdash DNN's probability to misclassify class $y$ as $x$ and vice-versa, and takes the average value between the two. For example, given two classes {\tt cello} and {\tt violin}, $\typeaconf$ estimates the mean probability of {\tt violin} misclassified to {\tt cello} and vice versa. Note that, this is a bi-directional score, \hbox{\emph{i.e.}}\xspace misclassification of $y$ as $x$ is the same as misclassification of $x$ as $y$.

\textit{Type2 confusions\xspace}: In multi-label classification, Type2 confusion occurs when an input image contains an object of class $x$ (\hbox{\emph{e.g.,}}\xspace  {\tt mouse}) and no object of class $y$ (\hbox{\emph{e.g.,}}\xspace {\tt keyboard}), but the model predicts both classes (see~\Cref{fig:coco_confusion_bugs}. For a pair of classes, this can be quantified as: $\typebconf(x, y) = \mean(\prob(( x, y)| x), \prob(( x, y)| y))$ to compute the probability to detect two objects in the presence of only one. For example, given two classes {\tt keyboard} and {\tt mouse}, $\typebconf$ estimates the mean probability of {\tt mouse} being predicted while predicting {\tt keyboard} and vice versa.
This is also a bi-directional score.

We measure $\typeaconf$ and $\typebconf$ by using a DNN's {\em true classification error} measured on a set of test images.
They create the DNN's true confusion characteristics between all possible class-pairs. We then draw the distributions of $\typeaconf$ and $\typebconf$. For example, ~\Cref{fig:confusion_dist} shows $\typebconf$ distribution for COCO~. The class-pairs with confusion scores greater than $1$ standard deviation from the mean-value are marked as pairs truly confused by the model and form our ground truth for confusion errors. For example, in the COCO~ dataset, there are 80 classes and thus 3160 class pairs (80*79/2); 178 class-pairs are ground-truth confusion errors.

\begin{figure
\centering
    \begin{subfigure}[b]{0.23\textwidth}
       \includegraphics[width=\linewidth]{figures/type2_distribution_coco_2-eps-converted-to.pdf}

        \caption{Confusions distribution}
        \label{fig:confusion_dist}
    \end{subfigure}
     ~

     \begin{subfigure}[b]{0.23\textwidth}
        \includegraphics[width=\linewidth]{figures/distance_coco_2-eps-converted-to.pdf}
        \caption{NAPVD distribution}
        \label{fig:distance_dist}
    \end{subfigure}

\caption{\small{\textbf{Identifying Type2 confusions\xspace for multi-classification applications. LHS shows how we marked the ground truth errors based on Type2 confusion score. RHS shows DeepInspect\xspace's predicted errors based on NAPVD score.}}
}
\label{fig:confusion_distribution}
\end{figure}

Note that, unlike how a bug/error is defined in traditional software engineering, our suspicious confusion pairs have an inherent probabilistic nature. For example, even if $a$ and $b$ represent a confusion pair, it does not mean that all the images containing $a$ or $b$ will be misclassified by the model. Rather, it means that compared with other pairs, images containing $a$ or $b$ tend to have a higher chance to be misclassified by the model.

\subsubsection{GT of Bias Errors}
\label{sec:gt_bias}
A DNN model is {\em biased} if it treats two classes differently.
For example, consider three classes: {\tt man}, {\tt woman}, and {\tt surfboard}. An unbiased model should not have different error rates while classifying {\tt man} or {\tt woman} in the presence of {\tt surfboard}.
To measure such bias formally, we define \textbf{confusion disparity} ($\textrm{cd}\xspace$) to measure differences in error rate between classes $x$ and $z$ and between $y$ and $z$: $\textrm{cd}\xspace(x, y, z) = |error(x, z) - error(y, z)|$, where the error measure can be either $\typeaconf$ or $\typebconf$ as defined earlier. $\textrm{cd}\xspace$ essentially estimates the disparity of the model's error between classes $x$, $y$ (\hbox{\emph{e.g.,}}\xspace~{\tt man}, {\tt woman}) \hbox{\emph{w.r.t.}}\xspace a third class $z$ (\hbox{\emph{e.g.,}}\xspace~{\tt surfboard}).

We also define an aggregated measure \textbf{average confusion disparity} (\textbf{avg\_cd}) between two classes $x$ and $y$ by summing up the bias between them over all third classes and taking the average:
\[
        \textrm{avg\_cd}\xspace(x, y) := \frac{1}{|O|-2} \sum_{z \in O, z \neq x, y} \textrm{cd}\xspace(x, y, z).
\]
Depending on the error types we used to estimate \textrm{avg\_cd}\xspace, we refer to $Type1$\_\textrm{avg\_cd}\xspace and $Type2$\_\textrm{avg\_cd}\xspace.
We measure \textrm{avg\_cd}\xspace using the true classification error rate reported for the test images. Similar to confusion errors, we draw the distribution of \textrm{avg\_cd}\xspace for all possible class pairs and then consider the pairs as {\em truly biased} if their \textrm{avg\_cd}\xspace score is higher than one standard deviation from the mean value. Such truly biased pairs form our ground truth for bias errors.

\subsection{Evaluating DeepInspect\xspace}
\label{sec:eval_metric}

We evaluate DeepInspect\xspace using a set of test images.

\noindent
\textbf{Error Reporting.}
DeepInspect\xspace reports confusion errors based on NAPVD (see~\Cref{eq:conf}) scores\textemdash lower NAPVD indicates errors.
We draw the distributions of NAPVDs for all possible class pairs,
as shown in~\Cref{fig:distance_dist}.
Class pairs having NAPVD scores lower than $1$ standard deviation from the mean score are marked as potential confusion errors.

As discussed in~\Cref{sec:bias}, DeepInspect\xspace reports bias errors based on \textrm{avg\_bias}\xspace score (see~\Cref{eq:abgbias}), where higher \textrm{avg\_bias}\xspace means class pairs are more prone to bias errors.
Similar to above, from the distribution of \textrm{avg\_bias}\xspace scores, DeepInspect\xspace predicts pairs with \textrm{avg\_bias}\xspace greater than $1$ \textrm{standard deviation} from the mean score to be erroneous.
Note that, while calculating error disparity between classes $a$, $b$ \hbox{\emph{w.r.t.}}\xspace $c$ (see \Cref{eq:bias}), if both $a$ and $b$ are far from $c$ in the embedded space $\rho$, disparity of their distances ($\Delta$)  should not reflect true bias. Thus, while calculating $\textrm{avg\_bias}\xspace(a,b)$ we further filter out the triplets where
$\Delta (c,a)>th \land \Delta (c,b)>th$, where $th$ is some pre-defined threshold. In our experiment, we remove all the class-pairs having $\Delta$ larger than $1$ standard deviation (\hbox{\emph{i.e.}}\xspace $th$) from the mean value of all $Delta$s across all the class-pairs.

\noindent
\textbf{Evaluation Metric.}\label{sec:eval_metric}
We evaluate DeepInspect\xspace in two ways:

\noindent
\textbf{Precision \& Recall.}
We use precision and recall to measure DeepInspect\xspace's accuracy.
For each error type t, suppose that E is the number of errors detected
by DeepInspect\xspace and A is the the number of true errors in the ground truth
set. Then the precision and recall of DeepInspect\xspace are $\frac{|A\cap E|}{|E|}$ and $\frac{|A\cap E|}{|A|}$ respectively.

\noindent
\textbf{Area Under Cost Effective Curve (AUCEC).}
Similarly to how static analysis warnings are ranked based on their priority levels~\cite{rahman2013and}, we also rank the erroneous class-pairs identified by DeepInspect\xspace based on the decreasing order of error proneness, \hbox{\emph{i.e.}}\xspace most error-prone pairs will be at the top.
To evaluate the ranking we use a cost-effectiveness measure~\cite{Arisholm2010Systematic}, AUCEC (Area Under the Cost-Effectiveness Curve), which has become standard to evaluate rank-based bug-prediction systems~\cite{rahman2013sample,kamei2010revisiting,rahman2011bugcache, rahman2013and, ray2016naturalness}.

Cost-effectiveness evaluates when we inspect/test top n\% class-pairs in the ranked list (\hbox{\emph{i.e.}}\xspace inspection cost), how many true errors are found (\hbox{\emph{i.e.}}\xspace effectiveness).
Both cost and effectiveness are normalized to 100\%.
~\Cref{fig:confusion-ce} shows cost on the x-axis, and effectiveness on the y-axis,  indicating the portion of the ground truth errors found. AUCEC is the area under this curve.

\noindent
\textbf{Baseline.}
\label{sec:baseline}
We compare DeepInspect\xspace \hbox{\emph{w.r.t.}}\xspace two baselines:

\noindent
(i) MODE-inspired: A popular way to inspect each image is to inspect a  feature vector, which is an output of an intermediate layer ~\cite{Ma:2018:MAN:3236024.3236082,zhang2018deeproad}.
However, abstracting a feature vector per image to the class level is non-trivial.
Instead, for a given layer, one could inspect the weight vector ($w_l = [w^0_l, w^1_l, ..., w^n_l]$) of a class, say $l$, where the superscripts represent a feature. Similar weight-vectors are used in MODE~\cite{Ma:2018:MAN:3236024.3236082} to compare the difference in feature importance between two image groups.
In particular, from the last linear layer before the output layer we extract such per-class weight vectors and compute the pairwise distances between the weight vectors. Using these pairwise distances we calculate  confusion and bias metrics as described in Section~\ref{sec:method}.

\noindent
(ii) Random: We also build a random model that picks random class-pairs for inspection~\cite{witten2005data} as a baseline.

For AUCEC evaluation, we further show the performance of an optimal model that ranks the class-pairs perfectly\textemdash if $n$\% of all the class-pairs are truly erroneous, the optimal model would rank them at the top such that with lower inspection budget most of the errors will be detected. The optimal curve gives the lower upper bound of the ranking scheme.

\noindent
\textbf{Research Questions.}
With this experimental setting, we investigate the following three research questions to evaluate DeepInspect\xspace for DNN image classifiers:

\begin{itemize}[leftmargin=*]
    \item
    \textbf{RQ1.}
    Can DeepInspect\xspace distinguish between different classes?
    \item
    \textbf{RQ2.}~Can DeepInspect\xspace identify the confusion errors?
    \item
    \textbf{RQ3.}~Can DeepInspect\xspace identify the bias errors?
\end{itemize}

\section{Results}
\label{sec:result}

We begin our investigation by checking whether de-facto neuron coverage-based metrics can capture class separation.

\smallskip
\RQ{1}{Can \tool distinguish between different classes?~}
\label{sec:rqa}

\noindent
\textbf{Motivation.} The heart of DeepInspect\xspace's error detection technique lies in the fact that the underlying Neuron Activation Probability metric ($\rho$) captures each class abstraction reasonably well and thus  distinguishes between classes that do not suffer from class-level violations. In this RQ we check whether this is indeed true. We also check whether a new metric $\rho$ is necessary, \hbox{\emph{i.e.}}\xspace, whether existing neuron-coverage metrics could capture such class separations.

\noindent
\textbf{Approach.} We evaluate this RQ \hbox{\emph{w.r.t.}}\xspace the training data since the DNN behaviors are not tainted with inaccuracies associated with the test images. Thus, all the class-pairs are benign. We evaluate this RQ in three settings: (i) using DeepInspect\xspace's metrics, (ii) neuron-coverage proposed by  Pei~\hbox{\emph{et al.}}\xspace~\cite{pei2017deepxplore}, and (iii) other neuron-activation related metrics proposed by DeepGauge~\cite{ma2018deepgauge}.

\noindent
\textbf{Setting-1. DeepInspect\xspace.}
Our metric, Neuron Activation Probability Matrix\xspace ($\rho$), by construction is designed per class.
Hence it would be unfair to directly measure its capability to distinguish between different classes. Thus, we pose this question in slightly a different way, as described below.
For multi-label classification, each image contains multiple class-labels.
For example, an image might have labels for both {\tt mouse} and {\tt keyboard}.
Such coincidence of labels may create confusion\textemdash if two labels always appear together in the ground truth set, no classifier can distinguish between them.
To check how many times two labels coincide,
we define a coincidence score between two labels $L_a$ and $L_b$ as:
$
\displaystyle
coincidence\left(L_a, L_b\right) =
mean(P\left(L_a, L_b|L_a\right),P\left(L_a, L_b|L_b\right))
$.

The above formula computes the minimum probability of labels $L_a$ and $L_b$ occurring together in an image given that one of them is present.
Note that this is a bi-directional score, \hbox{\emph{i.e.}}\xspace we treat the two labels similarly.
The $mean$ operation ensures we detect the least coincidence in either direction. A low value of coincidence score indicates two class-labels are easy to separate and vice versa.

Now, to check DeepInspect\xspace's capability to capture class separation, we simply check the correlation between
coincidence score and confusion score (\textsc{napvd}\xspace)
from Equation~\ref{eq:conf} for all possible class-label pairs.
Since only multi-label objects can have label coincidences, we perform this experiment for a pre-trained ResNet-50 model on the COCO~ multi-label classification task.

A Spearman correlation coefficient between the confusion and coincidence scores reaches a value as high as 0.96, showing strong statistical significance. The result indicates that DeepInspect\xspace can disambiguate most of the classes that have a low confusion scores.

Interestingly, we found some pairs where coincidence score is high, but DeepInspect\xspace was able to isolate them. For example, ({\tt cup,chair}), ({\tt toilet,sink}), \hbox{\emph{etc.}}\xspace. Manually investigating such cases reveals that although these pairs often appear together in the input images, there are also enough instances when they appear by themselves. Thus, DeepInspect\xspace disambiguates between these classes and puts them apart in the embedded space $\rho$.
These results indicate DeepInspect\xspace can also learn some hidden patterns from the context and, thus,
can go beyond inspecting the training data coincidence for evaluating model bias/confusion, which is the de facto technique among machine learning researchers~\cite{zhao2017men}.

\begin{figure}[!phtb]
\centering
\includegraphics[width=0.8\columnwidth]{figures/coco_rq1.pdf}
\caption{\textbf{\small Distribution of neuron coverage per class label, for 10 randomly picked class labels,
from the COCO~ dataset.
}}
\label{fig:coverage}
\end{figure}

Next, we investigate whether popular white-box metrics can distinguish between different classes.

\noindent
\textbf{Setting-2. Neuron Coverage ($NC$)}~\cite{pei2017deepxplore}
computes the ratio of the union of neurons activated by an input set and the total number of neurons in a DNN.
Here we compute $NC$ per class-label, \hbox{\emph{i.e.}}\xspace for a given class-label, we measure the number of neurons activated by the images tagged with that label \hbox{\emph{w.r.t.}}\xspace to the total neurons. The activation threshold we use is 0.5.
We perform this experiment on COCO~ and CIFAR-100~ to study multi- and single-label classifications.
Figure~\ref{fig:coverage} shows results for COCO~. We observe similar results for CIFAR-100~.

Each boxplot in the figure shows the distribution of neuron coverage per class-label across all the relevant images.
These boxplots visually show that {\em different labels} have very {\em similar $NC$} distribution.
We further compare these distributions using \mbox{Kruskal Test}~\cite{kruskaltest}, which is a non-parametric way of comparing more than two groups. Note that we choose a non-parametric measure as $NC$s may not follow normal distributions. (Kruskal Test is a parametric equivalent of the one-way analysis of variance (ANOVA).) The result reports a $p-value << 0.05$, \hbox{\emph{i.e.}}\xspace some differences exist across these distributions.
However, a pairwise Cohend's effect size for each class-label pair, as shown in the following table, shows
more than 56\% and 78\% class-pairs for CIFAR-100~ and COCO~ have small to negligible effect size.
This means neuron coverage cannot reliably distinguish a majority of the class-labels.

\begin{center}
{\scriptsize
    \begin{tabular}{l|rrrr}
    \toprule
         \multicolumn{5}{c}{  Effect Size of neuron coverage across different classes }  \\
    \toprule

    Exp Setting & negligible & {small} & {medium} & {large} \\
    \midrule
    COCO~ & 40.51\% & 38.19\% & 16.96\% & 4.34\% \\
    CIFAR-100~ & 31.94\% & 25.69\% & 23.87\% & 18.48\% \\
    \bottomrule
    \end{tabular}%
}
\end{center}

\noindent
\textbf{Setting-3.~DeepGauge~\cite{ma2018deepgauge}.}
Ma \hbox{\emph{et al.}}\xspace~\cite{ma2018deepgauge} argue that each neuron has a primary region of operation; they identify this region by using a boundary condition $[{low},{high}]$ on its output during training time; outputs outside this region ($(-\infty, {low}) \cup ({high}, +\infty)$) are marked as corner cases. They therefore introduce multi-granular neuron and layer-level coverage criteria. For neuron coverage they propose: (i) {\em k-multisection coverage} to evaluate how thoroughly the primary region of a neuron is covered, (ii) {\em boundary coverage} to compute how many corner cases are covered, and (iii) {\em strong neuron activation coverage} to measure how many corner case regions are covered in (${high}, +\infty$) region. For layer-level coverage, they define (iv) {\em top-k neuron coverage} to identify the most active k-neurons for each layer, and (v) {\em top-k neuron pattern} for each test-case to find a sequence of neurons from the top-k most active neurons across each layer.

We investigate whether each of these metrics can distinguish between different classes by measuring the above metrics for individual input classes following Ma \hbox{\emph{et al.}}\xspace's methodology.
We first profiled every neuron upper- and lower-bound for each class using the training images containing that class-label.
Next, we computed per-class neuron coverage using test images containing that class; for k-multisection coverage we chose $k=100$ to scale up the analysis. It should be noted that we also tried $k=1000$ (which is used in the original DeepGauge paper) and observed similar results (not shown here).

\begin{figure}[!phtb]
\centering
\scriptsize
 \includegraphics[width=0.8\columnwidth]{figures/coco_rq1-3.pdf}

\caption{\textbf{\small{Histogram of DeepGauge~\cite{ma2018deepgauge} multi-granular coverage per class label for COCO~ dataset}}
}
\label{fig:dg_coco}
\label{fig:dg}
\end{figure}

For layer-level coverage, we directly used the input images containing each class, where we select $k=1$.

\Cref{fig:dg_coco} shows the results as a histogram of the above five coverage criteria for the COCO~ dataset. For all five coverage criteria, there are many class-labels that share similar coverage. For example, in COCO~, there are $52$ labels with k-multisection neuron coverage with values between $0.31$ and $0.32$.
Similarly, there are $40$ labels with 0 neuron boundary coverage.
Therefore, none of the five coverage criteria are an effective way to distinguish between  different equivalence classes. The same conclusion was drawn for the CIFAR-100~ dataset.

\RS{1}{DeepInspect\xspace can disambiguate classes better than previous coverage-based metrics for the image classification task.}

We now investigate DeepInspect\xspace's capability in detecting confusion and bias errors in DNN models.

\input{tables/precision-recall.tex}

\RQ{2}{Can \tool identify the confusion errors?~}
\label{sec:rq2}

\noindent
\textbf{Motivation.} To evaluate how well DeepInspect\xspace can detect class-level violations, in this RQ, we report DeepInspect\xspace's ability to detect the first type of violation, \hbox{\emph{i.e.}}\xspace, Type1/Type2 confusions \hbox{\emph{w.r.t.}}\xspace to ground truth confusion errors, as described in~\Cref{sec:gt_conf}.

\begin{figure}[h]
\centering
    \begin{subfigure}[b]{0.22\textwidth}
       \includegraphics[width=\linewidth]{figures/coco_correlation-eps-converted-to.pdf}

        \caption{COCO~ dataset + ResNet-50}
        \label{fig:type2a}
    \end{subfigure}
    ~
    \begin{subfigure}[b]{0.22\textwidth}
       \includegraphics[width=\linewidth]{figures/cifar10s_correlation-eps-converted-to.pdf}

        \caption{ Robust CIFAR-10 Small}
        \label{fig:type1a}
    \end{subfigure}

\caption{\textbf{\small{Strong negative Spearman correlation (-0.55 and -0.86) between   \textsc{napvd}\xspace and ground truth confusion scores.
}}}
\label{fig:rq2-corr}
\end{figure}

We first explore the correlation between \textsc{napvd}\xspace and ground truth Type1/Type2 confusion score. Strong correlation has been found for all 8 experimental settings. ~\Cref{fig:rq2-corr} gives examples on COCO and CIFAR-10. These results indicate that \textsc{napvd}\xspace can be used to detect confusion errors\textemdash lower \textsc{napvd}\xspace means more confusion.

\input{tables/rq2}
\noindent
\textbf{Approach.}
By default, DeepInspect\xspace reports all the class-pairs with \textsc{napvd}\xspace scores one standard deviation less than the mean \textsc{napvd}\xspace score as error-prone (See~\Cref{fig:distance_dist}). In this setting, as the result shown on ~\Cref{tab:confusion_accuracy}, DeepInspect\xspace reports errors at high recall under most settings. Specifically, on CIFAR-100 and robust CIFAR-10 ResNet, DeepInspect\xspace can report errors as high as 71.8\%, and 100\%, respectively.
DeepInspect\xspace has identified thousands of confusion errors.

If higher precision is wanted, a user can choose to inspect only a small set of confused pairs based on \textsc{napvd}\xspace. As also shown in~\Cref{tab:confusion_accuracy}, when only the top1\% confusion errors are reported, a much higher precision is achieved for all the datasets. In particular, DeepInspect\xspace identifies 31 and 39 confusion errors for the COCO model and the CIFAR-100 model with 100\% and 79.6\% precision, respectively. The trade-off between precision and recall can be found on the cost-effective curves shown on ~\Cref{fig:confusion-ce}, which show overall performance of DeepInspect\xspace at different inspection cutoffs. Overall, \hbox{\emph{w.r.t.}}\xspace a random baseline mode, DeepInspect\xspace is gaining AUCEC performance from $61.6\%$ to $85.7\%$; \hbox{\emph{w.r.t.}}\xspace a MODE baseline mode, DeepInspect\xspace is gaining AUCEC performance from $10.2\%$ to $28.2\%$.

\begin{figure}[!phtb]
\centering
\includegraphics[width=\linewidth]{figures/confusion_std.pdf}
\caption{\textbf{\small AUCEC plot of Type1/Type2 Confusion errors in three different settings. The \red{red} vertical line marks 1-standard deviation less from mean \textsc{napvd}\xspace score. DeepInspect\xspace marks all class-pairs with \textsc{napvd}\xspace scores less than the red mark as potential errors.}}
\label{fig:confusion-ce}
\end{figure}

Figure~\ref{fig:coco_confusion_bugs} and Figure~\ref{fig:imagenet_confusion_bugs} give some specific confusion errors found by DeepInspect\xspace in the COCO~ and the ImageNet settings. In particular, as shown in Figure~\ref{fig:confusionbugs1}, when there is only a keyboard but no mouse in the image, the COCO model reports both. Similarly, Figure~\ref{fig:confusionbugs4} shows confusion errors on (cello, violin). There are several cellos in this image, but the model predicts it to show a violin.

\begin{figure
\centering

    \begin{subfigure}[b]{0.40\linewidth}
        \includegraphics[width=\linewidth]{{"figures/COCO_val2014_000000061034_no_mouse"}.jpg}
        \caption{\small(keyboard,mouse)}
        \label{fig:confusionbugs1}
    \end{subfigure}
    ~
    \begin{subfigure}[b]{0.40\linewidth}
        \includegraphics[width=\linewidth]{{"figures/COCO_val2014_000000044294_oven-_microwave"}.jpg}
        \caption{\small (oven,microwave)}
        \label{fig:confusionbugs2}
    \end{subfigure}

\caption{\textbf{\small Confusion errors identified in COCO~ model. In each pair the second object is mistakenly identified by the model.}}
\label{fig:coco_confusion_bugs}
\vspace{-0.2cm}
\end{figure}

\begin{figure}[!htpb]
\centering
      \begin{subfigure}[b]{0.4\linewidth}
          \includegraphics[width=\linewidth]{{"figures/ILSVRC2012_val_00035600"}.JPEG}
          \caption{\small (cello, violin)}
          \label{fig:confusionbugs4}
      \end{subfigure}
      ~
    \begin{subfigure}[b]{0.4\linewidth}
        \includegraphics[width=\linewidth]{{"figures/ILSVRC2012_val_00018849"}.JPEG}
        \caption{\small (library, bookshop)}
        \label{fig:confusionbugs5}
    \end{subfigure}

\caption{\textbf{\small Confusion errors identified in the ImageNet model. For each pair, the second object is mistakenly identified by the model.}
}
\label{fig:imagenet_confusion_bugs}
\end{figure}

Across all three relatively more robust
CIFAR-10 models DeepInspect\xspace identifies (cat, dog), (bird, deer) and (automobile, truck) as buggy pairs, where one class is very likely to be mistakenly classified as the other class of the pair.
This indicates that these confusion errors are to be tied to the training data, so all the models trained on this dataset including the robust models may have these errors. These results further show that the confusion errors are orthogonal to the norm-based adversarial perturbations and we need a different technique to address them.

We also note that the performance of all methods degrades quite a bit on ImageNet. ImageNet is known to have a complex structure, and all the tasks, including image classification and robust image classification \cite{Xie_2019_CVPR} usually have inferior performance compared with simpler datasets like CIFAR-10 or CIFAR-100. Due to such inherent complexity, the class representation in the embedded space is less accurate, and thus the relative distance between two classes may not correctly reflect a model’s confusion level between two classes.

\RS{2}{DeepInspect\xspace can successfully find confusion errors with precision 21\% to 100\% at top1\% for both single- and multi-object classification tasks.
DeepInspect\xspace also finds confusion errors in robust models.}

\RQ{3}{Can \tool identify the bias errors?~}

\begin{figure}[!htpb]
\centering
\begin{subfigure}[b]{0.23\textwidth}
       \includegraphics[width=\linewidth]{figures/bias_figures/coco,predicted,type2.pdf}
        \caption{COCO~}
        \label{fig:bias_coco_predicted_type2}
    \end{subfigure}
\begin{subfigure}[b]{0.23\textwidth}
        \includegraphics[width=\linewidth]{figures/bias_figures/cifar100,predicted,type1.pdf}
        \caption{CIFAR-100}
        \label{fig:bias_coco_predicted_type1}
    \end{subfigure}

\caption{\textbf{\small Strong positive Spearman's correlation (0.76 and 0.62) exist between \textrm{avg\_cd}\xspace and \textrm{avg\_bias}\xspace while
detecting classification bias.}
}
\label{fig:bias_coco_predicted}
\end{figure}

\noindent
\textbf{Motivation.} To assess DeepInspect\xspace's ability to detect class-level violations, in this RQ, we report DeepInspect\xspace's performance in detecting  the second type of violation, \hbox{\emph{i.e.}}\xspace, Bias errors as described in~\Cref{sec:gt_bias}.

\noindent
\textbf{Approach.}
We evaluate this RQ by estimating a model's bias (\textrm{avg\_bias}\xspace) using~\Cref{eq:abgbias} \hbox{\emph{w.r.t.}}\xspace the ground truth (\textrm{avg\_cd}\xspace), computed as in~\Cref{sec:gt_bias}. We first explore the correlation between pairwise \textrm{avg\_cd}\xspace and our proposed pairwise \textrm{avg\_bias}\xspace;
~\Cref{fig:bias_coco_predicted} shows the results for COCO~ and CIFAR-10.
Similar trends
were found in the other datasets we studied.
The results show that a strong correlation exists between \textrm{avg\_cd}\xspace and \textrm{avg\_bias}\xspace.
In other words, our proposed \textrm{avg\_bias}\xspace is a good proxy for detecting confusion errors.

\input{tables/rq3}

As in RQ2, we also do a precision-recall analysis \hbox{\emph{w.r.t.}}\xspace finding the bias errors across all the datasets.
We analyze the precision and recall of DeepInspect\xspace when reporting bias errors at the cutoff Top1$\%$(\textrm{avg\_bias}\xspace) and mean(\textrm{avg\_bias}\xspace)+standard deviation(\textrm{avg\_bias}\xspace), respectively. The results are shown in Table \ref{tab:bias_cutoffs}. At cutoff Top1$\%$(\textrm{avg\_bias}\xspace), DeepInspect\xspace
detects suspicious pairs with precision as high as 75\% and 84\% for COCO~ and imSitu, respectively. At cutoff mean(\textrm{avg\_bias}\xspace)+standard deviation(\textrm{avg\_bias}\xspace), DeepInspect\xspace has high recall but lower precision: DeepInspect\xspace detects ground truth suspicious pairs with recall at 75.9\% and 71.8\% for COCO~ and imSitu.  DeepInspect\xspace can report 657(=249+408) total true bias bugs across the two models. DeepInspect\xspace outperforms the random baseline by a large margin at both cutoffs. As in the case of detecting confusion errors, there is a significant trade-off between precision and recall.
This can be customized based on user needs.
The cost-effectiveness analysis in ~\Cref{fig:bias-ce} shows the entire spectrum.

\begin{figure}[h]
\centering
\includegraphics[width=\linewidth]{figures/bias_figures/bias_top_1std.pdf}
\caption{\textbf{\small Bias errors detected \hbox{\emph{w.r.t.}}\xspace the ground truth of \textrm{avg\_cd}\xspace beyond one standard deviation from mean.}
}
\label{fig:bias-ce}
\end{figure}

As shown in Figure~\ref{fig:bias-ce}, DeepInspect\xspace outperforms the baseline by a large margin.
The AUCEC gains of DeepInspect\xspace are from $37.1\%$ to $76.1\%$ w.r.t. the random baseline and from $6.0\%$ to $41.9\%$ w.r.t. the MODE baseline across the 8 settings. DeepInspect\xspace's performance is close to the optimal curve under some settings, specifically the AUCEC gains of the optimal over DeepInspect\xspace are only 7.11\% and 7.95\% under the COCO~ and ImSitu settings, respectively.

Inspired by \cite{zhao2017men}, which shows bias exists between men and women in COCO~ for the gender image captioning task, we analyze the most biased third class $c$ for $a$ and $b$ being men and women.
As shown in Figure \ref{fig:fairness_coco_predicted}, we found that sports like skiing, snowboarding, and surfboarding are more closely associated with men and thus misleads the model to predict the women in the images as men.
Figure \ref{fig:fairness_imsitu_predicted} shows results on imSitu, where we found that the model tends to associate the class ``inside'' with women while associating the class ``outside'' with men.

\begin{figure}[!htpb]
\centering
    \begin{subfigure}[b]{0.30\linewidth}
        \includegraphics[width=\linewidth]{figures/bias_figures/COCO_val2014_000000389273}
    \end{subfigure}
    ~
    \begin{subfigure}[b]{0.30\linewidth}
        \includegraphics[width=\linewidth]{figures/bias_figures/COCO_val2014_000000493442}
    \end{subfigure}
    ~
    \begin{subfigure}[b]{0.30\linewidth}
       \includegraphics[width=\linewidth]{figures/bias_figures/COCO_val2014_000000525899}
    \end{subfigure}

\caption{\textbf{\small The model classifies the women in these pictures as men in the COCO~ dataset.}}
\label{fig:fairness_coco_predicted}
\end{figure}

We generalize the idea by choosing classes $a$ and $b$ to be any class-pair. We found that similar bias also exists in the single-label classification settings. For example, in ImageNet, one of the highest biases is between Eskimo\_dog and rapeseed \hbox{\emph{w.r.t.}}\xspace Siberian\_husky. The model tends to confuse the two dogs but not Eskimo\_dog and rapeseed. This makes sense since Eskimo\_dog and Siberian\_husk are both dogs so more easily misclassified by the model.

\begin{figure}[!htpb]
\centering
    \begin{subfigure}[b]{0.25\linewidth}
        \includegraphics[width=\linewidth]{figures/bias_figures/slouching_23.jpg}
    \end{subfigure}
    ~
    \begin{subfigure}[b]{0.25\linewidth}
        \includegraphics[width=\linewidth]{figures/bias_figures/sweeping_99.jpg}
    \end{subfigure}

\caption{\textbf{\small The model classifies the man in the first figure to be a woman and the woman in the second figure to be a man.}}
\label{fig:fairness_imsitu_predicted}
\end{figure}

One of the fairness violations of a DNN system can be drastic differences in accuracy across groups divided according to some sensitive feature(s). In black-box testing, the tester can get a number indicating the degree of fairness has been violated by feeding into the model a validation set. In contrast, DeepInspect\xspace provides a new angle to the fairness violations. The neuron distance difference between two classes $a$ and $b$ \hbox{\emph{w.r.t.}}\xspace a third class $c$ sheds light on why the model tends to be more likely to confuse between one of them and $c$ than the other. We leave a more comprehensive examination on interpreting bias/fairness violations for future work.

\RS{3}{DeepInspect\xspace can successfully find bias errors for both single- and multi-label classification tasks, and even for the robust models, from 52\% to 84\% precision at top1\%.}

\section{Related Work}
\label{sec:rel}

\textbf{Software Testing \& Verification of DNNs.}
Prior research proposed different white-box testing criteria based on neuron coverage~\cite{pei2017deepxplore,ma2018deepgauge,tian2017deeptest} and neuron-pair coverage~\cite{sun2018testing}.
Sun \hbox{\emph{et al.}}\xspace~\cite{sun2018concolic} presented a concolic testing approach for DNNs called DeepConcolic. They showed that their concolic testing approach can effectively increase coverage and find adversarial examples. Odena and Goodfellow proposed TensorFuzz\cite{pmlr-v97-odena19a}, which is a general tool that combines coverage-guided fuzzing with property-based testing to generate cases that violate a user-specified objective. It has applications like finding numerical errors in trained neural networks, exposing disagreements between neural networks and their quantized versions, surfacing broken loss functions in popular GitHub repositories, and making performance improvements to TensorFlow.
There are also efforts to verify DNNs~\cite{pei2017towards,katz2017reluplex,huang2017safety,wang2018formal} against adversarial attacks. However, most of the verification efforts are limited to small DNNs and pixel-level properties.
It is not obvious how to directly apply these techniques to detect class-level violations.

\noindent
{\bf Adversarial Deep Learning.}
DNNs are known to be vulnerable to well-crafted inputs called adversarial examples, where the discrepancies are imperceptible to a human but can easily make DNNs fail~\cite{yuan2017adversarial,lu2017no, raghunathan2018certified, papernot2016cleverhans, evtimov2017robust, goodfellow2014explaining, kos2017adversarial, narodytska2016simple, nguyen2015deep, papernot2017practical, papernot2016limitations, szegedy2013intriguing, huang2017adversarial, kurakin2016adversarial}.
Much work has been done to defend against adversarial attacks~\cite{bastani2016measuring, carlini2017towards, feinman2017detecting, grosse2017statistical, gu2014towards, metzen2017detecting, papernot2017extending, papernot2016distillation, shaham2015understanding, xu2017feature, zheng2016improving, he2017adversarial, metric_adv}.
Our methods have potential to identify adversarial inputs.
Moreover, adversarial examples are usually out of distribution data and not realistic, while we can find both out-distribution and in-distribution corner cases.
Further, we can identify a general weakness or bug rather than focusing on crafted attacks that often require a strong attacker model (\hbox{\emph{e.g.,}}\xspace the attacker adds noise to a stop sign image).

\noindent
{\bf Interpreting DNNs.}
There has been much research on model interpretability and visualization~\cite{lipton2016mythos, zhang2018visual, selvaraju2016grad,montavon2017methods,bau2017network,dong2017towards}. A comprehensive study is presented by Lipton ~\cite{lipton2016mythos}.
Dong \hbox{\emph{et al.}}\xspace ~\cite{dong2017towards} observed that instead of learning the semantic features of whole objects, neurons tend to react to different parts of the objects in a recurrent manner. Our probabilistic way of looking at neuron activation per class aims to capture holistic behavior of
an entire class instead of an individual object so diverse features of class members can be captured.
Closest to ours is by Papernot \hbox{\emph{et al.}}\xspace~\cite{papernot2018deep}, who used nearest training points to explain adversarial attacks.
In comparison, we analyze the DNN's dependencies on the entire training/testing data and represent it in Neuron Activation Probability Matrix\xspace. We can explain the DNN's bias and weaknesses by inspecting this matrix.

\noindent
\textbf{Evaluating Models' Bias/Fairness.}
Evaluating the bias and fairness of a system is important both from a theoretical and a practical perspective~\cite{luong2011k, ZemelICML, zafar2017fairness, Brun:2018:SF:3236024.3264838}.
Related studies first define a fairness criteria and then try to optimize the original objective while satisfying the fairness criteria ~\cite{Dwork:2011, Hardt2016, barocas-hardt-narayanan, DBLP:conf/fat/MenonW18, DBLP:conf/nips/DoniniOBSP18, DBLP:journals/corr/abs-1901-10837}. These properties are defined either at individual~\cite{Dwork:2011,Kusner,Kim} or group levels~\cite{Calders,Hardt2016,ZafarWWW}.
In this work, we propose a definition of a bias error for image classification closely related to fairness notions at group-level. Class membership can be regarded as the sensitive feature and the equality that we want to achieve is for the confusion levels of two groups w.r.t. any third group. We showed the potential of DeepInspect\xspace to detect such violations.

Galhotra \hbox{\emph{et al.}}\xspace~\cite{galhotra2017fairness} first applied the notion of software testing to evaluating software fairness.
They mutate the sensitive features of the inputs and check whether the output changes. One major problem with their proposed method, Themis, is that it assumes the model takes into account sensitive attribute(s) during training and inference. This assumption is not realistic since most existing fairness-aware models drop input-sensitive feature(s). Besides, Themis will not work on image classification, where the sensitive attribute (\hbox{\emph{e.g.,}}\xspace, gender, race) is a visual concept that cannot be flipped easily.
In our work, we use a white-box approach to measure the bias learned by the model during training. Our testing method does not require the model to take into account any sensitive feature(s). We propose a new fairness notion for the setting of multi-object classification, {\em average confusion disparity}, and a proxy, {\em average bias}, to measure for any deep learning model even when only unlabeled testing data is provided. In addition, our method tries to provide an explanation behind the discrimination. A complementary approach by Papernot \hbox{\emph{et al.}}\xspace ~\cite{papernot2018deep} shows such explainability behind model bias in a single classification setting.

\section{Discussion \& Threats to Validity}
\label{sec:discussion}

\textbf{Discussion.}
In the
literature, bug detection, debugging, and repair are usually three distinct tasks, and there is a large body of work investigating each
separately.
In this work, we focus on bug detection for image classifier software.
A natural follow-up of our work will be debugging and repair leveraging DeepInspect\xspace's bug detection. We present some preliminary results and thoughts.

A commonly used approach to improving (\hbox{\emph{i.e.}}\xspace fixing) image classifiers is active learning, which consists of  adding more labeled data by smartly choosing what to label next.
In our case, we can use \textsc{napvd}\xspace to identify the most confusing class pairs,
and then target those pairs by collecting additional  examples that contain individual objects from the confusing pairs.
We download 105 sample images from Google Images that contain isolated examples of these categories so that the model learns to disambiguate them.
We retrain the model from scratch using the original training data and these additional examples. Using this approach, we have some preliminary results on the COCO~ dataset. After retraining, we find that the $\typebconf$ of the top confused pairs reduces. For example, the $\typebconf$(baseball bat, baseball glove) is reduced from 0.23 to 0.16, and $\typebconf$(refrigerator, oven) is reduced from 0.14 to 0.10. Unlike traditional active learning approaches that encourage labeling additional examples near the current decision boundary of the classifier, our approach encourages the labeling of problematic examples based on confusion bugs.

Another potential direction to explore is to use DeepInspect\xspace in tandem with debugging \& repair tools for DNN models like MODE~\cite{Ma:2018:MAN:3236024.3236082}.
DeepInspect\xspace enables the user to focus debugging effort on the vulnerable classes even in the absence of labeled data.
For instance, once DeepInspect\xspace identifies the vulnerable class-pairs, one can use the GAN-based approach proposed in MODE to generate more training data from these class-pairs, apply MODE to identify the most vulnerable features in these pairs to select for retraining.

We have also explored how the neuron coverage threshold($th$) used in computing $NAPVD$ affects our performance in detecting confusion and bias errors. We studied one multi-label classification task COCO~~and one single-label classification task CIFAR-100. Table \ref{tab:confusion_coco_threshold}, \ref{tab:confusion_cifar100_threshold}, \ref{tab:bias_coco_threshold}, \ref{tab:bias_cifar100_threshold} show how our precision and recall change when using different neuron coverage thresholds ($th$). We observed that for CIFAR-100 and COCO~~that DeepInspect’s accuracies are overall stable at $0.4\leq th \leq 0.75$. With smaller th($<0.25$), too many neurons are activated pulling the per-class activation-probability-vectors closer to each other. In contrast, with higher th($>0.75$), important activation information gets lost. Thus, we select $th=0.5$ for all the other experiments to avoid either issue.
\input{tables/threshold.tex}

\medskip
\noindent
\textbf{Threats to Validity.} We only test DeepInspect\xspace on 6 datasets under 8 settings. We include both
single-class and multi-class as well as regular and robust models to address these threats as much as possible.

Another limitation is that DeepInspect\xspace needs to decide thresholds for both confusion errors and bias errors, and a threshold for discarding low-confusion triplets in the estimation of \textrm{avg\_bias}\xspace. Instead of choosing fixed threshold, we mitigate this threat by choosing thresholds that are one standard deviation from the corresponding mean values and, also, reporting performance at top1\%.

The task of accurately classifying any image
is notoriously difficult.
We simplify the problem by testing the DNN model only for the classes that it has seen during training.
For example, while training, if a DNN does not learn to differentiate between black vs. brown {\tt cow}s (\hbox{\emph{i.e.}}\xspace, all the cow images only have label cow and they are treated as belonging to the same class by the DNN), DeepInspect\xspace will not be able to test these sub-groups.

\section{Conclusion}
\label{sec:conclusion}
Our testing tool for DNN image classifiers, DeepInspect\xspace, automatically detects confusion and bias errors in classification models. We applied DeepInspect\xspace to six different popular image classification datasets and eight pretrained DNN models, including three so-called relatively more robust models.
We show that DeepInspect\xspace can successfully detect class-level violations for both single- and multi-label classification models with high precision.

\section{0pt}{8pt plus 4pt minus 2pt}{4pt plus 2pt minus 2pt}

\usepackage{moreverb}

\everymath{\displaystyle}

\newcommand{\rom}[1]{\uppercase\expandafter{\romannumeral #1\relax}}
\newcommand{DeepTest\xspace}{DeepTest\xspace}
\newcommand{\hbox{\emph{cf.}}\xspace}{\hbox{\emph{cf.}}\xspace}
\newcommand{\ldots [deletia] \ldots}{\ldots [deletia] \ldots}
\newcommand{\hbox{\emph{et al.}}\xspace}{\hbox{\emph{et al.}}\xspace}
\newcommand{\hbox{\emph{e.g.,}}\xspace}{\hbox{\emph{e.g.,}}\xspace}
\newcommand{\hbox{\emph{i.e.}}\xspace}{\hbox{\emph{i.e.}}\xspace}
\newcommand{\hbox{\emph{s.t.}}\xspace}{\hbox{\emph{s.t.}}\xspace}
\newcommand{\hbox{\emph{w.r.t.}}\xspace}{\hbox{\emph{w.r.t.}}\xspace}
\newcommand{\hbox{\emph{viz.}}\xspace}{\hbox{\emph{viz.}}\xspace}
\newcommand{\hbox{\emph{v.s.}}\xspace}{\hbox{\emph{v.s.}}\xspace}
\newcommand{\hbox{\emph{etc.}}\xspace}{\hbox{\emph{etc.}}\xspace}

\newcommand{\defref}[1]{Definition~\ref{#1}}

\newcommand{\varname}[1]{{\small\texttt{#1}}\xspace}
\newcommand{\ch}[1]{{\color{black} #1}\xspace}

\def$\superscript{\S}${$\superscript{\S}$}
\def$^{\dag}${$^{\dag}$}
\def\superscript{\ddag}{\superscript{\ddag}}
\def$^{\natural}${$^{\natural}$}

\definecolor{gray50}{gray}{.5}
\definecolor{gray40}{gray}{.6}
\definecolor{gray30}{gray}{.7}
\definecolor{gray20}{gray}{.8}
\definecolor{gray10}{gray}{.9}
\definecolor{gray05}{gray}{.95}

\newlength\Linewidth
\def\findlength{\setlength\Linewidth\linewidth
\addtolength\Linewidth{-4\fboxrule}
\addtolength\Linewidth{-3\fboxsep}
}
\newenvironment{examplebox}{\par\begingroup
   \setlength{\fboxsep}{5pt}\findlength
   \setbox0=\vbox\bgroup\noindent
   \hsize=0.95\linewidth
   \begin{minipage}{0.95\linewidth}\normalsize}
    {\end{minipage}\egroup
    \textcolor{gray20}{\fboxsep1.5pt\fbox
     {\fboxsep5pt\colorbox{gray05}{\normalcolor\box0}}}
    \endgroup\par\noindent
    \normalcolor\ignorespacesafterend}
\let\Examplebox\examplebox
\let\endExamplebox\endexamplebox

\newcounter{RQCounter}
\newcounter{RQACounter}

\newcommand{\RQ}[2]{%
\refstepcounter{RQCounter} \label{#1}

 \noindent
 \textbf{RQ\arabic{RQCounter}.~#2}
}

\newcommand{\RQA}[2]{%
\refstepcounter{RQACounter} \label{#1}
\vspace{0.1in} \noindent\textbf{RQ\arabic{RQACounter}.~#2 \vspace{0.05in}}

}

\newcommand{\RQBoxed}[2]{%
\refstepcounter{RQCounter} \label{#1}
\begin{framed}%
\filbreak
\vspace{0.001in}

	\noindent\textbf{RQ\arabic{RQCounter}: }%
#2\end{framed}
}

\newcommand{\RS}[2]{%
\begin{framed}%
\filbreak

\textbf{Result {\ref{#1}}:~}{\emph {#2}}%
\end{framed}
}

\definecolor{javared}{rgb}{0.6,0,0}
\definecolor{javagreen}{rgb}{0.25,0.5,0.35}
\definecolor{javapurple}{rgb}{0.5,0,0.35}
\definecolor{javadocblue}{rgb}{0.25,0.35,0.75}

\lstdefinestyle{customc}{
  belowcaptionskip=\baselineskip,
  breaklines=true,

  xleftmargin=\parindent,
  language=java,
  showstringspaces=false,
  basicstyle=\scriptsize\ttfamily,
  keywordstyle=\bfseries\color{javapurple},
  commentstyle=\itshape\blue,

}

\lstset{escapechar=@,style=customc}

\DeclareCaptionLabelFormat{andimage}{#1~#2  \&  \figurename~\thefigure}

\section{Research Methods}

\end{document}
\endinput

Software Engineering (cs.SE); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)