Towards a Robust Differentiable Architecture Search under Label Noise

Neural Architecture Search (NAS) is the game changer in designing robust neural architectures. Architectures designed by NAS outperform or compete with the best manual network designs in terms of accuracy, size, memory footprint and FLOPs. That said, previous studies focus on developing NAS algorithms for clean high quality data, a restrictive and somewhat unrealistic assumption. In this paper, focusing on the differentiable NAS algorithms, we show that vanilla NAS algorithms suffer from a performance loss if class labels are noisy. To combat this issue, we make use of the principle of information bottleneck as a regularizer. This leads us to develop a noise injecting operation that is included during the learning process, preventing the network from learning from noisy samples. Our empirical evaluations show that the noise injecting operation does not degrade the performance of the NAS algorithm if the data is indeed clean. In contrast, if the data is noisy, the architecture learned by our algorithm comfortably outperforms algorithms specifically equipped with sophisticated mechanisms to learn in the presence of label noise. In contrast to many algorithms designed to work in the presence of noisy labels, prior knowledge about the properties of the noise and its characteristics are not required for our algorithm.


Introduction
To avoid exhausting engineering, Neural Architecture Search (NAS) has emerged as a leading mechanism for automatic design and wiring of neural networks. NAS has been successfully moving forward with diverse approaches to achieve a robust automatic architecture search e.g., evolution-based NAS [11,30,31,37], optimizationbased NAS [10,23,24,34,43], and Reinforcement Learning (RL) based NAS [29,62,63]. Specifically, a gradient-based method with a continuous architecture space called Differentiable ARchiTecture Search (DARTS) has attracted significant attention in NAS because of a reduced cost and complexity of searching for high performance architectures.
In this paper, we go beyond merely learning with NAS under regular assumptions of clean labels.
Supervised learning with neural networks often leads to a performance degradation due to overfitting, especially in the presence of label noise which often emerges due to data corruption and/or human annotation errors especially prominent in large scale datasets. As a result, neural networks fail to generalize well to previously unseen data and achieve suboptimal classification results. Given the importance of such problems, existing state-of-the-art methods are specifically designed to deal with the data noise by correcting labels [42], employing dedicated loss functions [5,26,46,58], reweighting samples [17,33], selecting samples [15], and modeling a transition matrix [28,40,48]. However, designing robust neural networks that mitigate overfitting due to label noise is still unexplored. To this end, we propose a structural approach, that is a method that requires no explicit changes to neither the loss functions nor the input samples nor the final outputs (predictions) but the neural network structure. Moreover, our approach does not require specific assumptions on the amount or the type of label noise.
Our primary focus 1 is to investigate and advance a robust method to search an architecture in the presence of label noise. To this end, we aim to answer the following questions: • Motivational Curiosity. As handcrafted neural networks (e.g., Inception [38]) overfit to noisy labels [55] (even the pretext labeling in self-supervised UnNAS is imperfect), is the test performance of neural networks constructed by NAS also degraded by the label noise?
• Research Curiosity. Can we design an operator 2 that is robust to noisy labels and helps existing NAS methods (e.g., DARTS [23]) to perform well in the presence of label noise?
The answer to the first question is affirmative as shown in Fig. 1 Figure 1: (a) Testing accuracy of vanilla vs. our approach (CIFAR-10, clean labels vs. 50%-symmetric label noise). The histogram of found operators (five runs) of (b) the normal cell of vanilla DARTS under 50%-symmetric label noise, (c) the normal cell of DARTS with nConv (our noise injecting operator) searched on CIFAR-10 (50%-symmetric noisy labels), and (d) the normal cell of vanilla DARTS with clean labels. The normal cell of vanilla DARTS under label noise is constructed poorly due to the large number of parameterless operations. It is apparent the network thus looses the learning capacity in an attempt to prevent overfitting to the noise by selecting parameterless operators. In contrast, DARTS with our proposed operator mitigates such a poor cell design as highlighted by the larger number of convolutional operators being selected in place of a parameterless operator.
DARTS [23] suffers when subjected to noisy labels as shown in Fig. 1 (a). Furthermore, the architecture searched by vanilla DARTS under label noise results in a bad architecture as shown in Fig. 1 (b) while our approach shown in Fig. 1 (c) produces a better designed cell. To answer the second question, we analyze the task of learning deep neural networks under noisy labels using Information Bottleneck. As a result of this analysis, we introduce a variant of the convolutional operation by injecting noise into the pipeline. Upon learning the parameters of the noise from data, we will empirically show that robustness against label corruption can be attained. In essence, we will show later that the noisy convolution regularizes and limits the gradient of the noisy samples during training. In short, our key contributions are: i. We show that the search through noisy labels degrades the classification performance of standard NAS.
ii. We provide an information theoretic framework to tackle learning noisy labels. Our proposed noise injection operator performs implicit regularization during learning preventing overfitting to noisy labels.
iii. The proposed operator is included into the NAS search space with the goal of preventing overfitting during training with noisy labels. A noise injection on the input of the operator turns activations of hidden units into the so-called stochastic activations.
iv. We show experimentally that architectures emerging from the NAS search with our proposed noise injecting operator outperform under fewer parameters the state of the art, especially in the case of no prior knowledge given e.g., the lack of the noise type/its rates.

Related work
Neural architecture search. NAS is the process of automating the design of a neural network architecture. NAS has been successfully applied to various recognition and image generation problems [9,12,62]. NAS algorithms can be computationally demanding e.g., when based on evolutionary algorithms [11,30,31,37] or reinforcement learning [6,62], which provide flexible schemes to sample and evaluate architectures from a vast pool of architectures (so-called search space). However, searching directly on a very large dataset with a large neural network is computationally expensive. Zoph et al.
[63] propose a scalable solution by searching a cell structure instead of the entire neural network structure. Due to the above issues, our method follows the formulation of differentiable NAS [23,53] which is computationally feasible on a broad range of tasks [20,21,50] and results in a cell transferrable to other tasks. Noisy labels. Label noise can harm the performance of neural networks if training is carried out as if data is clean. One of the solutions to tackle label noise is to relabel the samples as proposed in [39,42,51]. Loss correction approaches [5,17,32,58] achieve the same purpose of addressing label noise by compensating and modifying the loss functions. Several types of noise (e.g. symmetric) can even be modeled as a transition matrix [28,48] that indicates the probability of clean labels being flipped to noisy labels. Other approaches combat noisy labels by ranking [14], reweighting [33] and selecting [15] samples. Information bottleneck. Information bottleneck introduced in [41] provides a trade-off formulation between compression of inputs and prediction of outputs. Maximization of the information bottleneck can be approximated by the variational methods [3]. The variational information bottleneck uses so-called noisy computations for adversarial robustness and it achieves so-called sufficiency, minimality and invariance criteria described in [2]. Furthermore, Schwartz-Ziv and Tishby [36] propose a visualization technique operating in the so-called information plane to analyze the information bottleneck principle in deep learning. The theories and analyses from these prior works inspire our investigations of robust NAS learning in the presence of label noise under principles of the information bottleneck.
Training neural networks with noise. The effect of injecting noise into neural networks has been studied in various contexts. Noise injection as a form of regularization was shown to improve the generalization capability of the model [4,8,13]. Noh et al. [27] propose a noise injection method by applying gradient updates over multiple feedforwards of (random) noisy samples and the gradient is weighted based on the importance per sample in each iteration. The above methods differ from ours e.g., they act on the gradient to prevent saturation or consider optimized dropout. In contrast, we propose a parametric noise injector which prevents overfitting to the label noise and is inspired by the information bottleneck principle.

Robust Differentiable Architecture Search
A NAS algorithm operates in two stages, namely the search phase and the evaluation phase. In the search phase, NAS searches the space of cell architectures given a set of operations. During the evaluation phase, a neural network is constructed by stacking multiple cells prior to re-training it from scratch to evaluate the obtained architecture. In the noisy setting, NAS faces incorrect (noisy) labels during both the search and the evaluation phases.
To be more specific, let D be a dataset with M samples where x i ∈ R n and y i ∈ {0, 1} C . We define the noisy data asD = {(x i ,ỹ i )} whereỹ i ∈ {0, 1} C is a noisy label. There are two types of label noise that we consider in this work, namely symmetric and asymmetric noise. Symmetric noise is constructed by swapping clean labels so that labels of each class are contaminated in a chosen equal proportion by other classes. Asymmetric noise is generated by replacing a chosen percentage of labels of one class with labels of another class e.g., cat → tiger. The goal of NAS for noisy labels is to find a robust high-performance architecture that copes well with the label noise during training and testing (prevents overfitting to noisy labels).

Vanilla DARTS
In what follows, we build on the DARTS algorithm [23] which is the driving force behind several recent developments [20,21,50] due to its significant reduction in computational load of constructing high-performance architectures compared to non-differentiable NAS. Prior to introducing our key contributions, we briefly outline DARTS below.
To perform the search over architectures, DARTS formulates the search phase as a bilevel optimization problem. To this end, an inner and an outer objective function L trn and L val are introduced with the goal of finding the architecture parameterization α ∈ A that minimizes L val and the network weights θ that minimize L trn as follows: where θ * (α) represents the weights of the associated architecture that minimize the inner loss. Optimizing both losses is achieved by gradient descent. Unfortunately, optimization in the inner loop is time consuming. As such, DARTS applies a simple approximation with a single unrolling step to optimize the inner loss In the bi-level formulation, we encounter two derivatives (related to the network parameters and the architecture parameters, respectively) for which the chain rule is applied together with the implicit function theorem [7,19,35] which yields: To reduce the training time and computational complexity of the algorithm, one can disregard the higher-order derivatives in Eq. 2. This simplification leads to a speed-accuracy trade-off as observed in several studies (e.g., [23,53]) Following [23,30,63], we make use of two types of cells to design the network, namely I. the normal cell and II. the reduction cell. Each cell is represented by a DAG and can realize one operation from a complete set of candidate operations. The final architecture of the network is constructed by stacking the obtained cells together.
This enables us to make the composition of the cells both transferable between datasets and independent of the network depth. A cell is a Directed Acyclic Graph (DAG) with an ordered sequence of N nodes and is connected to its preceding two cells. Let O be the set of candidate operators (e.g., skip connection, 3×3 dilated convolution, etc.) defined for a cell. Consider a node i whose output represented by z (i) is obtained by: where ⇝ denotes the edge between nodes i and j with a total weight over possible operations given as: where o (i,j) r are individual operations. In essence, choosing the operation that will connect node j to node i is formulated by a mixing weight vector α (i,j) = (α ) using a softmax function. The task of architecture search is then to learn α = α (i,j) . At the end of the search phase, a discrete architecture is picked by choosing the most likely operation between the nodes. That

Learning in the presence of label noise via the information bottleneck principle
Differentiable NAS uses the cross-entropy loss to train (search and evaluation) networks. Below, we formulate a loss function which prevents the trained model from overfitting to noisy labels. To this end, we bring the information theoretic view and explain how it helps with training under label noise. We begin by defining several terms in our analysis: the input x ∈ X, the output y ∈ Y , the representation of the input z ∈ Z, entropy H( . Considering our NAS setting, we revisit the Information Bottleneck [41] principle which provides a computational framework that controls the trade-off between the compression of input x and the prediction of y according to: where β ≥ 0 is a hyper-parameter adjusting the level of relevant information captured by Z while ϕ denotes network parameters. For training neural networks under the regular setting (clean labels), one maximizes I(Z; Y, ϕ) to obtain a model with a high performance. However, increasing the value of I(Z; Y, ϕ) alone makes the model prone to overfitting [1] ie., the network memorizes the dataset and overfits to wrongly labeled datapoints which results in a performance degradation, as shown in our experiments § 4. Let us consider the clean and noisy labelsŶ and Y separately by splitting the first term in Eq. 5 which leads to: Substituting p(y, z) = p(y|z)p(z) in Eq. 6, we obtain: As estimating terms p( y|z; ϕ) and p(ŷ|z; ϕ) of the mutual information is often computationally intractable, it can be completed with the variational approximation. We note that: We formulate the lower bound for the information bottleneck with noisy labels as follows: Let q(z|x; ϕ) and r(z) be a variational approximation for p(z|x; ϕ) and p(z|ϕ), then we have the lower bound: where q(z|x; ϕ) denotes our variational approximation using the reparameterization trick [3,18]. Given the case with noisy labels, we can reduce I(Z; Y , ϕ) by controlling the first term in Eq. 9 to avoid overfitting in the training phase.
In fact, we can control the first term by adjusting the noise variance of the second term of Eq. 9 as follows. Suppose an encoder is used for q(z|x; ϕ), r(z) = N (0, I) and ϕ = {ϕ 1 , ϕ 2 }. We can calculate the KL divergence in the closed form as follows: , and j is the index element of µ x and σ x .

DARTS meets the label noise
The variational information bottleneck provides an inspiration for our noise injecting operators for differentiable NAS. It helps us limit overfitting and memorization to noisy labels. We argue that the designed operator should exhibit the variational approximation in Eq. 9 to adjust I(Z; Y ). To this end, a robust operator should inject the right levels   of noise to successfully train NAS in the presence of label noise. Thus, we propose a new operator, the noise injecting Convolution, nConv for short. In NAS, the architecture parameters α play an important role in assigning weights of operators. As we train the model under label noise, the larger the weights corresponding to our noise injecting operator are, the stronger the regularization effect becomes. We define the functionality of our nConv operator on its input z as z ∼ N (µ x , σ 2 x ) 3 . By properly scaling the noise parameters, we expect a neural network with nConv to exhibit robustness against noisy labels. To this end, we propose to learn the noise parameters (ie., µ x and σ 2 x ) based on the data distribution. Drawing inspiration from the reparameterization trick [18], this is achieved by learning µ and σ 2 through a network parameterized by ϕ as z ∼ q ( z|x; ϕ) (see Fig. 2 (a) for a conceptual diagram). In DARTS, ϕ corresponds to noise parameters of all instantiated noise injecting operators. Let us assume ϕ is contained within θ represents the entire learnable set of parameters of all instantiated operators. As in Eq. 9, the problem of learning the parameters of noise is cast as marginalizing the negative log-likelihood over the data with a KL divergence regularization term as: Note that, minimizing Eq. 11 also applies when we retrain the network in the evaluation phase by using a discrete version of α. Algorithm 1 details the steps of performing DARTS inclusive of our nConv operator.
In vanilla DARTS, two blocks of convolution, namely Separable Convolution (SepConv) and Dilated Convolution (DilConv) are used to construct the search space. Both blocks share the same architectural design (see Fig. 2 (b)). The variants of these two blocks are constructed by adjusting the kernel size, the dilation size, and the stride size of the underlying convolutions. We extend this basic block by Update θ, ϕ using ∇ θ,ϕ L NAS injecting noise at the input z (j) (see Fig. 2 (c)). We can readily extend both blocks with nConv to attain SepNConv and DilNConv blocks. At the cell level, both normal and reduction cells may include nConv in their structures. In the normal cell, the feature map is generated with the same dimension as the input. For the reduction cell, the input to the noise injection module is applied with a stride of two to shrink the cell output size by a factor of two. Selecting nConv in the cells encourages stochastic activations (the output of hidden units) that help regularize the neural network. Note that we experimentally observe that multiplication of noise and hidden units (e.g. dropout) in nConv does not work well to tackle the label noise problem. Thus, our noise injection function is defined by addition of noise to the hidden units.
Theoretical Interpretation. We wire up a network under noise conditions. To this end, it does not suffice to merely tackle the label noise at the classifier layer. From the information bottleneck, the model with the noise injector emerges x l+1 = f (x l ; Θ µ l )+N (0, I)g(x l ; Θ σ l ). In effect, g(·) can be understood as modeling the variance by Θ σ l and hence, the sum x l+1 = f (·)+N (0, I)g(·) pushes x l+1 to contain the information about the parameter uncertainty. The uncertainty flows through the layers as shown for example in Fig. 2. This propagation of uncertainty prevents trivial skip-connections.

The dynamics of training under label noise
In this experiment, we estimate I(Z; Y ) to analyze the mutual information between the hidden units of neural networks (with and w/o noise injecting operators) and their respective labels. The settings are adopted from [36] which provides a toy dataset and a technique to visualize the mutual information. The network for this experiment has six layers in total with Tanh activations on intermediate layers and a Sigmoid function for final outputs. We contaminate 20% training data from the toy dataset [36] that has only two categories. Fig. 3 shows that training the network with our approach (β = 1) under label noise is more stable across 600 epochs. The mutual information is indicative of a correlation between two variables. Fig. 3 (d) shows that a neural network w/o noise injecting operator overfits to noisy labels (top) while our proposed noise injection reduces overfitting (bottom). This experiment confirms our theoretical analysis that the variational information bottleneck, in essence, reduces the mutual information between the hidden units and noisy labels to combat memorization of corrupted labels.

Searching the architecture
We search an architecture with the following operators O: 3 × 3 separable convolution, 3 × 3 dilated convolution, 3 × 3 separable convolution with noise, 3 × 3 dilated convolution with noise, 3 × 3 max pooling, 3 × 3 average pooling, and the identity. Separable and dilated convolution follows the operations in [23,30,63]. The architecture search is run on CIFAR-10 consisting of training and validation sets. The setup is similar to DARTS in that there are 7 nodes for each cell and 16 initial filters. We use the momentum SGD, we set the learning rate and the momentum to 0.025 and 0.9, resp., and the weight decay to 3× 10 −4 . Moreover, a cosine schedule is employed to anneal the learning rate. A network with 8 cells is trained for 50 epochs with a batch size of 96. The search time is 6 hours on a NVIDIA Titan V GPU.

Datasets and implementations details
The evaluation phase follows the protocol in DARTS in which the architecture is found via a small task and then the cell structure is transferred to other (bigger) tasks. Normal cells and reduction cells are based on the architecture found on CIFAR-10, then the cells are stacked for evaluation. All experiments for the architecture evaluation are averaged over 3 runs. The model is trained for 300 epochs for all datasets. In the evaluation phase, the cells are stacked to form a group of 16 cells and 16 initial filters. The optimizer parameters are set following the setup during the search phase.
Benchmark datasets for robustness evaluation used in our experiments are CIFAR-10 and CIFAR-100 consisting of 50k samples and 10k samples for training and testing, respectively. The image size of both CIFAR datasets are 32× 32 following the standard protocol in past NAS and noisy label works [15,23]. We investigate both symmetric and asymmetric label noise. Symmetric noise rates on CIFAR-10 and CIFAR-100 are 20% and 50%, and the asymmetric noise rate is 40%. All results are reported based on the average of the last ten epochs with β = 1. We also run evaluations on a larger Tiny-ImageNet 4 with 200 classes and 120k images in total. The image size of Tiny-ImageNet is set to 64×64 following the setup in [52]. Finally, we experiment with the Clothing1M [49] which contains real-world noisy labels, 14 categories, and ∼1M images (224 × 224 size). The network is trained for 80 epochs. SGD with momentum is used to optimize the model with the learning rate set to 0.1. Remaining parameters follow the setup for CIFAR.

Comparison with Vanilla DARTS
Below, we compare the performance of Vanilla DARTS and our DARTS with the nConv operator (DARTS + nConv) to study the improvement of the latter when learning in the presence of label noise. We use DARTS with following operators O: 3 × 3 separable convolution, 3 × 3 dilated convolution, 5 × 5 separable convolution, 5 × 5 dilated convolution, 3 × 3 max pooling, 3 × 3 average pooling, and the identity. We use the same setup (e.g. optimizer parameters/batch size) for both techniques to search and evaluate the found architecture. The search/evaluation phases use the same label noise rates (e.g., searching on CIFAR-10 with 50% noise and evaluating the found architecture on CIFAR-100 with 50% noise). Fig. 4 shows that DARTS + nConv achieves superior performance compared to Vanilla DARTS.

Comparison with the state of the art for training under label noise
We compare our method to multiple baselines that are designed to tackle problems resulting from label noise. The baselines we compare to, use three different backbones: Conv-9 [15], ResNet-18, and ResNet-34 [16]. Table 2 shows the results on CIFAR-10 and CIFAR-100 in the presence of 20% as well as 50% symmetric noise. Our proposed method with nConv outperforms T-Revision [48] on all protocols by 4 https://tiny-imagenet.herokuapp.com/

Method
Backbone 20%-Sym 50%-Sym F-Correction [28] Conv-9     a substantial margin of 2%. In addition, Table 3 shows that our proposed method consistently performs better than the state of the art ie., joint-optimization [39], PENCIL [51], and F-Correction [28]. Note that our approach does not need any modifications of the loss function to train the model under symmetric or asymmetric noise. We further evaluate the performance on challenging datasets, namely Tiny-ImageNet and Clothing1M [49]. Note that our evaluation does not use pre-trained neural networks for both datasets. As shown in Table 4, DARTS + nConv also outperforms Co-teaching+ [52], Co-teaching [15], Men-torNet [17], and F-Correction [28]. When the noise rate increases from 20% to 50%, the performance gap is widened. As an example, the performance gap increases by 3% when comparing our method with Co-teaching [15]. Moreover, Table 5 shows that our method outperforms state-of-the-art methods on the Clothing1M dataset by 1.3%.
Our proposed method outperforms current state of the art on all datasets irrespective of noise rates while substantially reducing the number of parameters compared to Conv-9, ResNet-18, and ResNet34 backbones. The searched architecture has fewer parameters (size reduced by 10-20×) compared to ResNet backbones. Finally, our approach maintains only one neural network while previous methods (e.g., Decoupling [25] and Co-teaching [15]) train two networks simultaneously.

NAS benchmark evaluation
Below, we examine the capability of DARTS to obtain high performance architectures in the presence of label noise. We compare our method with Vanilla DARTS and PC-DARTS [50] using NAS-benchmark 1Shot1 [54]. There are three different search spaces (1,2, and 3) used in our evaluation which differ by the number of nodes and node parents. The search space 3 in this benchmark is the largest search space (∼360,000 architectures) among NAS benchmarks. CIFAR-10 is used to search the architecture with 50% symmetric noise. Fig. 5 shows that our method does not overfit to the noisy labels and performs better (lower test regret) compared to Vanilla DARTS and PC-DARTS.

Discussion
In our experiments, we have observed that reducing the mutual information by adjusting the noise variance (following the ELBO derivations) according to data helps us overcome overfitting to the label noise. The benefit of our structural approach is that we train a neural network in the presence of label noise with fewer hyper-parameters e.g., no noise rate [15] or a bootstrapping parameter [32]. Moreover, a standard training loss (e.g., cross-entropy) can be used directly without any modifications (e.g., multiplying with a certain weight or hinge loss) for both label noise cases.
In the recent NAS work, Liu et al. [22] proposes a selfsupervised method so-called UnNAS to search an optimal architecture. However, the labels of pretext tasks (e.g., rotation, scaling, permutations [56,57]) suffer from the noise depending on the initial state of the input. Even unsupervised, contrastive and hallucinating methods [44,45,[59][60][61] suffer from the diffusion, negative sampling noise or quantization noise. Yet, NAS using our approach is able to find a better architecture compared to UnNAS [22]. Fig. 5 (d) shows that our method outperforms the result of UnNAS by achieving lower test regret of NAS benchmark 1shot1.

Conclusions
We have presented a new operator coined nConv to search and train a robust neural network in the presence of label noise. We have shown that label noise harms the performance of Vanilla DARTS while DARTS + nConv performs robustly on problems with noisy labels. The proposed operator has a minimum amount of overhead compared to the existing operators in the search space (e.g., convolution operations in Vanilla DARTS). As changes are applied to the neural network structure, mechanisms such as sample selection, modified loss functions or transition matrices are not needed by our approach. Our empirical results also show that we can design a neural network with fewer parameters compared to ResNet backbones, without the loss of accuracy. In this supplementary material, we provide the implementation details of our method, additional results, and our training strategy.

Derivation
Recall in the main paper we have I(Z;Ŷ , ϕ) and I(Z; Y , ϕ) with the lower bound defined as follows: The lower bound L in Eq. 12 can be approximated using the formulation for variational inference q(z|x; ϕ): L = p(x)p(ŷ|x; ϕ)p(z|x; ϕ) logp(ŷ|z; ϕ) dx dŷ dz ≈E z∼q(z|x;ϕ) log p(ŷ n |z) The first and second terms in Eq. 13 are the average crossentropy and the third term is a regularization term. Our objective is to attain the low value of L but we should avoid minimizing the second term that operates on noisy labels to avoid overfitting to these erroneously labeled datapoints.
To this end, the noise injection by the noise injecting operator (which actually approximates the information bottleneck) acts as a regularization to the loss function.

Implementation details
Our noise injection operator consists of adaptive average pooling, two fully connected layers with a ReLU function baetween them, and a Sigmoid function. The average pooling operation shrinks the height and width R C×H×W of the hidden unit to R C×1×1 on which σ ∈ R C×1×1 is learnt with a simple Multi-layer Perceptron (MLP). Thus, a noise injection module which generates σ ∈ R C×1×1 , performs a channel-wise variance scaling step to the noise (tensor of size R C×H×W ) sampled from a Normal distribution. Subsequently, the scaled noise is directed to the output of the operator. Finally, the output is added with the residual of the input.
There are two types of convolutions in the search space: the dilated convolution and the separable convolution. For instance, the separable convolution with 3x3 kernel (sep conv 3x3) and the dilated convolution with 3x3 kernel (dil conv 3x3) are presented in Tables 6 and 7, respectively. We also provide examples of kernels given the injected noise in the separable convolution with 3x3 kernel (sepconv3x3 noise) and the dilation convolution with 3x3 kernel (dilconv3x3 noise).

Searching architectures
In Fig. 8, we show a cell architecture found via Vanilla DARTS with 20% and 50% symmetric noise on CIFAR-10. Most of the operations in the reduction cells for both noise cases have no parameters (e.g., skip connection and maxpooling). The parameterless cells in Fig. 8 have a smaller     parameter size compared to the cells found with our operator in Fig. 9. For instance, an architecture searched with Vanilla DARTS (16 stacked cells) has 0.6M parameters but an architecture including nConv (16 stacked cells) has 1.2M parameters. It appears that in the presence of the label noise, the standard architecture search engine tries to prevent overfitting by selecting parameterless operations which are not expressive enough and thus lead to sub-optimal architectures with poor performance.

The impact of the standard deviation
Below, we perform an ablation study to investigate the impact of different values for standard deviation of the noise injection parameter. The setup for this experiment follows the setup in § 4 with the toy data from [36] contaminated with 20% symmetric noise and the six layers neural network. We vary the standard deviation in range {0.01, 0.05, 0.2, 0.3} and the mean in range {0.1, 0.2, 0.3}, and train the model for 600 epochs. Fig. 6 shows that the standard deviation affects both the accuracy and the training loss when training the model. We observe that training without the noise injection performs poorly because the model overfits to the noisy labels given the lowest training loss attained. We found that injecting noise with standard deviation of 0.2 or 0.3 yields the highest validation accuracy while avoiding overfitting (indicated by the higher and smoothly decreasing training loss). In this experiment, the improvement using the noise injection module yields 18% improvement over the baseline which does not include the noise injection unit.
We also observe that varying the mean does not affect the performance significantly, thus, we conveniently set the mean to zero for all experiments in this work.

Gradient analysis under training with noisy labels
In this experiment, we analyze the impact that training samples with clean labels vs. noisy labels has on the gradient. This helps us understand from which samples the network learns and how overfitting occurs due to the label noise. Fig. 7 (a) shows the gradient norm as a function of epoch for Vanilla DARTS. The figure shows that in the early epochs, samples with clean labels dominate over samples with noisy labels, whereas in the later epochs of the training, the reverse is true. This phenomenon leads to a performance drop towards the end of training and the model parameters are comparatively more influenced by the noisy samples rather than the clean samples as the network tries to fit to datapoints with erroneous labels.
We then perform the equivalent experiment using our proposed method, shown in Fig. 7 (b). The figure shows that the impact of the gradient of the clean samples dominates the impact of the gradient of the noisy samples throughout the entire training. This illustrates that the risk of overfitting to noisy labels is substantially reduced, and that the network preferentially learns from the clean samples. This observation of how gradient behave translates into the superior performance (under noisy labels) of DARTS + nConv compared to Vanilla DARTS. ReLU 3x3 convolution, C channels, no bias 3x3 convolution, C channels, no bias 1x1 convolution, C channels, no bias 1x1 convolution, C channels, no bias Batch normalization Batch normalization -Noise injection ReLU ReLU 3x3 convolution, C channels, no bias 3x3 convolution, C channels, no bias 1x1 convolution, C channels, no bias 1x1 convolution, C channels, no bias Batch normalization Batch normalization Dilated convolution 3x3 (dil conv 3x3) Dilated Noisy Convolution 3x3 (dilconv3x3 noise) -Noise injection ReLU ReLU 3x3 convolution, C channels, no bias, dilation 2 3x3 convolution, C channels, no bias, dilation 2 1x1 convolution, C channels, no bias 1x1 convolution, C channels, no bias Batch normalization Batch normalization