DARTS+: Improved Differentiable Architecture Search with Early Stopping

Recently, there has been a growing interest in automating the process of neural architecture design, and the Differentiable Architecture Search (DARTS) method makes the process available within a few GPU days. In particular, a hyper-network called one-shot model is introduced, over which the architecture can be searched continuously with gradient descent. However, the performance of DARTS is often observed to collapse when the number of search epochs becomes large. Meanwhile, lots of"skip-connects"are found in the selected architectures. In this paper, we claim that the cause of the collapse is that there exist cooperation and competition in the bi-level optimization in DARTS, where the architecture parameters and model weights are updated alternatively. Therefore, we propose a simple and effective algorithm, named"DARTS+", to avoid the collapse and improve the original DARTS, by"early stopping"the search procedure when meeting a certain criterion. We demonstrate that the proposed early stopping criterion is effective in avoiding the collapse issue. We also conduct experiments on benchmark datasets and show the effectiveness of our DARTS+ algorithm, where DARTS+ achieves $2.32\%$ test error on CIFAR10, $14.87\%$ on CIFAR100, and $23.7\%$ on ImageNet. We further remark that the idea of"early stopping"is implicitly included in some existing DARTS variants by manually setting a small number of search epochs, while we give an explicit criterion for"early stopping".

The common practice of NAS first derives an architecture search space, and then finds the best architecture in the search space with a specific search method. Early works of NAS usually adopt the REINFORCE method [25,40] and evolutionary algorithms [26] for searching effective architectures. However, obtaining state-of-the-art architectures with such methods involves huge computational cost (e.g., thousands of GPU days [40]), because a large number of architectures need to be trained and evaluated. Recently, one-shot methods [2,3,20,25,34] are proposed to reduce the search cost. Among the one-shot methods, Liu, Simonyan, and Yang [20] propose the Differentiable Architecture Search (DARTS) method to relax the search space to be continuous, so that one can search architectures and learn model weights directly with gradient descent. In particular, DARTS encodes the architecture search space with continuous parameters and performs searching with bi-level optimization, where the model weights and architecture parameters are optimized with training data and validation data alternatively. DARTS can reduce the search cost from thousands of GPU days to a few GPU days while keeping comparable performance.
Despite the efficiency of DARTS, a severe issue underlying DARTS has been found [4]. Namely, after a certain searching epochs, the number of skip-connects increases dramatically in the selected architecture, which results in poor performance of the selected architecture. We call the phenomenon of performance drop after a certain number of epochs the "collapse" of DARTS. To tackle such issue, Chen et al. [4] propose a search space regularization in their P-DARTS, where dropout [29] is used to alleviate the dominance of skip-connects during the search procedure, and the number of skip-connects is manually controlled after the search procedure. However, this approach involves more hyper-parameters like dropout rate and the number of skip-connects, which need to be carefully tuned by human experts. Moreover, Stamoulis et al. [30] propose Single-Path NAS using the one-level optimization instead of the bi-level optimization in DARTS, where the architecture parameters and model weights are updated simultaneously. However, the search space of Single-Path NAS is carefully designed. If searching is done in the original search space of DARTS, one-level optimization will perform worse than bilevel ones [20]. In summary, the mechanism of the collapse of DARTS is still unknown and needs to be addressed.
In this paper, we first show the cause of the collapse of DARTS is that there exist cooperation and competition in the bi-level optimization in DARTS, where the architecture parameters and model weights are updated alternatively. In particular, we give an explanation of why lots of skip-connects are involved in the selected architectures in DARTS and why they hurt the performance. To avoid the collapse of DARTS, we add the simple and effective "early stopping" paradigm to the original DARTS, named "DARTS+", where the search procedure stops by a certain criterion, illustrated in Fig. 1(a). We remark that recent progresses of DARTS, including P-DARTS [4], Auto-DeepLab [19] and PC-DARTS [37], also adopt the early stopping idea implicitly where fewer search epochs are manually set in their methods.
Moreover, we conduct sufficient experiments on benchmark datasets to demonstrate the effectiveness of the proposed DARTS+ algorithm. Specifically, DARTS+ achieves the state-of-the-art 2.32% test error on CIFAR10 and 14.87% test error on CIFAR100, while the search time is less than 0.4 GPU days. When transferring to ImageNet, DARTS+ achieves the state-of-the-art 23.7% top-1 error and impressive 22.5% top-1 error if SE-Module [12] is introduced. DARTS+ is also able to search on ImageNet directly and achieves 23.9% top-1 error.
In summary, our main contributions are listed as follows: • We study the collapse issue of the DARTS method, and point out the underlying reason is the cooperation and competition in the bi-level optimization.
• We introduce an efficient "early stopping" paradigm to DARTS to avoid the collapse, and propose an effective criterion for early stopping.
• we conduct extensive experiments on benchmark datasets to demonstrate the effectiveness of the proposed algorithm, which achieves the state-of-the-art results on all of them.

Collapse of DARTS
There is a severe issue underlying DARTS [20], that lots of skip-connects tend to appear in the selected architecture when the number of searching epoch is large, making the performance poor. The phenomenon of performance drop is called the "collapse" of DARTS in our paper. In this section, we first give a quick review of the original DARTS, and then point out the collapse issue of DARTS. Moreover, we will discuss the cause of the collapse issue.

Preliminary: DARTS
The goal of DARTS is to search for a cell, which can be stacked to form a convolutional network or recursively connected to form a recurrent network. Each cell can be regraded as a directed acyclic graph (DAG) of N nodes {x i } N −1 i=0 , where each node represents a network layer. We denote the operation space as O, and each element is a candidate operation, e.g., zero, skip-connect, convolution, maxpool, etc. Each edge (i, j) of DAG represents the information flow from node x i to x j , which consists of the candidate operations weighted by the architecture parameter α (i,j) . In particular, each edge (i, j) can be formulated by a function , and the weight of each operation o ∈ O is a softmax of the architecture parameter α (i,j) , that is p . An intermediate node is x j = i<jō (i,j) (x i ), and the output node x N −1 is depth-wise concatenation of all the intermediate nodes excluding input nodes. The above hyper-network is called one-shot model, and we denote w as the weights of the hyper-network.
For the search procedure, we denote L train and L val as the training and validation loss respectively. Then the architecture parameters are learned with the following bi-level optimization problem: After obtaining architecture parameters α, the final discrete architecture is derived by: 1) setting o (i,j) = arg max o∈O,o =zero p

Collapse Issue
It has been observed in DARTS that lots of skip-connects are involved in the selected architecture, which makes the architecture shallow and the performance poor. As an example, let us consider searching on CIFAR100. The alpha value of skip-connects (green line in Fig. 2(c)) becomes large when the number of search epochs is large, and thus the number of skip-connects increases in the selected architecture as shown in the green line in Fig. 2(a). Such shallow network has less learnable parameters than deep ones, thus it has weaker expressive power. As a result, architectures with lots of skipconnects have poor performance, i.e., collapsed, indicated as the blue line in Fig. 2(a). To be more intuitive, we draw the selected architectures from different search epochs on CIFAR100 in Fig. 1(b). When the number of search epochs increases, the number of skip-connects in the selected architecture also increases. Such phenomenon can also be observed on other datasets, such as CIFAR10 and ImageNet.
To avoid the collapse, one might propose to adjust search hyper-parameters, such as 1) adjusting learning rates, 2) changing the portion of training and validation data, and 3) adding regularization on skip-connects like dropout. Unfortunately, such methods would only delay the collapse as the choice of hyper-parameters is not the essential cause of collapse.
The underlying reason of the collapse issue is that there exist cooperation and competition in the bi-level optimization in DARTS, where the architecture parameters and model weights are updated alternatively. Intuitively, the architecture parameters and model weights are well optimized at the beginning, and then gradually turn to compete against each other after a while. Since the model weights have more With a proper early stopping mechanism, we may obtain better architectures, or we may end up with collapsed architectures with lots of skip-connects.
advantages than the architecture parameters in the competition, (e.g., the number of the model weights is far more than the number of architecture parameters, the architecture parameters are not sensitive to the final loss in the bi-level optimization, etc.), the architecture parameters cannot beat the model weights in the competition. As a result, the performance of selected architecture will first increase and then decrease (See blue lines in Fig. 2(a)).
In particular, at the initial state of the search procedure, the one-shot model underfits the training data. Thus, architecture parameters α and model weights w, which are learnable parameters of the one-shot model, will get better together at the beginning of the search procedure. This is the cooperation period. Note that the first cells in the whole oneshot model can touch the fresh data information, while the data that feed to the last cells are much noisier. If we allow different cells to have distinct architectures in the oneshot model, the first cells will learn features more quickly than the last cells. Since the feature representation learned by the last cells is relatively worse than the one learned by the first cells, the last cells are more likely to select more skip-connects to obtain the good feature representation directly from the first cells. Fig. 3 shows the learned normal cell architectures at different layers if we allow different architectures at different stages 1 . It can be seen that the algorithm tends to select deep architectures with learnable operations (namely, operations with parameters to be learned such as convolutions) in the first cells ( Fig. 3(a)), while architectures with many skipconnects are preferred in the last cells (Fig. 3(c)). If different cells are forced to have the same architecture, as DARTS does, skip-connects will be broadcasted from the last cells  The major advantage of the proposed stopping criterion is its simplicity. Compared with other DARTS variants, DARTS+ only needs a few modifications based on DARTS, and can significantly increase the performance with less search time. As mentioned in Sec. 2.2, too many skipconnects will hurt the performance of DARTS. On the other hand, an appropriate number of skip-connects is helpful for transferring the information from first layers to last layers and stabilizing the training process, e.g., ResNet [10], which makes the architectures achieve better performance. Therefore, stopping by Criterion 1 is a reasonable choice. Criterion 1 is motivated by P-DARTS [4], where the number of skip-connects in the cell of final architecture is manually cut down to two. Although both DARTS+ and P-DARTS keep two skip-connects in the cell of their final architectures, DARTS+ is essentially different from P-DARTS in dealing with the skip-connects. P-DARTS does not intervene the number of skip-connects during the search procedure, but only replaces the redundant skip-connects with other operations as a post-processing after the search procedure finishes. In contrast, our DARTS+ ends up with desired architectures with a proper number of skip-connects to avoid the collapse of DARTS. It controls the number of skipconnects more directly and also more effectively (See Table 1 for a performance comparison between DARTS+ and P-DARTS). Now we give some intuition for Criterion 1 in Fig. 2. The red circles in Fig. 2(a-b) denote the points that the ranking of architecture parameters α for learnable operations (e.g.,  convolutions) becomes stable on CIFAR10, CIFAR100 and Tiny-ImageNet-200, respectively. Note that only the operation with the maximum α value is chosen in the selected architecture. When the ranking of α becomes stable, the operations to be selected are nearly determined, which implies that the search is almost saturated. The experiments also verify that after the point that the ranking of α becomes stable (red circles in Fig. 2(a)), the validation accuracies of selected architectures on all datasets (blue lines) tend to decrease, i.e., collapse. It can be seen that this point is close to the stopping point when Criterion 1 holds (the red dash line in Fig. 2(a)), thus stopping by Criterion 1 can prevent the collapse of DARTS.
Since the stable ranking of architecture parameters α for learnable operations indicates the saturated search procedure in DARTS, we can also use the following stopping criterion: Criterion 1* The search procedure stops when the ranking of architecture parameters α for learnable operations becomes stable for a determined number of epochs (e.g., 10 epochs).
We remark that Criterion 1 is much easier to operate, but if one needs stopping more precisely or other search spaces are involved, Criterion 1* could be used instead. We further remark that our early stopping paradigm solves an intrinsic issue of DARTS and is orthogonal to other tricks, thus it has potential to be used in other DARTS-based algorithms to achieve better performance.
We note that recent state-of-the-art differentiable architecture search methods also introduce the early stopping idea in an ad hoc manner. To avoid the collapse, P-DARTS [4] uses 1) searching for 25 epochs instead of 50 epochs, 2) adopting dropout after skip-connects, and 3) manually reducing the number of skip-connects to two. Auto-DeepLab [19] starts to update architecture parameters for a few epochs after updating weight parameters. PC-DARTS [37] uses partialchannel connections to reduce search time, and therefore more epochs are needed for convergence of searching. Thus, setting 50 training epochs is also an implicit early stopping paradigm.

Datasets
In this section, we conduct extensive experiments on benchmark classification datasets to evaluate the effectiveness of the proposed DARTS+ algorithm. We use four popular datasets including CIFAR10 [16], CIFAR100 [16], Tiny-ImageNet-200 3 and ImageNet [6]. CIFAR10/100 consists of 50K training images and 10K testing images and the resolution is 32 × 32. Tiny-ImageNet-200 contains 100K 64 × 64 training images and 10K testing images. ImageNet is obtained from ILSVRC2012 [27], which contains more than 1.2M training images and 50K validation images. We follow the general setting on the ImageNet dataset where the images are resized to 224 × 224 for training and testing.

Architecture Search
Unless specified, we use Criterion 1 as the stopping condition for DARTS+ in the experiments. Note that the stopping points by Criterion 1 and 1* are almost the same in the proposed search space, and we will discuss the two criteria in detail in Sec. 4.4.

Implementation Details.
We have similar experimental settings as DARTS. The experiments are carried out in two stages: architecture search and architecture evaluation. The search space is the same as DARTS which has 8 candidate operations including skip-connect, max-pool-3x3, avg-pool-3x3, sep-conv-3x3, sep-conv-5x5, dil-conv-3x3, dil-conv-5x5, zero, and the structure of each operation is exactly the same as DARTS.
For CIFAR10 and CIFAR100, we use the same one-shot model as the original DARTS in which 8 cells (i.e. 6 normal cells and 2 reduction cells) with 16 channels are trained for searching. We use a half of the training data to train the model weights and the other half to update the architecture parameters. We search for a maximum of 60 epochs with batch size 64. We use SGD to optimize the model weights with initial learning rate 0.025, momentum 0.9 and weight decay 3 × 10 −4 . Adam [15] is used to optimize architecture parameters with initial learning rate 3 × 10 −4 , momentum (0.5, 0.999) and weight decay 10 −3 . Early stopping is applied at certain epoch when Criterion 1 introduced in Sec. 3 is met.
For Tiny-ImageNet-200, the one-shot model is almost the same as CIFAR10/100 except that a 3 × 3 convolution layer with stride 2 is added on the first layer to reduce the input resolution from 64 × 64 to 32 × 32. Other settings are the same as those used in CIFAR10/100, including the "early stopping" criterion.
For ImageNet, following [37], the one-shot model starts with three 3 × 3 convolution layers with stride 2 to reduce the resolution from 224 × 224 to 28 × 28, and the rest of the network consists of 8 cells. We select 10% data from the training set for updating model weights, and another 10% for updating architecture parameters. We search with batch size 512 for both training and validation sets. SGD is used for model weights training with initial learning rate 0.2 (cosine 3 https://tiny-ImageNet.herokuapp.com/ learning rate decay), momentum 0.9, and weight decay 3 × 10 −4 . The architecture parameters are trained with Adam with learning rate 3 × 10 −3 , momentum (0.5, 0.999) and weight decay 10 −3 .
For all the datasets, the one-shot model weights and architecture parameters are optimized alternatively. The cell structure is determined by architecture parameters, following DARTS [20].
Search Results and Analysis. The proposed DARTS+ needs less searching time as "early stopping" is adopted. For CIFAR10, the search procedure requires 0.4 GPU days with a single Tesla V100 GPU, and stops at about 35 epochs. For CIFAR100, the searching time is 0.2 GPU days and the search procedure stops at about 18 epochs. For Tiny-ImageNet-200, searching stops at about 10 epochs. For Im-ageNet, the search procedure involves 200 epochs, and it requires 6.8 GPU days on Tesla P100 GPU.
The selected architectures are shown in Fig. 4. We observe that the cells searched by DARTS+ contain most convolutions and a few skip-connects.
It should be noticed that DARTS+ succeeds in searching with all three datasets including CIFAR10/100, Tiny-ImageNet-200 and ImageNet. However, the original DARTS fails to search on CIFAR100 as the selected architecture is full of skip-connects [4], and most previous works on differentiable search [4,20,34] do not search on ImageNet.

Architecture Evaluation
For each selected architecture, we follow the configurations and hyper-parameters of the previous works [4,20] for evaluation on different datasets.
Results on CIFAR10 and CIFAR100. We use network of 20 cells and 36 initial channels for evaluation to ensure a comparable model size as other baseline models. We use the whole training set to train the model for 2000 epochs with batch size 96 to ensure convergence. Other hyper-parameters are set the same as the ones in the search stage. Following existing works [18,25,26,41], we also add some enhancements including cutout, path dropout with probability 0.2 and auxiliary towers with weight 0.4.
The evaluation results are summarized in Table 1. For each selected cell from either CIFAR10 or CIFAR100, we report the performance on both datasets. With the simple "early stopping" paradigm, we achieve the best results with 2.32% test error on CIFAR10 and 14.87% test error on CIFAR100. The results are much better than the original DARTS, which gets 3% test error on CIFAR10 and 19.5% test error on CIFAR100 (See Fig. 2(a) with large epochs). The proposed DARTS+ is much simpler and better than other modified DARTS algorithms like P-DARTS and PC-DARTS. ProxylessNAS uses a different search space, and it involves more search time. Moreover, DARTS+ is much easier to implement than other modified DARTS variants including ASAP. We point out that "early stopping" can be used in many other differentiable search algorithms and search spaces to obtain better architectures.
We further increase the initial channel number from 36 to 50, and add more augmentation tricks including AutoAug-   ment [5] and mixup [39] to achieve better results. Table 1 shows that DARTS+ achieves impressive 1.68% test error on CIFAR10 and 13.03% test error on CIFAR100, demonstrating the effectiveness of DARTS+.
Results on Tiny-ImageNet-200. The network is similar as CIFAR10/100 where 20 cells and 36 channels are involved, except that an additional 3×3 convolution layer with stride 2 is inserted in the first layer. We transfer the architectures searched from other algorithms to Tiny-ImageNet-200 and evaluate the performance for fair comparison. Other experimental settings are the same as CIFAR10/100. The results are shown in Table 2. DARTS+ achieves the state-of-the-art 28.3% test error, which is much better than other baselines. Note that architecture searched on Tiny-ImageNet-200 with DARTS+ performs much better than DARTS and its parameter size is much larger than DARTS, because DARTS suffers from collapse and the architecture searched with DARTS contains lots of skip-connects. Desired architectures are more likely to be generated with the "early stopping" paradigm.
Results on ImageNet. We use the architecture searched directly from ImageNet for evaluation, and the architecture from CIFAR100 to test the tranferability of the selected architecture. We follow DARTS such that the number of cells is 14 and the initial number of channels is 48. We train the  The model is optimized with the SGD optimizer with an initial learning rate 0.8 (cosine decayed to 0), momentum of 0.9 and weight decay 3×10 −5 . We use learning rate warmup [9] for the first 5 epochs and other training enhancements including label smoothing [31] and auxiliary loss tower.
The experimental results are shown in Table 3. Note that we re-implement PC-DARTS and the results are reported. When searching on ImageNet with the proposed DARTS+, the selected architecture achieves impressive 23.9%/7.4% top-1/top-5 error, and the architecture transferred from CI-FAR100 achieves state-of-the-art 23.7%/7.2% error. The results imply that DARTS with "early stopping" succeeds in searching a good architecture with impressive performance on large-scale datasets with limited time.
We also adopt SE-module [12] in the architecture transferred from CIFAR100, and introduce AutoAugment [5] and mixup [39] for training to obtain better model. The results are shown in Table 3, and we achieve 22.5%/6.4% top-1/top-5 error with only additional 3M flops, showing the effectiveness of the selected architecture.

Effectiveness of Early Stopping
To further verify the effectiveness of DARTS+, we conduct extensive experiments on selected architectures at different epochs. The classification results on CIFAR10, CIFAR100 and Tiny-ImageNet-200 are shown in Fig. 2. We also point out the time to "early stop" under two criteria, marked as "red dashed line" and "red circle" respectively. We observe that the selected architecture performs worse with larger epochs, implying that the original DARTS suffers from the collapse issue. In contrast, "early stopping" is able to generate good architectures at both stopping criteria, regardless of the type of datasets.
We also compare "early stopping" Criterion 1 and 1* in Table 1 and Fig. 2. We observe that both criteria achieve comparable performance on all datasets as the stopping points are very close.

Conclusion
In this paper, we conduct comprehensive analysis and extensive experiments to show that DARTS suffers from the collapse problem, which is mainly caused by the cooperation and competition problem in the bi-level optimization in DARTS. We propose the "DARTS+" algorithm, in which the "early stopping" paradigm is introduced to avoid the collapse of DARTS. The experiments show that we succeed in searching on various benchmark datasets including largescale ImageNet with limited GPU days, and the resulting architectures achieve the state-of-the-art performances on all benchmark datasets. Moreover, it should be noticed that many recent progresses of DARTS could use "early stopping" to achieve better results, and the proposed "early stopping" criteria could be applied to many other types of search spaces, including the RandWire search space [35], mobile convnets search space [30], etc.