We present an end-to-end design methodology for efficient deep learning deployment. Unlike previous methods that separately optimize the neural network architecture, pruning policy, and quantization policy, we jointly optimize them in an end-to-end manner. To deal with the larger design space it brings, we train a quantization-aware accuracy predictor that fed to the evolutionary search to select the best fit. We first generate a large dataset of <NN architecture, ImageNet accuracy> pairs without training each architecture, but by sampling a unified supernet. Then we use these data to train an accuracy predictor without quantization, further using predictor-transfer technique to get the quantization-aware predictor, which reduces the amount of post-quantization fine-tuning time. Extensive experiments on ImageNet show the benefits of the end-to-end methodology: it maintains the same accuracy (75.1%) as ResNet34 float model while saving 2.2× BitOps comparing with the 8-bit model; we obtain the same level accuracy as MobileNetV2+HAQ while achieving 2×/1.3× latency/energy saving; the end-to-end optimization outperforms separate optimizations using ProxylessNAS+AMC+HAQ by 2.3% accuracy while reducing orders of magnitude GPU hours and CO2 emission.
 Deep learning has prevailed in many real-world applications like autonomous driving, robotics, and mobile VR/AR, while efficiency is the key to bridge research and deployment. Given a constrained resource budget on the target hardware (e.g., latency, model size, and energy consumption), it requires an elaborated design of network architecture to achieve the optimal performance within the constraint. Traditionally, the deployment of efficient deep learning can be split into model architecture design and model compression (pruning and quantization). Some existing works (Han et al., 2016b; have shown that such a sequential pipeline can significantly reduce the cost of existing models. Nevertheless, careful hyper-parameter tuning is required to obtain optimal performance (He et al., 2018) . The number of hyper-parameters grows exponentially when we consider the three stages in the pipeline together, which will soon exceed acceptable human labor bandwidth. To tackle the problem, recent works have applied AutoML techniques to automate the process. Researchers proposed Neural Architecture Search (NAS) (Zoph & Le, 2017; Real et al., 2018; Liu et al., 2018a; b; Zhong et al., 2018; Elsken et al., 2018; Cai et al., 2018a; b; Luo et al., 2018; Kamath et al., 2018) to automate the model design, outperforming the human-designed models by a large margin. Based on a similar technique, researchers adopt reinforcement learning to compress the model by automated pruning (He et al., 2018) and automated quantization . However, optimizing these three factors in separate stages will lead to sub-optimal results: e.g., the best network architecture for the full-precision model is not necessarily the optimal one after pruning and quantization. Besides, this three-step strategy also requires considerable search time and energy consumption (Strubell et al., 2019) . Therefore, we need a joint, end-to-end solution to optimize the deep learning model for a certain hardware platform. However, directly extending existing AutoML techniques to our end-to-end model optimization setting can be problematic. Firstly, the joint search space is cubic compared to stage-wise search, making the search difficult. Introducing pruning and quantization into the pipeline will also greatly increase the total search time, as both of them require time-consuming post-processing (e.g., finetuning) to get accuracy approximation Yang et al., 2018) . Moreover, the search space of each step in pipeline is hard to be attested to be disentangle, and each step has its own optimization objective (eg. acc, latency, energy), so that the final policy of the pipeline always turns out to be sub-optimal. To this end, we proposed EMS, an end-to-end design method to solve this problem. Our approach is derived from one-shot NAS (Guo et al., 2019; Brock et al., 2018; Pham et al., 2018; Bender et al., 2018; Liu et al., 2019a; Yu & Huang, 2019) . We reorganize the traditional pipeline of "model design→pruning→quantization" into "architecture search + mixed-precision search". The former consists of both coarse-grained architecture search (topology, operator choice, etc.) and fine-grained channel search (replacing the traditional channel pruning (He et al., 2017) ). The latter aims to find the optimal mixed-precision quantization policy trading off between accuracy and resource consumption. We work on both aspects to address the search efficiency. For architecture search, we proposed to train a highly flexible super network that supports not only the operator change but also fine-grained channel change, so that we can perform joint search over architecture and channel number. For the mixed-precision search, since quantized accuracy evaluation requires time-consuming fine-tuning, we instead use a predictor to predict the accuracy after quantization. Nevertheless, collecting data pairs for predictor training could be expensive (also requires fine-tuning). We proposed PredictorTransfer Technique to dramatically improve the sample efficiency. Our quantization-aware accuracy predictor is transferred from full-precision accuracy predictor, which is firstly trained on cheap data points collected using our flexible super network (evaluation only, no training required). Once the predictor P (arch, prune, quantization) is trained, we can perform search at ultra fast speed just using the predictor. With the above design, we are able to efficiently perform joint search over model architecture, channel number, and mixed-precision quantization. The predictor can also be used for new hardware and deployment scenarios, without training the whole system again. Extensive experiment shows the superiority of our method: while maintaining the same level of accuracy (75.1%) with ResNet34 float model, we achieve 2.2× reduction in BitOps compared to the 8-bit version; we obtain the same level accuracy as MobileNetV2+HAQ, and achieve 2×/1.3× latency/energy saving; our models outperform separate optimizations using ProxylessNAS+AMC+HAQ by 2.3% accuracy under same latency constraints, while reducing orders of magnitude GPU hours and CO 2 emission. The contributions of this paper are: • We devise an end-to-end methodology EMS to jointly perform NAS-pruning-quantization, thus unifying the conventionally separated stages into an integrated solution. • We propose a predictor-transfer method to tackle the high cost of the quantization-aware accuracy predictor's dataset collection NN architecture, quantization policy, accuracy . • Such end-to-end method can efficiently search efficient models. With the supernet and the quantization-aware accuracy predictor, it only takes minutes to search a compact model for a new platform, enabling automatic model adjustment in diverse deployment scenarios. We propose EMS, an end-to-end design method for architecting mixed-precision model. Unlike former works that decouple into separated stages, we directly search for the optimal mixed-precision architecture without multi-stage optimization. We use predictor-base method that can have no extra evaluation for target dataset, which greatly saves GPU hours for searching under an upcoming scenario, thus reducing marginally CO 2 emission and cloud compute cost. To tackle the problem for high expense of data collection, we propose predictor-transfer technique to make up for the limitation of data. Comparisons with state-of-the-art models show the necessity of joint optimization and prosperity of our end-to-end design method.
We present an end-to-end design methodology for efficient deep learning deployment. 
{"src": "scitldr-AIC_train", "paper_id": "Bkgr2kHKDH"}