
We conduct quantitative analysis to reveal the root cause of FLOPs-latency nonproportionality on DC accelerators. Although recent work [1]} has observed the similar behavior, our roofline model and analysis is the first to show the fundamental reasons for latency to be much less correlated to FLOPs on GPUs and TPUs than on CPUs. Moreover, our analysis also discovers the performance bottlenecks of CNNs and inspires enhancements for both NAS and compound model scaling.

We design a DC-accelerator-optimized search space, with space-to-batch, space-to-depth, fused convolution structures, and block-wise activation functions, to compose CNNs with higher operational intensity and efficiency for better accuracy and speed trade-offs.

We propose latency-aware compound scaling (LACS), the first compound scaling method with accuracy and latency as the multi-objective. Aftering taking into latency into account, our LACS discovers network depth should grow much faster than image size and network width, which is quite different from previous single-objective compound scaling [2]}.

Our unified NAS and LACS produce EfficientNet-X, with up to 2X speedup over the EfficientNet and up to 7X speedup over RegNet/ResNeSt on TPUs and GPUs.

