In this paper, we aim at bridging this gap and designing model families with high accuracy and inference speed, by taking into consideration hardware architecture details of TPUs and GPUs for both NAS and model scaling. We first analyze DC accelerators to find performance bottlenecks. Our analysis reveals the root cause of the recent observed FLOPs-latency nonpropotionality [1]}. We discover that SOTA CNNs suffer from low operational intensity and parallelism, which causes low computation rate (, FLOPs/sec or Ops/secWhen operations are done in different data types such as bfloat16 [2]}, float16 [3]}, and tf32 [4]}, the computation rate is usually denoted as OPS, , OPs/Second. Hereafter in this paper, we use FLOPs/sec and Ops/sec interchangeably unless noted otherwise.) and sub-optimal inference latency and throughput on TPU/GPU accelerators. With these insights, we augment state-of-the-art (SOTA) NAS with DC accelerator optimized search space to improve CNN model operational intensity and efficiency. Concretely, we create a new search space with accelerator-friendly operations including space-to-depth, space-to-batch, fused convolution structures, and block-wise searchable activation as shown in Figure REF . We propose latency-aware compound scaling (LACS) that uses a multi-objective of both accuracy and inference speed to search for scaling factors to generate a model family. LACS is the first compound scaling method with a multi-objective including both latency and accuracy.
