This trade-off is more complicated in convolution structures such as mobile inverted bottleneck conv (MBConv) [1]}, an important convolution structure in the baseline search space. MBConv is a macro block that includes a expansion layer of 1x1 convolutions, a depthwise convolution, and a projection layer of 1x1 convolutions, together with activation, batch normalization, and skip-connections. A fused variant of MBConv (fused MBConv) combines the depthwise convolutions with the expansion or projection layer as a vanilla convolution. These trade-offs involving \(W\) , \(I\) , and \(E\)  (as shown in Equation REF  and REF ) are too complicated for manual optimization but are well-suited for NAS to explore. Concretely, fused MBConv has higher operational intensity (good for speed) but higher FLOPs (bad for speed) than MBConv. Thus, fused MBConv can possibly be either faster or slower than MBConv, depending on the shape and size of weights and activations of the macro op. Moreover, the fused MBConv andMBConv contribute differently to the final model accuracy.  Thus, we added the fused MBConv into the baseline factorized search space [2]}. Although recently NAS for mobile devices [3]} also uses a similar op, our work is the first to provide the in-depth analysis and employ such ops into the DC accelerator search spaces. Our search indeed finds a combination of fused MBConv and MBConv to get the models with Pareto-optimized latency and accuracy as shown in Table REF .
