Lightweight Spatial Convolution:
SqueezeNet [1]} brought on several micro-architectural enhancements to AlexNet [2]}. These include replacing 3\(\times \) 3 kernels with point-wise filters, decreasing input channel count using point-wise filters as a linear bottleneck, and late downsampling to enhance feature maps. The resulting network consists of stacked "fire modules", with each module containing a bottleneck layer (layer with point-wise filters) and an excitation layer (mix of point-wise and 3\(\times \) 3 filters). Using pruning, quantization, and encoding, SqueezeNet reduced the size of AlexNet by 510\(\times \)  (\(<\)  0.5 MB). MobileNetsV1 [3]} introduced depthwise separable convolution [4]} (channel-wise convolution followed by bottleneck layer), and width and resolution multipliers to control layer width and input resolution of AlexNet. Depth-wise separable convolution is 9\(\times \)  cheaper and induces 7-9\(\times \)  memory savings over 3\(\times \) 3 kernels. MobileNetV2 [5]} introduced the concepts of inverted residuals and linear bottleneck, where a residual connection exists between bottleneck layers rather than excitation layers and a linear output is enforced at the last convolution of a residual block. To reduce channel count, the depthwise separable convolution layer can be enclosed between the pointwise group convolution layer with channel shuffle, thereby improving the semantic relation between input and output channels across all the groups through the use of wide activation maps [6]}. Instead of having residual connections across two layers, the gradient highway can act as a medium to feed each layer activation maps of all preceding layers. This is known as channel-wise feature concatenation [7]} and encourages reuse and stronger propagation of low-complexity diversified feature maps and gradients while drastically reducing network parameter count.
