Attention Mechanisms, Transformers, and Autoencoders:
Attention mechanisms allow neural networks to focus on and extract important features from long temporal sequences. Multi-head self-attention forms the central component in transformers, extracting domain-invariant long-term dependencies from sequences without recurrent units while being efficient and parallelizable [1]}. Attention condensers are lightweight, self-contained, and standalone attention mechanisms independent of local context convolution kernels that learn condensed embeddings of the semantics of both local and cross-channel activations [2]}. Each module contains an encoder-decoder architecture coupled with a self-attention mechanism. Coupled with machine design exploration, attention condensers have been used for image classification (4.17\(\times \)  fewer parameters than MobileNetsV1)  [3]}, keyword spotting (up to 507\(\times \)  fewer parameters over previous work) [2]}, and semantic segmentation (72\(\times \)  fewer parameters over RefineNet and EdgeSegNet) [5]} at the edge. Long-short range attention (LSRA) uses two heads (convolution and attention) to capture both local and global context, expanding the bottleneck while using condensed embeddings to reduce computation cost [6]}. Combined with pruning and quantization, LSRA transformers can be 18\(\times \)  smaller than the vanilla transformer architecture. MobileViT combines the benefits of convolutional networks and transformers by replacing local processing in convolution with global processing, allowing lightweight and low-latency transformers to be implemented using convolution [7]}. Instead of using special attention and transformer blocks, transformer knowledge distillation teaches a small transformer to mimic the behavior of a larger transformer, allowing up to 7.5\(\times \)  smaller and 9.4\(\times \)  faster inference over bidirectional encoder representations from transformers [8]}. Customized data layout and loop reordering of each attention kernel, coupled with quantization, has allowed porting transformers onto microcontrollers [9]} by minimizing computationally intensive data marshaling operations. The use of depthwise and pointwise convolution has been shown to yield autoencoder architectures as small as 2.7 kB for anomaly detection [10]}.
