However, the architecture modules that were developed for convolutional models only improve attention models by a small amount.
When Squeeze-and-Excitation (SE) and SiLU/Swish-1 are added, ResNet improves by 1.3% while HaloNet only improves by 0.4%. We speculate that HaloNet models benefit from the gating and multiplicative interactions that comprise self-attention and do not need explicit gating such as SE. Further research must be conducted in order to discover architecture modules that can consistently improve a variety of self-attention models. Inspired by these findings, we decided to use label smoothing, SiLU/Swish-1, and RandAugment in our HaloNet \(H0-H7\)  models. We also use stochastic depth for our larger models [1]}, [2]}.
<TABLE>