Since the ResNet structure was initially designed for convolutions, we suspect that designing architectures specifically for attention may improve HaloNet. In our work we maintained homogeneity across all layers of model for hyperparameters such as the block (\(b)\)  and halo (\(h\) ) sizes. We also hope that using automated architecture search methods [1]} to optimize these hyperparameters for specific accelerators will lead to better local attention architectures. In our work, we train with comparable image sizes as EfficientNet models to determine if attention models can scale to larger images.
