Convex neural networks. Starting with [1]}, there has been a long line of work demonstrating that various ReLU-activation neural network architectures have equivalent convex optimization problems. These include two-layer convolutional and vector-output networks [2]}, [3]}, [4]}, as well as deeper ones [5]}, networks with Batch Normalization [6]}, and Wasserstein GANs [7]}. Recent work has also demonstrated how to efficiently optimize the simplest forms of these equivalent convex networks, and incorporate additional constraints for adversarial robustness [8]}, [9]}. However, none of these works have analyzed the building blocks of transformers, which are a leading method in many state-of-the-art vision and language processing tasks.
