In this section, we seek to compare the performance of the transformer heads we have analyzed in this work to baseline convex optimization methods. This comparison allows us to illustrate the implicit biases imposed by these novel heads in a practical example. In particular, we consider the task of training a single new block of these convex heads for performing an image classification task. This is essentially the task of transfer learning without fine-tuning existing weights of the backbone network, which may be essential in computation and memory-constrained settings at the edge. For few-shot fine-tuning transformer tasks, non-convex optimization has observed to be unstable under different random initializations [1]}. Furthermore, fine-tuning only the final layer a network is a common practice, which performs very well in spurious correlation benchmarks [2]}.
