
We thus employ any generic convex loss function and standard weight-decay in our formulation. While direct convex analysis when \(\sigma (\cdot )\)  represents the softmax activation is intractable, we can analyze this architecture for many other activation functions. In particular, self-attention with both linear and ReLU activation functions has been proposed with performance on par to standard softmax activation networks [1]}, [2]}, [3]}, [4]}. Thus, we will analyze linear, ReLU, and gated ReLU activation variants of the multi-head self-attention.
