Jürgen Schmidhuber Transformer self-attention feedforward neural network gradient descent outer products self-attention
In 1992, Jürgen Schmidhuber also published an alternative to RNNs which is now called a linear Transformer or a  Transformer with linearized self-attention (save for a normalization operator). It learns internal spotlights of attention: a slow feedforward neural network learns by gradient descent to control the fast weights of another neural network through outer products of self-generated activation patterns FROM and TO (which are now called key and value for self-attention). This fast weight attention mapping is applied to a query pattern.
