a softmax-transformer initialized
from a pretrained an Rfa-Gaussian model.
If Rfa's good performance can be attributed
to learning a good approximation to softmax,
both, without finetunining,
should perform similarly to the pretrained models.
However, this is not the case on IWSLT14 DE-EN.
Both pretrained models achieve more than 35.2 development set BLEU.
In contrast, (i) and (ii) respectively get 2.3 and 1.1 BLEU without finetuning,
hardly beating a randomly-initialized untrained model.
This result aligns with the observation by [1]},
and suggests that it is not the case
that Rfa performs well because it learns
to imitate softmax attention's outputs.
<FIGURE>Knowledge Transfer from Softmax Attention to RFA
We first supplement the observation in Appendix REF 
by finetuning (i) on the same pretraining data.
Figure REF  plots the learning curves.
It takes Rfa roughly 1,500 steps to reach similar training
loss to the pretrained model.
As a baseline, “Rfa Reset”
resets the multihead attention parameters
(i.e., those for query, key, value, and output projections)
to randomly initialized ones.
Its learning curve is similar to that of (i),
suggesting that the pretrained multihead attention parameters
are no more useful to Rfa than randomly initialized ones.
To further confirm this observation,
“softmax Reset” resets the multihead attention parameters
without changing the attention functions.
It converges to the pretraining loss in less than 200 steps.
Takeaway.
From the above results on IWSLT14,
pretrained knowledge in a softmax transformer
cannot be directly transferred to an Rfa model.
However, from Figure REF  and
a much larger-scale experiment by [1]},
we do observe that
Rfa can recover the pretraining loss,
and the computation cost of finetuning
is much less than training a model from scratch.
This suggests some potential applications.
For example, one might be able to initialize an Rfa language model
from a softmax transformer pretrained on large-scale data
(e.g., GPT-3; [3]}),
and finetune it at a low cost.
The outcome would be an Rfa model retaining most of the pretraining knowledge,
but is much faster and more memory-friendly to sample from.
We leave such exploration to future work.
