The high-level idea to overcome this problem is that, instead of normalizing logits relative to a reference token, we shall normalize the logits to be logprobs. That is, we recover the logits with the normalization \(\sum_{j}\exp(z_{j})=1\). With this normalization it is no longer necessary to include a reference token in every query.