Since our encoder model is pretrained on text (we utilize T5), we must reformat \(\mathbf{v}\) to be fed to a language encoder. The simplest method is to project \(\mathbf{v}\) to \(\mathbb{R}^{d}\) and feed it as an input hidden vector. However, given the large size of the vocabulary \(|\mathcal{V}|\) and the fact that it has been passed through a softmax, this would cause a large reduction in rank and a loss of information8.