Training SetupWe trained all of the models using the unsupervised training procedure described in Section 3.2. For each dataset, we divided the auxiliary knowledge base into equal chunks of size \(256\) by concatenating or splitting the original chunks based on their length. We also added two special tokens, \(<\)BOS\(>\) and \(<\)EOS\(>\), to demarcate the original chunks' beginnings and ends to preserve the documents' structure.