The RALM architecture initially used a single document for retrieval augmentation. However, it was discovered that RALM's performance significantly improved when the number of retrieved paragraphs was increased. (Izacard and Grave, 2020) Therefore, they proposed a new method called Fusion-in-Decoder (FiD) which involves keeping the retriever unchanged, using the encoder in LM to encode the related documents one by one, and then connecting the related documents and giving them to the decoder for output. Then Hofstatter et al. (2023) improved on the FiD. They constrained the information flow from encoder to decoder. FiD-Light with reranking was also tuned via text source pointers to improve the topmost source accuracy. Izacard and Grave (2020) applied knowledge distillation to the FiD model, also known as FiD-KD, using cross-attention scores from a sequence-to-sequence reader to obtain synthetic targets for the retriever. Singh et al. (2021) proposed an enhancement approach that differs from knowledge distillation in that it uses an end-to-end training approach requiring fewer documents, training cycles, and no supervised initialization compared to FiD-KD.