We train an example retriever model (EgRet) on our extracted exemplification dataset from training portion of ELI5 [1]}, which contains \(65K\)  extracted context-example pairs.
Our retriever consists of dual Transformer encoders [2]}, one to encode the context query and the other to encode candidate examples (Figure REF ). Both encoders are initialized as pretrained RoBERTa-base models [3]}, as in the dense-RELiC model of [4]}. To obtain a query embedding \(c_i\), we feed the query encoder the surrounding context of a masked-out example, as in Figure REF . Similarly, we use the other encoder to compute embeddings of the ground-truth example (\(e_i^+\)) as well as negative examples sampled from other contexts, forming a set \(E\)  of example embeddings. We fine-tune both encoders in EgRet with a contrastive learning objective [5]}, [6]}:
\(\mathcal {L}(\theta ) &= - \sum _{(\leavevmode {\color {mypurple}c_i}, \leavevmode {\color {mygreen}e_i}) \in E} \log \frac{\exp \leavevmode {\color {mypurple}c_i} \cdot \leavevmode {\color {mygreen}e_i^+} }{\sum _{\leavevmode {\color {mygreen}e_j} \in E} \exp \leavevmode {\color {mypurple} c_i} \cdot \leavevmode {\color {mygreen}e_j} }\) 
