An important task for designing QA systems is answer sentence selection (AS2) : selecting the sentence containing (or constituting)  the answer to a question from a set of retrieved relevant documents.
In this paper, we propose three novel sentence-level transformer pre-training objectives that incorporate paragraph-level semantics within and across documents, to improve the performance of transformers for AS2, and mitigate the requirement of large labeled datasets.
Specifically, the model is tasked to predict whether: (i)  two sentences are extracted from the same paragraph, (ii)  a given sentence is extracted from a given paragraph, and (iii)  two paragraphs are extracted from the same document.
Our experiments on three public and one industrial AS2 datasets demonstrate the empirical superiority of our pre-trained transformers over baseline models such as RoBERTa and ELECTRA for AS2.