Cross Document Language Modeling
We introduce a new pretraining approach for language models that are geared to support multi document NLP tasks. Our crossdocument language model (CD LM) improves masked language modeling for these tasks with two key ideas. First, we pretrain with multiple related documents in a single input, via cross document masking, which encourages the model to learn cross document and long range relationships. Second, extending the recent Longformer model, we pretrain with long contexts of several thousand tokens and introduce a new attention pattern that uses sequence level global attention to predict masked tokens, while retaining the familiar local attention elsewhere. We show that our CD LM sets new state of the art results for several multi text tasks, including crossdocument event and entity coreference resolution, paper citation recommendation, and documents plagiarism detection, while using a significantly reduced number of training parameters relative to prior works1.
ArXiv