| license: cc-by-nc-sa-4.0 | |
| pipeline_tag: fill-mask | |
| language: en | |
| tags: | |
| - legal | |
| - long-documents | |
| # Legal Longformer (base) | |
| This is a derivative model based on the [LexLM (base)](https://huggingface.co/lexlms/roberta-base-cased) RoBERTa model. All model parameters where cloned from the original model, while the positional embeddings were extended by cloning the original embeddings multiple times following [Beltagy et al. (2020)](https://arxiv.org/abs/2004.05150) using a python script similar to this one (https://github.com/allenai/longformer/blob/master/scripts/convert_model_to_long.ipynb). |