task_categories:text-generation task_categories:fill-mask task_ids:masked-language-modeling annotations_creators:no-annotation language_creators:found multilinguality:monolingual size_categories:1K<n<10K source_datasets:extended language:en license:cc-by-nc-sa-4.0 legal law arxiv:2305.07507 region:us
LegalLAMA: Legal LAnguage Model Analysis (LAMA) (LAMA) dataset.
@inproceedings{chalkidis-garneau-etal-2023-lexlms,
    title = {{LeXFiles and LegalLAMA: Facilitating English Multinational Legal Language Model Development}},
    author = "Chalkidis*, Ilias and 
              Garneau*, Nicolas and
              Goanta, Catalina and 
              Katz, Daniel Martin and 
              Søgaard, Anders",
    booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics",
    month = july,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/xxx",
}
---
annotations_creators:
- no-annotation
language_creators:
- found
language:
- en
license:
- cc-by-nc-sa-4.0
multilinguality:
- monolingual
size_categories:
- 1K<n<10K
source_datasets:
- extended
task_categories:
- text-generation
- fill-mask
task_ids:
- masked-language-modeling
pretty_name: LegalLAMA
tags:
- legal
- law
---

# Dataset Card for "LegalLAMA"

## Table of Contents
- [Dataset Description](#dataset-description)
  - [Dataset Summary](#dataset-summary)
  - [Dataset Specifications](#supported-tasks-and-leaderboards)
 
## Dataset Description
- **Homepage:** https://github.com/coastalcph/lexlms
- **Repository:** https://github.com/coastalcph/lexlms
- **Paper:** https://arxiv.org/abs/2305.07507
- **Point of Contact:** [Ilias Chalkidis](mailto:ilias.chalkidis@di.ku.dk)

### Dataset Summary

LegalLAMA is a diverse probing benchmark suite comprising 8 sub-tasks that aims to assess the acquaintance of legal knowledge that PLMs acquired in pre-training.

### Dataset Specifications

| Corpus                               | Corpus alias         | Examples  | Avg. Tokens | Labels |
|--------------------------------------|----------------------|-----------|-------------|--------|
| Criminal Code Sections (Canada)      | `canadian_sections`  | 321       | 72          | 144    |
| Legal Terminology (EU)               | `cjeu_term`          | 2,127     | 164         | 23     |
| Contractual Section Titles (US)      | `contract_sections`  | 1,527     | 85          | 20     |
| Contract Types (US)                  | `contract_types`     | 1,089     | 150         | 15     |
| ECHR Articles (CoE)                  | `ecthr_articles`     | 5,072     | 69          | 13     |
| Legal Terminology (CoE)              | `ecthr_terms`        | 6,803     | 97          | 250    |
| Crime Charges (US)                   | `us_crimes`          | 4,518     | 118         | 59     |
| Legal Terminology (US)               | `us_terms`           | 5,829     | 308         | 7      |

### Usage

Load a specific sub-corpus, given the corpus alias, as presented above.

```python
from datasets import load_dataset
dataset = load_dataset('lexlms/legal_lama', name='ecthr_terms')
```

### Citation

[*Ilias Chalkidis\*, Nicolas Garneau\*, Catalina E.C. Goanta, Daniel Martin Katz, and Anders Søgaard.*
*LeXFiles and LegalLAMA: Facilitating English Multinational Legal Language Model Development.*
*2022. In the Proceedings of the 61th Annual Meeting of the Association for Computational Linguistics. Toronto, Canada.*](https://aclanthology.org/2023.acl-long.865/)
```
@inproceedings{chalkidis-etal-2023-lexfiles,
    title = "{L}e{XF}iles and {L}egal{LAMA}: Facilitating {E}nglish Multinational Legal Language Model Development",
    author = "Chalkidis, Ilias  and
      Garneau, Nicolas  and
      Goanta, Catalina  and
      Katz, Daniel  and
      S{\o}gaard, Anders",
    booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.acl-long.865",
    pages = "15513--15535",
}

```