task_categories:fill-mask annotations_creators:other language_creators:found multilinguality:multilingual size_categories:1M<n<10M source_datasets:original language:bg language:cs language:da language:de language:el language:en language:es language:et language:fi language:fr language:ga language:hr language:hu language:it language:lt language:lv language:mt language:nl language:pl language:pt language:ro language:sk language:sl language:sv license:cc-by-4.0 region:us
---
annotations_creators:
- other
language_creators:
- found
language:
- bg 
- cs 
- da 
- de 
- el 
- en
- es 
- et 
- fi 
- fr 
- ga
- hr 
- hu 
- it 
- lt 
- lv 
- mt
- nl 
- pl 
- pt 
- ro 
- sk 
- sl 
- sv
license:
- cc-by-4.0
multilinguality:
- multilingual
paperswithcode_id: null
pretty_name: "EurlexResources: A Corpus Covering the Largest EURLEX Resources"
size_categories:
- 1M<n<10M
source_datasets:
- original
task_categories:
- fill-mask

---

# Dataset Card for EurlexResources: A Corpus Covering the Largest EURLEX Resources

## Table of Contents

- [Table of Contents](#table-of-contents)
- [Dataset Description](#dataset-description)
    - [Dataset Summary](#dataset-summary)
    - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
    - [Languages](#languages)
- [Dataset Structure](#dataset-structure)
    - [Data Instances](#data-instances)
    - [Data Fields](#data-fields)
    - [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
    - [Curation Rationale](#curation-rationale)
    - [Source Data](#source-data)
    - [Annotations](#annotations)
    - [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
    - [Social Impact of Dataset](#social-impact-of-dataset)
    - [Discussion of Biases](#discussion-of-biases)
    - [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
    - [Dataset Curators](#dataset-curators)
    - [Licensing Information](#licensing-information)
    - [Citation Information](#citation-information)
    - [Contributions](#contributions)

## Dataset Description

- **Homepage:**
- **Repository:** [GitHub](https://github.com/JoelNiklaus/LegalDatasets/tree/main/pretrain/eurlex)
- **Paper:** 
- **Leaderboard:**
- **Point of Contact:** [Joel Niklaus](mailto:joel.niklaus.2@bfh.ch)

### Dataset Summary

This dataset contains large text resources (~179GB in total) from EURLEX that can be used for pretraining language models.

Use the dataset like this:
```python
from datasets import load_dataset
config = "de_caselaw" # {lang}_{resource}
dataset = load_dataset("joelito/eurlex_resources", config, split='train', streaming=True) 
```

### Supported Tasks and Leaderboards

The dataset supports the task of masked language modeling.

### Languages

The following languages are supported: bg, cs, da, de, el, en, es, et, fi, fr, ga, hr, hu, it, lt, lv, mt, nl, pl, pt, ro, sk, sl, sv

## Dataset Structure

### Data Instances

The file format is jsonl.xz and there is one split available ("train"). 
The following resource types are supported: caselaw, decision, directive, intagr, proposal, recommendation, regulation

More information about the resource types can be found here:
- Caselaw: [EU](https://eur-lex.europa.eu/collection/eu-law/eu-case-law.html)
- Decision: [EU](https://eur-lex.europa.eu/EN/legal-content/summary/european-union-decisions.html), [Wikipedia](https://en.wikipedia.org/wiki/Decision_(European_Union))
- Directive: [EU](https://european-union.europa.eu/institutions-law-budget/law/types-legislation_en), [Wikipedia](https://en.wikipedia.org/wiki/Directive_(European_Union))
- Recommendation: [EU](https://eur-lex.europa.eu/EN/legal-content/glossary/recommendation.html), [Wikipedia](https://en.wikipedia.org/wiki/Recommendation_(European_Union))
- Regulation: [EU](https://european-union.europa.eu/institutions-law-budget/law/types-legislation_en), [Wikipedia](https://en.wikipedia.org/wiki/Regulation_(European_Union))
- Intagr: [EU](https://eur-lex.europa.eu/collection/eu-law/inter-agree.html), [Wikipedia](https://en.wikipedia.org/wiki/Treaties_of_the_European_Union)
- Proposal: No resource found


| Source             |   Size (MB) |       Words |   Documents |   Words/Document |
|:-------------------|------------:|------------:|------------:|-----------------:|
| all_all            |      180668 | 12106556233 |     8306749 |             1457 |
| all_caselaw        |       34939 |  3413551598 |     2487794 |             1372 |
| all_decision       |       28519 |  1698585620 |     1267402 |             1340 |
| all_directive      |        4786 |   368577940 |      104187 |             3537 |
| all_intagr         |       11421 |   743271516 |      274485 |             2707 |
| all_proposal       |       26526 |  2087989530 |      702392 |             2972 |
| all_recommendation |        1886 |   164979037 |       80277 |             2055 |
| all_regulation     |       72590 |  3629600992 |     3390212 |             1070 |
| bg_all             |        7819 |   398067053 |      348691 |             1141 |
| bg_caselaw         |        1588 |   109749174 |      104434 |             1050 |
| bg_decision        |        1248 |    58817972 |       54075 |             1087 |
| bg_directive       |         263 |    15731608 |        4388 |             3585 |
| bg_intagr          |         603 |    31292848 |       11581 |             2702 |
| bg_proposal        |        1083 |    60674956 |       29251 |             2074 |
| bg_recommendation  |          89 |     5588991 |        3321 |             1682 |
| bg_regulation      |        2943 |   116211504 |      141641 |              820 |
| cs_all             |        8360 |   471961631 |      449793 |             1049 |
| cs_caselaw         |        1163 |   110005022 |      104519 |             1052 |
| cs_decision        |        1102 |    58921128 |       54075 |             1089 |
| cs_directive       |         186 |    13951134 |        4388 |             3179 |
| cs_intagr          |         449 |    28106332 |       11581 |             2426 |
| cs_proposal        |         840 |    61838692 |       29252 |             2113 |
| cs_recommendation  |          64 |     5416549 |        3323 |             1630 |
| cs_regulation      |        4557 |   193722774 |      242655 |              798 |
| da_all             |        8932 |   671484862 |      332500 |             2019 |
| da_caselaw         |        1746 |   185589641 |       88234 |             2103 |
| da_decision        |        1356 |    89498535 |       54085 |             1654 |
| da_directive       |         207 |    17525792 |        4388 |             3994 |
| da_intagr          |         506 |    35596169 |       11582 |             3073 |
| da_proposal        |        1399 |   119759476 |       29257 |             4093 |
| da_recommendation  |         100 |     9463897 |        3352 |             2823 |
| da_regulation      |        3618 |   214051352 |      141602 |             1511 |
| de_all             |        9607 |   695512401 |      348290 |             1996 |
| de_caselaw         |        1930 |   193232441 |      104228 |             1853 |
| de_decision        |        1449 |    93688222 |       53980 |             1735 |
| de_directive       |         218 |    17337760 |        4385 |             3953 |
| de_intagr          |         531 |    36791153 |       11580 |             3177 |
| de_proposal        |        1556 |   126987454 |       29219 |             4346 |
| de_recommendation  |         109 |     9608034 |        3318 |             2895 |
| de_regulation      |        3813 |   217867337 |      141580 |             1538 |
| el_all             |       12469 |   696216541 |      349667 |             1991 |
| el_caselaw         |        2951 |   202027703 |      105138 |             1921 |
| el_decision        |        1823 |    94919886 |       54150 |             1752 |
| el_directive       |         321 |    19411959 |        4390 |             4421 |
| el_intagr          |         701 |    38965777 |       11584 |             3363 |
| el_proposal        |        2085 |   128005737 |       29290 |             4370 |
| el_recommendation  |         145 |     9344866 |        3357 |             2783 |
| el_regulation      |        4443 |   203540613 |      141758 |             1435 |
| en_all             |        9217 |   769465561 |      348641 |             2207 |
| en_caselaw         |        1846 |   222891827 |      104422 |             2134 |
| en_decision        |        1504 |   114626013 |       54054 |             2120 |
| en_directive       |         204 |    18860876 |        4388 |             4298 |
| en_intagr          |         499 |    39029843 |       11581 |             3370 |
| en_proposal        |        1538 |   140781768 |       29242 |             4814 |
| en_recommendation  |          97 |    10091809 |        3320 |             3039 |
| en_regulation      |        3530 |   223183425 |      141634 |             1575 |
| es_all             |        8588 |   725125274 |      348443 |             2081 |
| es_caselaw         |        1870 |   220621730 |      104312 |             2115 |
| es_decision        |        1334 |    98163499 |       54001 |             1817 |
| es_directive       |         221 |    21484479 |        4385 |             4899 |
| es_intagr          |         516 |    41841805 |       11581 |             3612 |
| es_proposal        |        1366 |   133674486 |       29224 |             4574 |
| es_recommendation  |          82 |     8864018 |        3319 |             2670 |
| es_regulation      |        3199 |   200475257 |      141621 |             1415 |
| et_all             |        6090 |   328068754 |      349615 |              938 |
| et_caselaw         |        1074 |    93096396 |      105111 |              885 |
| et_decision        |        1069 |    50752324 |       54159 |              937 |
| et_directive       |         177 |    11555930 |        4390 |             2632 |
| et_intagr          |         436 |    24018147 |       11584 |             2073 |
| et_proposal        |         810 |    51600852 |       29283 |             1762 |
| et_recommendation  |          61 |     4451369 |        3355 |             1326 |
| et_regulation      |        2464 |    92593736 |      141733 |              653 |
| fi_all             |        7346 |   404265224 |      349633 |             1156 |
| fi_caselaw         |        1596 |   126525296 |      105119 |             1203 |
| fi_decision        |        1227 |    59659475 |       54163 |             1101 |
| fi_directive       |         204 |    12766491 |        4389 |             2908 |
| fi_intagr          |         463 |    25392311 |       11584 |             2192 |
| fi_proposal        |        1075 |    69198401 |       29288 |             2362 |
| fi_recommendation  |          73 |     5070392 |        3356 |             1510 |
| fi_regulation      |        2707 |   105652858 |      141734 |              745 |
| fr_all             |        9937 |   828959218 |      348295 |             2380 |
| fr_caselaw         |        2158 |   246262666 |      104228 |             2362 |
| fr_decision        |        1473 |   108648744 |       53981 |             2012 |
| fr_directive       |         222 |    20308801 |        4385 |             4631 |
| fr_intagr          |         536 |    41986012 |       11580 |             3625 |
| fr_proposal        |        1592 |   149134298 |       29218 |             5104 |
| fr_recommendation  |         112 |    11510415 |        3318 |             3469 |
| fr_regulation      |        3845 |   251108282 |      141585 |             1773 |
| ga_all             |        1028 |    65030095 |      349778 |              185 |
| ga_caselaw         |          11 |      696305 |      105205 |                6 |
| ga_decision        |          87 |     4415457 |       54189 |               81 |
| ga_directive       |          18 |     1512027 |        4390 |              344 |
| ga_intagr          |          19 |     1820723 |       11586 |              157 |
| ga_proposal        |         289 |    26106889 |       29298 |              891 |
| ga_recommendation  |          10 |      902390 |        3361 |              268 |
| ga_regulation      |         594 |    29576304 |      141749 |              208 |
| hr_all             |        4594 |   258816068 |      348691 |              742 |
| hr_caselaw         |         617 |    62432734 |      104434 |              597 |
| hr_decision        |         596 |    31911903 |       54075 |              590 |
| hr_directive       |         156 |    10855913 |        4388 |             2474 |
| hr_intagr          |         450 |    24962086 |       11581 |             2155 |
| hr_proposal        |         552 |    33437815 |       29251 |             1143 |
| hr_recommendation  |          40 |     3612247 |        3321 |             1087 |
| hr_regulation      |        2183 |    91603370 |      141641 |              646 |
| hu_all             |        6653 |   375253894 |      349605 |             1073 |
| hu_caselaw         |        1278 |   110179375 |      105144 |             1047 |
| hu_decision        |        1147 |    57108172 |       54156 |             1054 |
| hu_directive       |         200 |    13568304 |        4389 |             3091 |
| hu_intagr          |         470 |    27258501 |       11586 |             2352 |
| hu_proposal        |         912 |    60882750 |       29291 |             2078 |
| hu_recommendation  |          70 |     5312868 |        3357 |             1582 |
| hu_regulation      |        2576 |   100943924 |      141682 |              712 |
| it_all             |        9586 |   768605772 |      333631 |             2303 |
| it_caselaw         |        1889 |   206117726 |       89560 |             2301 |
| it_decision        |        1445 |   102848859 |       53983 |             1905 |
| it_directive       |         217 |    19687773 |        4385 |             4489 |
| it_intagr          |         528 |    40134330 |       11580 |             3465 |
| it_proposal        |        1533 |   140713925 |       29218 |             4816 |
| it_recommendation  |         109 |    10923431 |        3318 |             3292 |
| it_regulation      |        3865 |   248179728 |      141587 |             1752 |
| lt_all             |        6400 |   364361783 |      200565 |             1816 |
| lt_caselaw         |        1137 |   101808706 |      105477 |              965 |
| lt_decision        |        1096 |    55850308 |       21990 |             2539 |
| lt_directive       |         185 |    13078983 |        3239 |             4037 |
| lt_intagr          |         452 |    27009631 |        7481 |             3610 |
| lt_proposal        |         850 |    58553579 |       29272 |             2000 |
| lt_recommendation  |          64 |     5121089 |        3363 |             1522 |
| lt_regulation      |        2617 |   102939487 |       29743 |             3460 |
| lv_all             |        6349 |   363239195 |      349919 |             1038 |
| lv_caselaw         |        1153 |   103456811 |      105242 |              983 |
| lv_decision        |        1103 |    55512944 |       54224 |             1023 |
| lv_directive       |         186 |    13023024 |        4392 |             2965 |
| lv_intagr          |         452 |    26693107 |       11630 |             2295 |
| lv_proposal        |          96 |    58176216 |       29298 |             1985 |
| lv_recommendation  |          64 |     5074494 |        3361 |             1509 |
| lv_regulation      |        2545 |   101302599 |      141772 |              714 |
| mt_all             |        6540 |   367834815 |      350292 |             1050 |
| mt_caselaw         |        1164 |   100423543 |      105479 |              952 |
| mt_decision        |        1109 |    55239141 |       54280 |             1017 |
| mt_directive       |         203 |    14355266 |        4392 |             3268 |
| mt_intagr          |         470 |    27701991 |       11675 |             2372 |
| mt_proposal        |         878 |    59749277 |       29274 |             2041 |
| mt_recommendation  |          65 |     5039600 |        3363 |             1498 |
| mt_regulation      |        2650 |   105325997 |      141829 |              742 |
| nl_all             |        9586 |   770312808 |      349407 |             2204 |
| nl_caselaw         |        1847 |   206271837 |      105005 |             1964 |
| nl_decision        |        1456 |   104060901 |       54152 |             1921 |
| nl_directive       |         217 |    19529361 |        4388 |             4450 |
| nl_intagr          |         529 |    40247634 |       11584 |             3474 |
| nl_proposal        |        1540 |   141258274 |       29279 |             4824 |
| nl_recommendation  |         111 |    11002405 |        3355 |             3279 |
| nl_regulation      |        3886 |   247942396 |      141644 |             1750 |
| pl_all             |        6677 |   406648795 |      350349 |             1160 |
| pl_caselaw         |        1231 |   115824759 |      105479 |             1098 |
| pl_decision        |        1125 |    60407576 |       54287 |             1112 |
| pl_directive       |         197 |    14672157 |        4392 |             3340 |
| pl_intagr          |         466 |    28543668 |       11680 |             2443 |
| pl_proposal        |         886 |    64728230 |       29317 |             2207 |
| pl_recommendation  |          68 |     5769893 |        3363 |             1715 |
| pl_regulation      |        2703 |   116702512 |      141831 |              822 |
| pt_all             |        8450 |   675152149 |      348449 |             1937 |
| pt_caselaw         |        1763 |   198084937 |      104312 |             1898 |
| pt_decision        |        1327 |    93278293 |       54007 |             1727 |
| pt_directive       |         217 |    19831549 |        4385 |             4522 |
| pt_intagr          |         504 |    37999753 |       11581 |             3281 |
| pt_proposal        |        1361 |   127461782 |       29224 |             4361 |
| pt_recommendation  |          81 |     8396661 |        3319 |             2529 |
| pt_regulation      |        3197 |   190099174 |      141621 |             1342 |
| ro_all             |        6315 |   415038571 |      350300 |             1184 |
| ro_caselaw         |        1110 |   114780999 |      105516 |             1087 |
| ro_decision        |        1047 |    59479553 |       54281 |             1095 |
| ro_directive       |         206 |    16101628 |        4392 |             3666 |
| ro_intagr          |         481 |    31497000 |       11675 |             2697 |
| ro_proposal        |         805 |    62130419 |       29274 |             2122 |
| ro_recommendation  |          63 |     5977913 |        3363 |             1777 |
| ro_regulation      |        2603 |   125071059 |      141799 |              882 |
| sk_all             |        6484 |   392235510 |      350570 |             1118 |
| sk_caselaw         |        1160 |   110125141 |      105608 |             1042 |
| sk_decision        |        1111 |    59576875 |       54349 |             1096 |
| sk_directive       |         188 |    14132755 |        4393 |             3217 |
| sk_intagr          |         458 |    28298155 |       11676 |             2423 |
| sk_proposal        |         859 |    63726047 |       29290 |             2175 |
| sk_recommendation  |          66 |     5654790 |        3364 |             1680 |
| sk_regulation      |        2642 |   110721747 |      141890 |              780 |
| sl_all             |        6222 |   394814289 |      350574 |             1126 |
| sl_caselaw         |        1071 |   111238184 |      105608 |             1053 |
| sl_decision        |        1075 |    59454906 |       54349 |             1093 |
| sl_directive       |         176 |    13908097 |        4393 |             3165 |
| sl_intagr          |         441 |    28239078 |       11676 |             2418 |
| sl_proposal        |         812 |    63391970 |       29290 |             2164 |
| sl_recommendation  |          62 |     5628775 |        3364 |             1673 |
| sl_regulation      |        2585 |   112953279 |      141894 |              796 |
| sv_all             |        7419 |   500085970 |      351051 |             1424 |
| sv_caselaw         |        1585 |   162108645 |      105980 |             1529 |
| sv_decision        |        1213 |    71744934 |       54357 |             1319 |
| sv_directive       |         195 |    15386273 |        4393 |             3502 |
| sv_intagr          |         463 |    29845462 |       11676 |             2556 |
| sv_proposal        |        1059 |    86016237 |       29292 |             2936 |
| sv_recommendation  |          79 |     7152141 |        3366 |             2124 |
| sv_regulation      |        2825 |   127832278 |      141987 |              900 |

### Data Fields

[More Information Needed]

### Data Splits

[More Information Needed]

## Dataset Creation


### Curation Rationale

[More Information Needed]

### Source Data

#### Initial Data Collection and Normalization

The data has been downloaded using the R package [eurlex](https://cran.r-project.org/web/packages/eurlex/vignettes/eurlexpkg.html) between June and August 2022.


#### Who are the source language producers?

[More Information Needed]


### Annotations

#### Annotation process

[More Information Needed]

#### Who are the annotators?

[More Information Needed]

### Personal and Sensitive Information

[More Information Needed]

## Considerations for Using the Data

### Social Impact of Dataset

[More Information Needed]

### Discussion of Biases

[More Information Needed]

### Other Known Limitations

[More Information Needed]

## Additional Information

### Dataset Curators

[More Information Needed]

### Licensing Information

[CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)
[see also the legal notice](https://eur-lex.europa.eu/content/legal-notice/legal-notice.html)

### Citation Information

[More Information Needed]

### Contributions

Thanks to [@JoelNiklaus](https://github.com/joelniklaus) for adding this dataset.
