jsaizant commited on
Commit
4caddd8
1 Parent(s): 96ca2e8

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +70 -70
README.md CHANGED
@@ -300,76 +300,76 @@ Feel free to click the expand button below to see the full list of sources.
300
  <details>
301
  <summary>Data Sources</summary>
302
 
303
- | Dataset | Language | Source |
304
- |-----------------------------------------------------|------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------|
305
- | Auxiliary Mathematics Problems and Solutions (AMPS) dataset | en | Hendrycks et al., 2021 |
306
- | Aya Dataset (w/o Evaluation Suite) | eu, hr, nl, fi, ka, hu, lt, nn, ro, sk, lv, cy, bg, cs, en, fr, de, ga, mt, pl, ru, sl, sv, ca, da, et, gl, el, it, no, pt, sr, es, uk | Singh et al., 2024 |
307
- | BIGPATENT | en | Sharma et al., 2019 |
308
- | Biomedical | es | Internally generated scientific dataset: Dialnet, Scielo, CSIC, TDX, BSC, UCM |
309
- | Brazilian Portuguese Web as Corpus | pt | Wagner Filho et al., 2018 |
310
- | Bulgarian National Corpus | bg | [Link](http://old.dcl.bas.bg/dataset/BulNC.7z) |
311
- | CaBeRnet: a New French Balanced Reference Corpus | fr | Popa-Fabre et al., 2020 |
312
- | CATalog | ca | Palomar-Giner et al., 2024 |
313
- | Colossal OSCAR 1.0 | bg, ca, cs, cy, da, de, el, en, es, et, eu, fi, fr, ga, gl, hr, hu, it, lt, lv, mt, nl, nn, no, oc, pl, pt, ro, ru, sh, sk, sl, sr, sv, uk | Brack et al., 2024 |
314
- | Corpus of academic Slovene KAS 2.0 | sl | Žagar et al., 2022 |
315
- | Corpus of laws and legal acts of Ukraine | uk | [Link](https://lang.org.ua/en/corpora/#anchor7) |
316
- | Corpus of State-related content from the Latvian Web (Processed) | lv | [Link](https://catalog.elra.info/en-us/repository/browse/ELRA-W0169/) |
317
- | CorpusNÓS | gl | de-Dios-Flores et al., 2024 |
318
- | Crawl of Bulgarian news websites | bg | [Link](http://old.dcl.bas.bg/dataset/Bulgarian_news.7z) |
319
- | Croatian web corpus hrWaC 2.1 | hr | Ljubešić & Klubička, 2014 |
320
- | CURLICAT Corpus | bg, hr, hu, pl, ro, sk, sl | Váradi et al., 2022 |
321
- | DaNewsroom | da | Varab & Schluter, 2020 |
322
- | Danish GigaWord | da | Strømberg-Derczynski et al., 2021 |
323
- | DeWaC | de | [Link](https://docs.sslmit.unibo.it/doku.php?id=corpora:dewac) |
324
- | DK-CLARIN Reference Corpus of General Danish | da | [Link](https://korpus.dsl.dk/clarin/) |
325
- | Estonian National Corpus 2021 | et | Koppel & Kallas, 2022 |
326
- | Estonian Reference Corpus | et | [Link](https://www.cl.ut.ee/korpused/segakorpus/) |
327
- | EurLEX-Resources | bg, cs, da, de, el, en, es, et, fi, fr, ga, hr, hu, it, lt, lv, mt, nl, pl, pt, ro, sk, sl, sv | [Link](https://huggingface.co/datasets/joelniklaus/eurlex_resources) |
328
- | EusCrawl (w/o Wikipedia or NC-licenses) | eu | Artetxe et al., 2022 |
329
- | FineWeb-Edu (350BT subset) | en | Penedo et al., 2024 |
330
- | French Public Domain Books | fr | [Link](https://huggingface.co/datasets/PleIAs/French-PD-Books) |
331
- | French Public Domain Newspapers | fr | [Link](https://huggingface.co/datasets/PleIAs/French-PD-Newspapers) |
332
- | Greek Legal Code | el | Papaloukas et al., 2021 |
333
- | Greek Web Corpus | el | Outsios et al., 2018 |
334
- | HPLTDatasets v1 - Spanish | es | de Gibert et al., 2024 |
335
- | Irish Universal Dependencies | ga | [Link](https://universaldependencies.org/ga/index.html) |
336
- | ITWaC | it | [Link](https://docs.sslmit.unibo.it/doku.php?id=corpora:itwac) |
337
- | Korpus Malti | mt | Micallef et al., 2022 |
338
- | Korpus slovenských právnych predpisov v1.9 | sk | [Link](https://www.juls.savba.sk/data/marcell/legal-sk-20220322-1.9.ver.xz) |
339
- | Latxa Corpus v1.1 | eu | Etxaniz et al., 2024 [Link](https://huggingface.co/datasets/HiTZ/latxa-corpus-v1.1) |
340
- | Legal | es | Internally generated legal dataset: BOE, BORME, Senado, Congreso, Spanish court orders, DOGC |
341
- | MaCoCu web corpus | bg, ca, el, hr, mt, sl, sr, uk | Bañón et al., 2022 |
342
- | MARCELL Romanian legislative subcorpus v2 | ro | [Link](https://elrc-share.eu/reposMARCELL%20Romanian%20legislative%20subcorpus%20v2itory/browse/marcell-romanian-legislative-subcorpus-v2/2da548428b9d11eb9c1a00155d026706ce94a6b59ffc4b0e9fb5cd9cebe6889e/) |
343
- | MC4-Legal | bg, cs, da, de, el, en, es, et, fi, fr, ga, hu, it, lt, lv, mt, nl, pl, pt, ro, sk, sl, sv | [Link](https://huggingface.co/datasets/joelito/legal-mc4) |
344
- | NKJP-PodkorpusMilionowy-1.2 (National Corpus of Polish) | pl | Lewandowska-Tomaszczyk et al., 2013 |
345
- | Norwegian Colossal Corpus | nn, no | Kummervold et al., 2021 |
346
- | Occitan Corpus | oc | Provided by [IEA](https://www.institutestudisaranesi.cat/) |
347
- | od-justice 2.0 | sk | [Link](https://www.juls.savba.sk/data/od-justice/od-justice-2.0.ver.xz) |
348
- | Open Legal Data - German court decisions and laws | de | Ostendorff et al., 2020 |
349
- | OpenSubtitlesv2016 | bg, ca, cs, da, de, el, en, es, et, eu, fi, fr, gl, hr, it, lt, lv, nl, no, pl, pt, ro, sk, sl, sr, sv, uk | Lison & Tiedemann, 2016 |
350
- | ParlamentoPT | pt | Rodrigues et al., 2023 |
351
- | Parlamint corpus | at, bg, cz, dk, ee, es, es-ga, fi, fr, gb, gr, hr, hu, it, lv, nl, no, pl, pt, rs, se, si | Erjavec et al., 2021 |
352
- | peS2o | en | Soldaini & Lo, 2023 |
353
- | PG-19 | en | Rae et al., 2019 |
354
- | Pile of Law (selected subsets) | en | Henderson* et al., 2022 |
355
- | Polish Parliamentary Corpus / Korpus Dyskursu Parlamentarnego | pl | Ogrodniczuk, 2018 |
356
- | proof-pile | en | [Link](https://huggingface.co/datasets/hoskinson-center/proof-pile) |
357
- | RedPajama-Data T1 (StackExchange subset) | en | Computer, 2023 |
358
- | Scientific | es | Internally generated scientific dataset: Wikipedia LS, Pubmed, MeSpEn, patents, clinical cases, medical crawler |
359
- | slWaC web corpus | sl | Erjavec et al., 2015 |
360
- | SoNaR Corpus NC 1.2 | nl | [Link](https://taalmaterialen.ivdnt.org/download/tstc-sonar-corpus/) |
361
- | Spanish Crawling | ca, es, eu, gl | Relevant Spanish websites crawling |
362
- | Spanish Legal Domain Corpora | es | Gutiérrez-Fandiño et al., 2021 |
363
- | SrpKorSubset (news, legal, academic, conversation, literary) | sr | [Link](http://www.korpus.matf.bg.ac.rs/) |
364
- | Starcoder | code | Li et al., 2023 |
365
- | SYN v9: large corpus of written Czech | cs | Křen et al., 2021 |
366
- | The Danish Parliament Corpus 2009 - 2017, v1 | da | Hansen, 2018 |
367
- | The Gaois bilingual corpus of English-Irish legislation (Irish legislation) | ga | [Link](https://portulanclarin.net/repository/browse/the-gaois-bilingual-corpus-of-english-irish-legislation-processed/daeac17c9e3511ea9b7f02420a000407b83de243dc0b469aab41084386c5b80f/) |
368
- | The Pile (PhilPapers subset) | en | Gao et al., 2021 |
369
- | The Swedish Culturomics Gigaword Corpus | sv | Rødven-Eide, 2016 |
370
- | Welsh-GOV | cy | Crawling from [Link](https://www.llyw.cymru) |
371
- | Wikimedia dumps | bg, ca, cs, da, de, el, en, es, et, eu, fi, fr, ga, gl, hr, hu, it, lt, lv, mt, nl, nn, no, pl, pt, ro, sh, sk, sl, sr, uk | [Link](https://dumps.wikimedia.org/) |
372
- | Yle Finnish News Archive | fi | [Link](http://urn.fi/urn:nbn:fi:lb-2021050401) |
373
 
374
  <details>
375
  <summary>References</summary>
 
300
  <details>
301
  <summary>Data Sources</summary>
302
 
303
+ | Dataset | Language | Source |
304
+ |-----------------------------------------------|---------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------|
305
+ | Parlamint corpus | at, bg, cz, dk, ee, es, es-ga, fi, fr, gb, gr, hr, hu, it, lv, nl, no, pl, pt, rs, se, si | Erjavec et al., 2021 |
306
+ | Bulgarian National Corpus | bg | [Link](http://old.dcl.bas.bg/dataset/BulNC.7z) |
307
+ | Crawl of Bulgarian news websites | bg | [Link](http://old.dcl.bas.bg/dataset/Bulgarian_news.7z) |
308
+ | Colossal OSCAR 1.0 | bg, ca, cs, cy, da, de, el, en, es, et, eu, fi, fr, ga, gl, hr, hu, it, lt, lv, mt, nl, nn, no, oc, pl, pt, ro, ru, sh, sk, sl, sr, sv, uk | Brack et al., 2024 |
309
+ | Wikimedia dumps | bg, ca, cs, da, de, el, en, es, et, eu, fi, fr, ga, gl, hr, hu, it, lt, lv, mt, nl, nn, no, pl, pt, ro, sh, sk, sl, sr, uk | [Link](https://dumps.wikimedia.org/) |
310
+ | OpenSubtitlesv2016 | bg, ca, cs, da, de, el, en, es, et, eu, fi, fr, gl, hr, it, lt, lv, nl, no, pl, pt, ro, sk, sl, sr, sv, uk | Lison & Tiedemann, 2016 |
311
+ | MaCoCu web corpus | bg, ca, el, hr, mt, sl, sr, uk | Bañón et al., 2022 |
312
+ | EurLEX-Resources | bg, cs, da, de, el, en, es, et, fi, fr, ga, hr, hu, it, lt, lv, mt, nl, pl, pt, ro, sk, sl, sv | [Link](https://huggingface.co/datasets/joelniklaus/eurlex_resources) |
313
+ | MC4-Legal | bg, cs, da, de, el, en, es, et, fi, fr, ga, hu, it, lt, lv, mt, nl, pl, pt, ro, sk, sl, sv | [Link](https://huggingface.co/datasets/joelito/legal-mc4) |
314
+ | CURLICAT Corpus | bg, hr, hu, pl, ro, sk, sl | Váradi et al., 2022 |
315
+ | CATalog | ca | Palomar-Giner et al., 2024 |
316
+ | Spanish Crawling | ca, es, eu, gl | Relevant Spanish websites crawling |
317
+ | Starcoder | code | Li et al., 2023 |
318
+ | SYN v9: large corpus of written Czech | cs | Křen et al., 2021 |
319
+ | Welsh-GOV | cy | Crawling from [Link](https://www.llyw.cymru) |
320
+ | DaNewsroom | da | Varab & Schluter, 2020 |
321
+ | Danish GigaWord | da | Strømberg-Derczynski et al., 2021 |
322
+ | DK-CLARIN Reference Corpus of General Danish | da | [Link](https://korpus.dsl.dk/clarin/) |
323
+ | The Danish Parliament Corpus 2009 - 2017, v1 | da | Hansen, 2018 |
324
+ | DeWaC | de | [Link](https://docs.sslmit.unibo.it/doku.php?id=corpora:dewac) |
325
+ | Open Legal Data - German court decisions and laws | de | Ostendorff et al., 2020 |
326
+ | Greek Legal Code | el | Papaloukas et al., 2021 |
327
+ | Greek Web Corpus | el | Outsios et al., 2018 |
328
+ | Auxiliary Mathematics Problems and Solutions (AMPS) dataset | en | Hendrycks et al., 2021 |
329
+ | BIGPATENT | en | Sharma et al., 2019 |
330
+ | FineWeb-Edu (350BT subset) | en | Penedo et al., 2024 |
331
+ | peS2o | en | Soldaini & Lo, 2023 |
332
+ | PG-19 | en | Rae et al., 2019 |
333
+ | Pile of Law (selected subsets) | en | Henderson* et al., 2022 |
334
+ | proof-pile | en | [Link](https://huggingface.co/datasets/hoskinson-center/proof-pile) |
335
+ | RedPajama-Data T1 (StackExchange subset) | en | Computer, 2023 |
336
+ | The Pile (PhilPapers subset) | en | Gao et al., 2021 |
337
+ | Biomedical | es | Internally generated scientific dataset: Dialnet, Scielo, CSIC, TDX, BSC, UCM |
338
+ | HPLTDatasets v1 - Spanish | es | de Gibert et al., 2024 |
339
+ | Legal | es | Internally generated legal dataset: BOE, BORME, Senado, Congreso, Spanish court orders, DOGC |
340
+ | Scientific | es | Internally generated scientific dataset: Wikipedia LS, Pubmed, MeSpEn, patents, clinical cases, medical crawler |
341
+ | Spanish Legal Domain Corpora | es | Gutiérrez-Fandiño et al., 2021 |
342
+ | Estonian National Corpus 2021 | et | Koppel & Kallas, 2022 |
343
+ | Estonian Reference Corpus | et | [Link](https://www.cl.ut.ee/korpused/segakorpus/) |
344
+ | EusCrawl (w/o Wikipedia or NC-licenses) | eu | Artetxe et al., 2022 |
345
+ | Latxa Corpus v1.1 | eu | Etxaniz et al., 2024 [Link](https://huggingface.co/datasets/HiTZ/latxa-corpus-v1.1) |
346
+ | Aya Dataset (w/o Evaluation Suite) | eu, hr, nl, fi, ka, hu, lt, nn, ro, sk, lv, cy, bg, cs, en, fr, de, ga, mt, pl, ru, sl, sv, ca, da, et, gl, el, it, no, pt, sr, es, uk | Singh et al., 2024 |
347
+ | Yle Finnish News Archive | fi | [Link](http://urn.fi/urn:nbn:fi:lb-2021050401) |
348
+ | CaBeRnet: a New French Balanced Reference Corpus | fr | Popa-Fabre et al., 2020 |
349
+ | French Public Domain Books | fr | [Link](https://huggingface.co/datasets/PleIAs/French-PD-Books) |
350
+ | French Public Domain Newspapers | fr | [Link](https://huggingface.co/datasets/PleIAs/French-PD-Newspapers) |
351
+ | Irish Universal Dependencies | ga | [Link](https://universaldependencies.org/ga/index.html) |
352
+ | The Gaois bilingual corpus of English-Irish legislation (Irish legislation) | ga | [Link](https://portulanclarin.net/repository/browse/the-gaois-bilingual-corpus-of-english-irish-legislation-processed/daeac17c9e3511ea9b7f02420a000407b83de243dc0b469aab41084386c5b80f/) |
353
+ | CorpusNÓS | gl | de-Dios-Flores et al., 2024 |
354
+ | Croatian web corpus hrWaC 2.1 | hr | Ljubešić & Klubička, 2014 |
355
+ | ITWaC | it | [Link](https://docs.sslmit.unibo.it/doku.php?id=corpora:itwac) |
356
+ | Corpus of State-related content from the Latvian Web (Processed) | lv | [Link](https://catalog.elra.info/en-us/repository/browse/ELRA-W0169/) |
357
+ | Korpus Malti | mt | Micallef et al., 2022 |
358
+ | SoNaR Corpus NC 1.2 | nl | [Link](https://taalmaterialen.ivdnt.org/download/tstc-sonar-corpus/) |
359
+ | Norwegian Colossal Corpus | nn, no | Kummervold et al., 2021 |
360
+ | Occitan Corpus | oc | Provided by [IEA](https://www.institutestudisaranesi.cat/) |
361
+ | NKJP-PodkorpusMilionowy-1.2 (National Corpus of Polish) | pl | Lewandowska-Tomaszczyk et al., 2013 |
362
+ | Polish Parliamentary Corpus / Korpus Dyskursu Parlamentarnego | pl | Ogrodniczuk, 2018 |
363
+ | Brazilian Portuguese Web as Corpus | pt | Wagner Filho et al., 2018 |
364
+ | ParlamentoPT | pt | Rodrigues et al., 2023 |
365
+ | MARCELL Romanian legislative subcorpus v2 | ro | [Link](https://elrc-share.eu/reposMARCELL%20Romanian%20legislative%20subcorpus%20v2itory/browse/marcell-romanian-legislative-subcorpus-v2/2da548428b9d11eb9c1a00155d026706ce94a6b59ffc4b0e9fb5cd9cebe6889e/) |
366
+ | Korpus slovenských právnych predpisov v1.9 | sk | [Link](https://www.juls.savba.sk/data/marcell/legal-sk-20220322-1.9.ver.xz) |
367
+ | od-justice 2.0 | sk | [Link](https://www.juls.savba.sk/data/od-justice/od-justice-2.0.ver.xz) |
368
+ | Corpus of academic Slovene KAS 2.0 | sl | Žagar et al., 2022 |
369
+ | slWaC web corpus | sl | Erjavec et al., 2015 |
370
+ | SrpKorSubset (news, legal, academic, conversation, literary) | sr | [Link](http://www.korpus.matf.bg.ac.rs/) |
371
+ | The Swedish Culturomics Gigaword Corpus | sv | Rødven-Eide, 2016 |
372
+ | Corpus of laws and legal acts of Ukraine | uk | [Link](https://lang.org.ua/en/corpora/#anchor7) |
373
 
374
  <details>
375
  <summary>References</summary>