task_categories:text-generation size_categories:10M<n<100M language:de arxiv:2212.10440 region:us
---
task_categories:
- text-generation
language:
- de
size_categories:
- 10M<n<100M
dataset_info:
  features:
  - name: id
    dtype: int64
  - name: text
    dtype: string
  - name: meta
    struct:
    - name: warc_headers
      struct:
      - name: warc-record-id
        dtype: string
      - name: warc-date
        dtype: string
      - name: content-type
        dtype: string
      - name: content-length
        dtype: int32
      - name: warc-type
        dtype: string
      - name: warc-identified-content-language
        dtype: string
      - name: warc-refers-to
        dtype: string
      - name: warc-target-uri
        dtype: string
      - name: warc-block-digest
        dtype: string
    - name: identification
      struct:
      - name: label
        dtype: string
      - name: prob
        dtype: float32
    - name: harmful_pp
      dtype: float32
    - name: tlsh
      dtype: string
    - name: quality_warnings
      sequence: string
    - name: categories
      sequence: string
    - name: sentence_identifications
      list:
      - name: label
        dtype: string
      - name: prob
        dtype: float32
  splits:
  - name: train
    num_bytes: 382684030510
    num_examples: 53172498
  download_size: 80368267320
  dataset_size: 382684030510
---
# Oscar 2023_01 DE Deduplicated
This is a deduplicated version of the german subset of the [23.01 OSCAR Corpus](https://github.com/ChenghaoMou/text-dedup), a large, crawled, and processed text dataset
curated by the OSCAR project (Open Super-large Crawled Aggregated coRpus).
OSCAR 23.01 is the January 2023 version of the OSCAR Corpus based on the November/December 2022 dump of Common Crawl. 
While being quite similar to OSCAR 22.01, it contains several new features, including KenLM-based adult content detection, [...].

It was deduplicated using a MinHash implementation from the `text-dedup` library by `ChenghaoMou` available on [GitHub](https://github.com/ChenghaoMou/text-dedup). with the following command:

```bash
python -m text_dedup.minhash --path oscar-corpus/OSCAR-2301 --name "de" --cache_dir "../cache" --split "train" --column "text" --batch_size 10000 --output output/minhash_oscar_de_dedup
```
Find a filtered version of this dataset at [bjoernp/oscar2301_de_deduped_filtered](https://huggingface.co/datasets/bjoernp/oscar2301_de_deduped_filtered).

## Deduplication statistics
|  Step |  Runtime |
|---|---|
| Loading  | 10.64s |
| MinHashing  | 10574.02s |
| Clustering  | 12187.65s |
| Filtering  | 4198.70s |
| Saving  | 3560.06s |
| Total  | 30531.07s | 

|  Dataset | Number of documents  |
|---|---|
| Before                          | 103299215 |
| After                           | 53172498  |

## Dataset scheme:
```json
{
   "text":"English sentence\nphrase en français\n????????????", // (1)
   "meta":{
       "warc_headers":{ // (2)
          "warc-identified-content-language":"fra,eng",
          "warc-target-uri":"https://fr.wikipedia.org/wiki/...",
          "warc-record-id":"<urn:uuid:29eaa920-d299-4b1d-b687-c72bd8d68116>",
          "warc-type":"conversion",
          "content-length":"35298", // (3)
          "warc-refers-to":"<urn:uuid:39e42055-0d94-4e45-9c6c-9e7056635d64>",
          "warc-block-digest":"sha1:WFH2A5WHCS2H365GIAFYQPI7UOAMFGHB", // (3)
          "warc-date":"2022-11-26T09:45:47Z",
          "content-type":"text/plain"
       },
      "identification":{ // (4)
         "label":"fr",
         "prob":0.8938327
      },
      "harmful_pp":4063.1814, // (5)
      "tlsh":"tlsh:T125315FF2B6088901EEA097015DB39B4600B...", // (6)
      "quality_warnings":[ // (7)
         "short_sentences",
         "header",
         "footer"
      ],
      "categories":[ // (8)
         "examen_pix",
         "liste_bu"
      ],
      "sentence_identifications":[ // (9)
         {
            "label":"fr",
            "prob":0.99837273
         },
         {
            "label":"en",
            "prob":0.9992377
         },
         null
      ]
   }
}

```

## Licensing
(from the original OSCAR Corpus. We cannot reasonably comply with takedown requests.)
```
These data are released under this licensing scheme
We do not own any of the text from which these data has been extracted.
We license the actual packaging, the metadata and the annotations of these data under the Creative Commons CC0 license ("no rights reserved") http://creativecommons.org/publicdomain/zero/1.0/
To the extent possible under law, the OSCAR project, Inria, the Univertity of Mannheim and DFKI GmbH have waived all copyright and related or neighboring rights to OSCAR
This work is published from: France and Germany.

[[[
Should you consider that our data contains material that is owned by you and should therefore not be reproduced here, please:
* Clearly identify yourself, with detailed contact data such as an address, telephone number or email address at which you can be contacted.
* Clearly identify the copyrighted work claimed to be infringed.
* Clearly identify the material that is claimed to be infringing and information reasonably sufficient to allow us to locate the material.

We will comply to legitimate requests by removing the affected sources from the next release of the corpus.
]]]
```

## Citation
```
@ARTICLE{2022arXiv221210440J,
       author = {{Jansen}, Tim and {Tong}, Yangling and {Zevallos}, Victoria and {Ortiz Suarez}, Pedro},
        title = "{Perplexed by Quality: A Perplexity-based Method for Adult and Harmful Content Detection in Multilingual Heterogeneous Web Data}",
      journal = {arXiv e-prints},
     keywords = {Computer Science - Computation and Language},
         year = 2022,
        month = dec,
          eid = {arXiv:2212.10440},
        pages = {arXiv:2212.10440},
          doi = {10.48550/arXiv.2212.10440},
archivePrefix = {arXiv},
       eprint = {2212.10440},
 primaryClass = {cs.CL},
       adsurl = {https://ui.adsabs.harvard.edu/abs/2022arXiv221210440J},
      adsnote = {Provided by the SAO/NASA Astrophysics Data System}
}

@inproceedings{abadji-etal-2022-towards,
    title = "Towards a Cleaner Document-Oriented Multilingual Crawled Corpus",
    author = "Abadji, Julien  and
      Ortiz Suarez, Pedro  and
      Romary, Laurent  and
      Sagot, Beno{\^\i}t",
    booktitle = "Proceedings of the Thirteenth Language Resources and Evaluation Conference",
    month = jun,
    year = "2022",
    address = "Marseille, France",
    publisher = "European Language Resources Association",
    url = "https://aclanthology.org/2022.lrec-1.463",
    pages = "4344--4355",
    abstract = "The need for large corpora raw corpora has dramatically increased in recent years with the introduction of transfer learning and semi-supervised learning methods to Natural Language Processing. And while there have been some recent attempts to manually curate the amount of data necessary to train large language models, the main way to obtain this data is still through automatic web crawling. In this paper we take the existing multilingual web corpus OSCAR and its pipeline Ungoliant that extracts and classifies data from Common Crawl at the line level, and propose a set of improvements and automatic annotations in order to produce a new document-oriented version of OSCAR that could prove more suitable to pre-train large generative language models as well as hopefully other applications in Natural Language Processing and Digital Humanities.",
}

      
@inproceedings{AbadjiOrtizSuarezRomaryetal.2021,
  author    = {Julien Abadji and Pedro Javier Ortiz Su{\'a}rez and Laurent Romary and Beno{\^i}t Sagot},
  title     = {Ungoliant: An optimized pipeline for the generation of a very large-scale multilingual web corpus},
  series = {Proceedings of the Workshop on Challenges in the Management of Large Corpora (CMLC-9) 2021. Limerick, 12 July 2021 (Online-Event)},
  editor    = {Harald L{\"u}ngen and Marc Kupietz and Piotr Bański and Adrien Barbaresi and Simon Clematide and Ines Pisetta},
  publisher = {Leibniz-Institut f{\"u}r Deutsche Sprache},
  address   = {Mannheim},
  doi       = {10.14618/ids-pub-10468},
  url       = {https://nbn-resolving.org/urn:nbn:de:bsz:mh39-104688},
  pages     = {1 -- 9},
  year      = {2021},
  abstract  = {Since the introduction of large language models in Natural Language Processing, large raw corpora have played a crucial role in Computational Linguistics. However, most of these large raw corpora are either available only for English or not available to the general public due to copyright issues. Nevertheless, there are some examples of freely available multilingual corpora for training Deep Learning NLP models, such as the OSCAR and Paracrawl corpora. However, they have quality issues, especially for low-resource languages. Moreover, recreating or updating these corpora is very complex. In this work, we try to reproduce and improve the goclassy pipeline used to create the OSCAR corpus. We propose a new pipeline that is faster, modular, parameterizable, and well documented. We use it to create a corpus similar to OSCAR but larger and based on recent data. Also, unlike OSCAR, the metadata information is at the document level. We release our pipeline under an open source license and publish the corpus under a research-only license.},
  language  = {en}
}

@article{kreutzer-etal-2022-quality,
    title = "Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets",
    author = {Kreutzer, Julia  and
      Caswell, Isaac  and
      Wang, Lisa  and
      Wahab, Ahsan  and
      van Esch, Daan  and
      Ulzii-Orshikh, Nasanbayar  and
      Tapo, Allahsera  and
      Subramani, Nishant  and
      Sokolov, Artem  and
      Sikasote, Claytone  and
      Setyawan, Monang  and
      Sarin, Supheakmungkol  and
      Samb, Sokhar  and
      Sagot, Beno{\^\i}t  and
      Rivera, Clara  and
      Rios, Annette  and
      Papadimitriou, Isabel  and
      Osei, Salomey  and
      Suarez, Pedro Ortiz  and
      Orife, Iroro  and
      Ogueji, Kelechi  and
      Rubungo, Andre Niyongabo  and
      Nguyen, Toan Q.  and
      M{\"u}ller, Mathias  and
      M{\"u}ller, Andr{\'e}  and
      Muhammad, Shamsuddeen Hassan  and
      Muhammad, Nanda  and
      Mnyakeni, Ayanda  and
      Mirzakhalov, Jamshidbek  and
      Matangira, Tapiwanashe  and
      Leong, Colin  and
      Lawson, Nze  and
      Kudugunta, Sneha  and
      Jernite, Yacine  and
      Jenny, Mathias  and
      Firat, Orhan  and
      Dossou, Bonaventure F. P.  and
      Dlamini, Sakhile  and
      de Silva, Nisansa  and
      {\c{C}}abuk Ball{\i}, Sakine  and
      Biderman, Stella  and
      Battisti, Alessia  and
      Baruwa, Ahmed  and
      Bapna, Ankur  and
      Baljekar, Pallavi  and
      Azime, Israel Abebe  and
      Awokoya, Ayodele  and
      Ataman, Duygu  and
      Ahia, Orevaoghene  and
      Ahia, Oghenefego  and
      Agrawal, Sweta  and
      Adeyemi, Mofetoluwa},
    journal = "Transactions of the Association for Computational Linguistics",
    volume = "10",
    year = "2022",
    address = "Cambridge, MA",
    publisher = "MIT Press",
    url = "https://aclanthology.org/2022.tacl-1.4",
    doi = "10.1162/tacl_a_00447",
    pages = "50--72",
    abstract = "With the success of large-scale pre-training and multilingual modeling in Natural Language Processing (NLP), recent years have seen a proliferation of large, Web-mined text datasets covering hundreds of languages. We manually audit the quality of 205 language-specific corpora released with five major public datasets (CCAligned, ParaCrawl, WikiMatrix, OSCAR, mC4). Lower-resource corpora have systematic issues: At least 15 corpora have no usable text, and a significant fraction contains less than 50{\%} sentences of acceptable quality. In addition, many are mislabeled or use nonstandard/ambiguous language codes. We demonstrate that these issues are easy to detect even for non-proficient speakers, and supplement the human audit with automatic analyses. Finally, we recommend techniques to evaluate and improve multilingual corpora and discuss potential risks that come with low-quality data releases.",
}

@inproceedings{ortiz-suarez-etal-2020-monolingual,
    title = "A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languages",
    author = "Ortiz Su{'a}rez, Pedro Javier  and
      Romary, Laurent  and
      Sagot, Benoit",
    booktitle = "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics",
    month = jul,
    year = "2020",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/2020.acl-main.156",
    pages = "1703--1714",
    abstract = "We use the multilingual OSCAR corpus, extracted from Common Crawl via language classification, filtering and cleaning, to train monolingual contextualized word embeddings (ELMo) for five mid-resource languages. We then compare the performance of OSCAR-based and Wikipedia-based ELMo embeddings for these languages on the part-of-speech tagging and parsing tasks. We show that, despite the noise in the Common-Crawl-based OSCAR data, embeddings trained on OSCAR perform much better than monolingual embeddings trained on Wikipedia. They actually equal or improve the current state of the art in tagging and parsing for all five languages. In particular, they also improve over multilingual Wikipedia-based contextual embeddings (multilingual BERT), which almost always constitutes the previous state of the art, thereby showing that the benefit of a larger, more diverse corpus surpasses the cross-lingual benefit of multilingual embedding architectures.",
}

@inproceedings{OrtizSuarezSagotRomary2019,
  author    = {Pedro Javier {Ortiz Su{'a}rez} and Benoit Sagot and Laurent Romary},
  title     = {Asynchronous pipelines for processing huge corpora on medium to low resource infrastructures},
  series = {Proceedings of the Workshop on Challenges in the Management of Large Corpora (CMLC-7) 2019. Cardiff, 22nd July 2019},
  editor    = {Piotr Bański and Adrien Barbaresi and Hanno Biber and Evelyn Breiteneder and Simon Clematide and Marc Kupietz and Harald L{"u}ngen and Caroline Iliadi},
  publisher = {Leibniz-Institut f{"u}r Deutsche Sprache},
  address   = {Mannheim},
  doi       = {10.14618/ids-pub-9021},
  url       = {http://nbn-resolving.de/urn:nbn:de:bsz:mh39-90215},
  pages     = {9 -- 16},
  year      = {2019},
  abstract  = {Common Crawl is a considerably large, heterogeneous multilingual corpus comprised of crawled documents from the internet, surpassing 20TB of data and distributed as a set of more than 50 thousand plain text files where each contains many documents written in a wide variety of languages. Even though each document has a metadata block associated to it, this data lacks any information about the language in which each document is written, making it extremely difficult to use Common Crawl for monolingual applications. We propose a general, highly parallel, multithreaded pipeline to clean and classify Common Crawl by language; we specifically design it so that it runs efficiently on medium to low resource infrastructures where I/O speeds are the main constraint. We develop the pipeline so that it can be easily reapplied to any kind of heterogeneous corpus and so that it can be parameterised to a wide range of infrastructures. We also distribute a 6.3TB version of Common Crawl, filtered, classified by language, shuffled at line level in order to avoid copyright issues, and ready to be used for NLP applications.},
  language  = {en}
}
```