language:ca license:cc-by-sa-4.0 region:us
---
language: ca
license: cc-by-sa-4.0
extra_gated_prompt: 'By accessing this dataset, you agree to abide by the BigScience
  Ethical Charter. The charter can be found at:

  https://hf.co/spaces/bigscience/ethical-charter'
extra_gated_fields:
  I have read and agree to abide by the BigScience Ethical Charter: checkbox
---
ROOTS Subset: roots_ca_catalan_textual_corpus

# Catalan Textual Corpus

- Dataset uid: `catalan_textual_corpus`

### Description

The Catalan Textual Corpus is a 1760-million-token web corpus of Catalan built from several sources: existing corpus such as DOGC, CaWac (non-dedup version), Oscar (unshuffled version), Open Subtitles, Catalan Wikipedia; and three brand new crawlings: the Catalan General Crawling, obtained by crawling the 500 most popular .cat and .ad domains; the Catalan Government Crawling, obtained by crawling the .gencat domain and subdomains, belonging to the Catalan Government; and the ACN corpus with 220k news items from March 2015 until October 2020, crawled from the Catalan News Agency.

It consists of 1.758.388.896 tokens, 73.172.152 sentences and 12.556.365 documents. Documents are separated by single new lines. These boundaries have been preserved as long as the license allowed it.

### Homepage

https://zenodo.org/record/4519349

### Licensing

- open license
- cc-by-4.0: Creative Commons Attribution 4.0 International


### Speaker Locations

- Southern Europe
- Spain


### Sizes

- 0.7691 % of total
- 64.8510 % of ca

### BigScience processing steps

#### Filters applied to: ca

- dedup_document
- dedup_template_soft
- filter_remove_empty_docs
- filter_small_docs_bytes_1024

