<commit_before># TED 2020 - Parallel Sentences Corpus

This dataset contains a crawl of nearly 4000 TED (https://www.ted.com/) and TED-X transcripts from July 2020. The transcripts have been translated by a global community of volunteers (https://www.ted.com/participate/translate) to more than 100 languages.

The transcripts for the different languages have been sentence aligned to generate a parallel corpus that can be used to train machine translation systems.


## Download
Tab-Seperated file containing all aligned sentences:
[https://public.ukp.informatik.tu-darmstadt.de/reimers/sentence-transformers/datasets/ted2020.tsv.gz](https://public.ukp.informatik.tu-darmstadt.de/reimers/sentence-transformers/datasets/ted2020.tsv.gz)

The ted2020.tsv.gz is a tab-seperated file containing all aligned sentences. The first line specifies the language code. Additionally, the first column is 'talkid' with the information from which talk the sentence stems. 



Source-code & raw JSON data files (not sentence aligned):
[https://public.ukp.informatik.tu-darmstadt.de/reimers/sentence-transformers/datasets/ted2020-source-files.zip](https://public.ukp.informatik.tu-darmstadt.de/reimers/sentence-transformers/datasets/ted2020-source-files.zip)





## Usage

You can easily load the dataset using Pandas:
```python
import pandas
import csv

df = pandas.read_csv('ted2020.tsv.gz', sep='\t', keep_default_na=False, encoding='utf8', quoting=csv.QUOTE_NONE)
```



Or with Python CSV reader:
```python
import csv
import gzip

with gzip.open('ted2020.tsv.gz', 'rt', encoding='utf8') as fIn:
   reader = csv.DictReader(fIn, delimiter='\t', quoting=csv.QUOTE_NONE )
   for line in reader:
      print(line)
```
<commit_msg>Update TED2020.md<commit_after># TED 2020 - Parallel Sentences Corpus

This dataset contains a crawl of nearly 4000 TED (https://www.ted.com/) and TED-X transcripts from July 2020. The transcripts have been translated by a global community of volunteers (https://www.ted.com/participate/translate) to more than 100 languages.

The transcripts for the different languages have been sentence aligned to generate a parallel corpus that can be used to train machine translation systems.


## Download
The TED 2020 dataset is not available for download anymore.



## Usage

You can easily load the dataset using Pandas:
```python
import pandas
import csv

df = pandas.read_csv('ted2020.tsv.gz', sep='\t', keep_default_na=False, encoding='utf8', quoting=csv.QUOTE_NONE)
```



Or with Python CSV reader:
```python
import csv
import gzip

with gzip.open('ted2020.tsv.gz', 'rt', encoding='utf8') as fIn:
   reader = csv.DictReader(fIn, delimiter='\t', quoting=csv.QUOTE_NONE )
   for line in reader:
      print(line)
```

{'fraction_non_alphanumeric': 0.10252654705236178, 'fraction_numerical': 0.02856096667887221, 'mean_word_length': 4.763713080168777, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 8, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4502144', 'n_tokens_mistral': 941, 'n_tokens_neox': 805, 'n_words': 277}