<jupyter_start><jupyter_text>Create a corpus and clean text<jupyter_code>import os
from datasets import Dataset, load_dataset<jupyter_output><empty_output><jupyter_text>Settings  DatasetsThe datasets should only have one column named `"text"` as this column will be used to extract attributes.<jupyter_code>dataset_constructors = []<jupyter_output><empty_output><jupyter_text>Helper function as dataset has only the remove_column function<jupyter_code>def select_columns(dataset, columns):
    return dataset.remove_columns([c for c in dataset.features.keys() if c not in columns])<jupyter_output><empty_output><jupyter_text>Racehttps://huggingface.co/datasets/race<jupyter_code>def race():
    dataset = load_dataset('race', 'all', split='train+validation+test')
    # rename the column 'article' to 'text'
    dataset = dataset.rename_column('article', 'text')
    # remove all other columns as we do not need it
    dataset = select_columns(dataset, ['text'])
    return dataset

dataset_constructors.append(race)<jupyter_output><empty_output><jupyter_text>Wikitexthttps://huggingface.co/datasets/wikitext<jupyter_code>def wikitext():
    dataset = load_dataset('wikitext', 'wikitext-103-raw-v1', split='validation+test+train')
    # filter empty text / titles / headers
    dataset = dataset.filter(lambda x: len(x['text']) > 64 and not x['text'].startswith(' ='))
    return dataset

dataset_constructors.append(wikitext)<jupyter_output><empty_output><jupyter_text>CC_Newshttps://huggingface.co/datasets/cc_news<jupyter_code>def cc_news():
    dataset = load_dataset('cc_news', split='train[:40%]')
    # remove all other columns as we do not need it
    dataset = select_columns(dataset, ['text'])
    return dataset

dataset_constructors.append(cc_news)<jupyter_output><empty_output><jupyter_text>DuoRChttps://huggingface.co/datasets/duorc<jupyter_code>def duorc():
    dataset = load_dataset('duorc', 'SelfRC', split='train+validation+test')
    # rename the column 'article' to 'text'
    dataset = dataset.rename_column('plot', 'text')
    # remove unnecessary columns
    dataset = select_columns(dataset, ['text', 'plot_id'])
    # remove duplicate texts
    dataset = Dataset.from_pandas(dataset.to_pandas().drop_duplicates(subset=['plot_id'])[['text']])
    # remove all other columns as we do not need it
    dataset = select_columns(dataset, ['text'])
    return dataset

dataset_constructors.append(duorc)<jupyter_output><empty_output><jupyter_text>IMDBhttps://huggingface.co/datasets/imdb<jupyter_code>def imdb():
    dataset = load_dataset('imdb', split='train+test+unsupervised')
    # remove all other columns as we do not need it
    dataset = select_columns(dataset, ['text'])
    return dataset

dataset_constructors.append(imdb)<jupyter_output><empty_output><jupyter_text>Reddit TIFUhttps://huggingface.co/datasets/reddit_tifu<jupyter_code>def reddit_tifu():
    dataset = load_dataset('reddit_tifu', 'short', split='train')
    # rename the column 'article' to 'text'
    dataset = dataset.rename_column('documents', 'text')
    # remove all other columns as we do not need it
    dataset = select_columns(dataset, ['text'])
    return dataset

dataset_constructors.append(reddit_tifu)<jupyter_output><empty_output><jupyter_text>CNN DailyMailhttps://huggingface.co/datasets/cnn_dailymail<jupyter_code>def cnn_dailymail():
    dataset = load_dataset('cnn_dailymail', '3.0.0', split='train[:50%]+validation[:50%]+test[:50%]')
    # rename the column 'article' to 'text'
    dataset = dataset.rename_column('article', 'text')
    # remove all other columns as we do not need it
    dataset = select_columns(dataset, ['text'])
    return dataset

dataset_constructors.append(cnn_dailymail)<jupyter_output><empty_output><jupyter_text>Reddithttps://huggingface.co/datasets/reddit<jupyter_code>def reddit():
    dataset = load_dataset('reddit', 'default', split='train[:20%]')
    # rename the column 'content' to 'text'
    dataset = dataset.rename_column('content', 'text')
    # remove all other columns as we do not need it
    dataset = select_columns(dataset, ['text'])
    return dataset

dataset_constructors.append(reddit)<jupyter_output><empty_output><jupyter_text>Load data and mergeCreate temporary `.parquet` files that only contain articles in a column `"text"`.<jupyter_code>print('Combine datasets:', [f.__name__ for f in dataset_constructors])
# make tmp/ folder
if not os.path.exists('tmp/'):
    os.makedirs('tmp/')
# generate datasets and write to file
for f in dataset_constructors:
    if not os.path.exists(f'tmp/{f.__name__}.parquet'):
        f().to_parquet(f'tmp/{f.__name__}.parquet')<jupyter_output>Reusing dataset cc_news (C:\Users\mauri\.cache\huggingface\datasets\cc_news\plain_text\1.0.0\ae469e556251e6e7e20a789f93803c7de19d0c4311b6854ab072fecb4e401bd6)
Reusing dataset cnn_dailymail (C:\Users\mauri\.cache\huggingface\datasets\cnn_dailymail\3.0.0\3.0.0\3cb851bf7cf5826e45d49db2863f627cba583cbc32342df7349dfe6c38060234)
Using custom data configuration default
Reusing dataset reddit (C:\Users\mauri\.cache\huggingface\datasets\reddit\default\1.0.0\98ba5abea674d3178f7588aa6518a5510dc0c6fa8176d9653a3546d5afcb3969)<jupyter_text>Construct one corpus based on all `.parquet` files<jupyter_code>corpus = Dataset.from_parquet([f'tmp/{f.__name__}.parquet' for f in dataset_constructors]).shuffle()
corpus<jupyter_output>Using custom data configuration default-4a5372619005ca18<jupyter_text>Save data<jupyter_code>corpus.to_parquet('data/corpus.parquet')<jupyter_output><empty_output><jupyter_text>If the corpus is already present in `data/`, start the notebook from here to create different subsets<jupyter_code>corpus = Dataset.from_parquet('data/corpus.parquet')<jupyter_output>Using custom data configuration default-63640bc2f75a5f0c<jupyter_text>Remove temporary files<jupyter_code>#import shutil
#if os.path.exists('tmp/'):
#    shutil.rmtree('tmp/')<jupyter_output><empty_output><jupyter_text>Example<jupyter_code>for i in range(3):
    print(corpus[i], '\n')<jupyter_output>{'text': "Thank you for acknowledging the fact that my school sucks!! If it means anything, I live in a fairly conservative area. The worst thing about all of this though was that when I first told my teacher that I really trusted, she told the guidance office (as I expected/wanted), and it took them over a month to finally follow up and talk to me for the first time. I was really depressed when I first told my original teacher and I attribute my lower grades my first quarter to that. Sigh. \n Anyways! I know that age isn't really a factor, but I also know there won't be much done other than filling a police report at this point, which is fine with me. I just want him to know that he doesn't have control over me (and even though I haven't had direct contact with him for nearly a year, he and his actions still control so much of my life...), and that I can do whatever the hell I want :) \n I've considered calling the hotline, but my shit counselor told me that they would tell me the exa[...]
{'fraction_non_alphanumeric': 0.11122173232826056, 'fraction_numerical': 0.02531645569620253, 'mean_word_length': 6.00398406374502, 'pattern_counts': {'":': 0, '<': 65, '<?xml version=': 0, '>': 66, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 1, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '439698', 'license': 'Apache-2.0', 'n_tokens_mistral': 2405, 'n_tokens_neox': 2294, 'n_words': 575, 'path': '0_make_corpus.ipynb', 'repo_name': 'm4urin/BERT_bias', 'size': '7031'}