<jupyter_start><jupyter_text>Preparing a poetry corpus for trainingBy [<NAME>](http://www.decontextualize.com/)I wanted to train a VAE with [BPEmb](https://nlp.h-its.org/bpemb/)'s pretrained sub-word embeddings. This notebook helps create the dataset. First, install `bpemb`.<jupyter_code>!pip install bpemb<jupyter_output><empty_output><jupyter_text>Load `BPEmb` with the desired vocabulary size. The `BPEmb` object downloads the models and embeddings on demand, so this might take a while the first time you run it.<jupyter_code>import bpemb
import json, gzip, random
bp = bpemb.BPEmb(lang='en', dim=200, vs=25000)<jupyter_output><empty_output><jupyter_text>Download the [Project Gutenberg Poetry Corpus](https://github.com/aparrish/gutenberg-poetry-corpus) and change the path below to its location on your drive. The following loads in all ~3M lines of poetry:<jupyter_code>lines = []
for line in gzip.open("/Users/allison/projects/gutenberg-dammit-archive/gutenberg-poetry-v001.ndjson.gz"):
    data = json.loads(line.strip())
    lines.append(data['s'])
len(lines)<jupyter_output><empty_output><jupyter_text>Shuffle:<jupyter_code>random.shuffle(lines)<jupyter_output><empty_output><jupyter_text>Set the name of the dataset and the size of the training, validation and test sets:<jupyter_code>dataset_name = 'poetry_10k_sample'
train_size = 10000
valid_size = 1000
test_size = 1000<jupyter_output><empty_output><jupyter_text>Then write out the files, using `bpemb` to encode to the fixed vocabulary size.<jupyter_code>!mkdir datasets/{dataset_name}_data
with gzip.open("datasets/%s_data/%s.train.txt.gz" % (dataset_name, dataset_name), "wt") as fh:
    for line in lines[:train_size]:
        fh.write(' '.join(bp.encode(line)) + "\n")
with gzip.open("datasets/%s_data/%s.valid.txt.gz" % (dataset_name, dataset_name), "wt") as fh:
    for line in lines[train_size:train_size+valid_size]:
        fh.write(' '.join(bp.encode(line)) + "\n")
with gzip.open("datasets/%s_data/%s.test.txt.gz" % (dataset_name, dataset_name), "wt") as fh:
    for line in lines[train_size+valid_size:train_size+valid_size+test_size]:
        fh.write(' '.join(bp.encode(line)) + "\n")
with open("datasets/%s_data/vocab.txt" % dataset_name, "w") as fh:
    for item in bp.words:
        fh.write(item + "\n")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1366968520914187, 'fraction_numerical': 0.01164294954721863, 'mean_word_length': 5.803519061583578, 'pattern_counts': {'":': 0, '<': 26, '<?xml version=': 0, '>': 26, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '401538', 'license': 'MIT', 'n_tokens_mistral': 853, 'n_tokens_neox': 811, 'n_words': 188, 'path': 'prep-poetry-sample-data.ipynb', 'repo_name': 'chaosmosister/vae-lagging-encoder', 'size': '2328'}