<jupyter_start><jupyter_text>Integration test on Wikitext-2> Training a Language Model on WT2<jupyter_code>path = untar_data(URLs.WIKITEXT_TINY)<jupyter_output><empty_output><jupyter_text>The dataset comes with all the wrticles concatenated. We split them to be able to shuffle at the beginning of each epoch.<jupyter_code>def istitle(line):
    return len(re.findall(r'^ = [^=]* = $', line)) != 0

def read_file(filename):
    articles = L()
    with open(filename, encoding='utf8') as f:
        lines = f.readlines()
    current_article = ''
    for i,line in enumerate(lines):
        current_article += line
        if i < len(lines)-2 and lines[i+1] == ' \n' and istitle(lines[i+2]):
            articles.append(current_article.split(' '))
            current_article = ''
    articles.append(current_article.split(' '))
    return articles<jupyter_output><empty_output><jupyter_text>Then we put our list of tokenized texts together in an `LM_Dataset`. It will return tuples of sequences of `seq_len`, with the second sequence between the first one shifted by one on the right.<jupyter_code>bs,sl = 104,72
train = LM_Dataset(read_file(path/'train.txt'), bs=bs, seq_len=sl, shuffle=True)
valid = LM_Dataset(read_file(path/'valid.txt'), bs=bs, seq_len=sl)
print(train[0])<jupyter_output>(['', '\n', '=', 'Geopyxis', 'carbonaria', '=', '\n', '\n', 'Geopyxis', 'carbonaria', 'is', 'a', 'species', 'of', 'fungus', 'in', 'the', 'genus', 'Geopyxis', ',', 'family', '<unk>', '.', 'First', 'described', 'to', 'science', 'in', '1805', ',', 'and', 'given', 'its', 'current', 'name', 'in', '1889', ',', 'the', 'species', 'is', 'commonly', 'known', 'as', 'the', 'charcoal', 'loving', 'elf', '@-@', 'cup', ',', 'dwarf', '<unk>', 'cup', ',', '<unk>', '<unk>', 'cup', ',', 'or', 'pixie', 'cup', '.', 'The', 'small', ',', '<unk>', '@-@', 'shaped', 'fruitbodies', 'of', 'the'], ['\n', '=', 'Geopyxis', 'carbonaria', '=', '\n', '\n', 'Geopyxis', 'carbonaria', 'is', 'a', 'species', 'of', 'fungus', 'in', 'the', 'genus', 'Geopyxis', ',', 'family', '<unk>', '.', 'First', 'described', 'to', 'science', 'in', '1805', ',', 'and', 'given', 'its', 'current', 'name', 'in', '1889', ',', 'the', 'species', 'is', 'commonly', 'known', 'as', 'the', 'charcoal', 'loving', 'elf', '@-@', 'cup', ',', 'dwarf', '<unk>', [...]<jupyter_text>We can then wrap our `LM_Dataset`s in a `TfmdList` to apply the `Numericalize` transform. We can't use a `TfmdDS` because our elements are already tuples and `TfmdDS` is there to create such tuples from individual items. Since we already have tuples, we specify `as_item=False`.<jupyter_code>count = Counter([p for t in train.ds for p in t])
vocab = make_vocab(count)
train_ds = TfmdList(train, tfms=Numericalize(vocab), as_item=False, wrap_l=False)
valid_ds = TfmdList(valid, tfms=Numericalize(vocab), as_item=False, wrap_l=False)<jupyter_output><empty_output><jupyter_text>Last but not least, we need to use a special sampler that will make sure we ask for the correct sequences to form a batch: in the first batch we don't want the sequences 0,1,2,3... (they are contiguous in the source obtained by concatenating all texts) but the sequences 0,`num_batches`,`2*num_batches`,...<jupyter_code>train_dl = TfmdDL(train_ds, bs=bs, sampler=LM_Sampler(train), tfms=Cuda(), num_workers=0)
valid_dl = TfmdDL(valid_ds, bs=bs, sampler=LM_Sampler(valid), tfms=Cuda(), num_workers=0)
dbch = DataBunch(train_dl, valid_dl)
dbch.show_batch()
config = awd_lstm_lm_config.copy()
config.update({'input_p': 0.6, 'output_p': 0.4, 'weight_p': 0.5, 'embed_p': 0.1, 'hidden_p': 0.2})
model = get_language_model(AWD_LSTM, len(vocab), config=config)
opt_func = partial(Adam, wd=0.1, eps=1e-7)
cb_funcs = [partial(MixedPrecision, clip=0.1), partial(RNNTrainer, alpha=3, beta=2)]
learn = Learner(model, dbch, loss_func=CrossEntropyLossFlat(), opt_func=opt_func, cb_funcs=cb_funcs, metrics=[accuracy, Perplexity()])
%prun learn.fit_one_cycle(1, 5e-3, moms=(0.8,0.7,0.8), div=10)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.2033558727773604, 'fraction_numerical': 0.01652892561983471, 'mean_word_length': 5.9825174825174825, 'pattern_counts': {'":': 0, '<': 28, '<?xml version=': 0, '>': 28, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '350473', 'license': 'Apache-2.0', 'n_tokens_mistral': 1546, 'n_tokens_neox': 1493, 'n_words': 382, 'path': 'dev/35_tutorial_wikitext.ipynb', 'repo_name': 'nareshr8/fastai_dev', 'size': '3993'}