<jupyter_start><jupyter_text>Homework> Training your own language model In the lesson we saw how to train a language model with the IMDB dataset and how to use it to create movie reviews. Here, we will take a different dataset and do the same process with a different dataset. We will consider the dataset of amazon reviews and compare its outputs with a model trained on the IMDB dataset. In order to speed things up, we will consider sub-samples of both datasets as they contain from $10^5$ to $10^6$ reviews. We will work with $\sim10^3$ samples on each dataset. This would be quite unreasonable to do if we were not using a pre-trained language model. Since we're fine-tunning the language model, it already knows a lot about English and the world, so a few samples will suffice for the training. If we had to build an entire language model from scratch, we would need quite a lot of text.   Movie reviews<jupyter_code>path = untar_data(URLs.IMDB_SAMPLE)
dls_lm = DataBlock(blocks=TextBlock.from_df('text', is_lm=True),
                   get_x=ColReader('text'), splitter=RandomSplitter(0.1)
                   ).dataloaders(pd.read_csv(path/'texts.csv'), bs=128)<jupyter_output><empty_output><jupyter_text>Always have a look at the data!<jupyter_code>dls_lm.show_batch(max_n=3)<jupyter_output><empty_output><jupyter_text>We're good to train our model. Let's fine-tune a language model from wikipedia.<jupyter_code>imdb_learn = language_model_learner(dls_lm, AWD_LSTM, drop_mult=0.3, metrics=[accuracy, Perplexity()]).to_fp16()
imdb_learn.fine_tune(4, base_lr=2e-2)<jupyter_output><empty_output><jupyter_text>Let's test the model :)<jupyter_code>#hide
n_words = 50
stem = "Such terrible acting"
pred = imdb_learn.predict(stem, n_words, temperature=0.8)
print(pred)<jupyter_output><empty_output><jupyter_text>We can see how this model is notably worse than the one we showed at class trained with the entire dataset, but we're just playing a bit with the concept. In fact, we can check what would happen if we trained a model without ULMFiT, starting from scratch.<jupyter_code>imdb_learn_0 = language_model_learner(dls_lm, AWD_LSTM, drop_mult=0.3, metrics=[accuracy, Perplexity()], 
                                      pretrained=False).to_fp16()
imdb_learn_0.fit_one_cycle(5, lr_max=5e-2)<jupyter_output><empty_output><jupyter_text>The model is significantly worse at predicting hte next word (8% accuracy vs 27%). Let's have a look at a prediction<jupyter_code>stem = "Such terrible acting"
pred = imdb_learn_0.predict(stem, n_words, temperature=0.8)
print(pred)<jupyter_output><empty_output><jupyter_text>Awful >.< These were way too little samples and training for such a model. However, with transfer learning, we have been able to train a somewhat decent model to make movie reviews. Of course, we saw that with more data and training the results are much better.   Amazon reviewsLet's train now a model to generate amazon reviews. This is a much more challenging task, provided that amazon reviews are, generally, awfully written and shorter. Sometimes, it seems that people completely forget that grammar exists!<jupyter_code>path = untar_data(URLs.AMAZON_REVIEWS)
samples = 5000
a_df = pd.read_csv(path/"train.csv", names=["label", "title", "text"]).sample(samples).reset_index()
dls_lm = DataBlock(blocks=TextBlock.from_df('text', is_lm=True),
                   get_x=ColReader('text'), splitter=RandomSplitter(0.1)
                   ).dataloaders(a_df, bs=128)<jupyter_output><empty_output><jupyter_text>Always have a look at the data! Let's see some reviews.<jupyter_code>dls_lm.show_batch(max_n=3)<jupyter_output><empty_output><jupyter_text>We can now train our model, just as in the IMDB dataset.<jupyter_code>amzn_learn = language_model_learner(dls_lm, AWD_LSTM, drop_mult=0.3, metrics=[accuracy, Perplexity()]).to_fp16()
amzn_learn.fine_tune(4, base_lr=2e-2)
stem = "Hair dryer"
pred = amzn_learn.predict(stem, n_words, temperature=0.8)
print(pred)<jupyter_output><empty_output><jupyter_text>The model was not sure about whether it was good or bad, but it does look like an amazon review about a hair dryer.  Model comparisonThis is where you get creative. Compare the predictions of both models given the same starting points. You can play around and try to find ambiguous starting points, things that work in both contexts (movie reviews and product reviews), completely start deep into one field and see what the opposite model manages to output, etc.Have a bit of fun!<jupyter_code>stem = "Harry Potter never "
pred_amzn = amzn_learn.predict(stem, n_words, temperature=0.8)
pred_imdb = imdb_learn.predict(stem, n_words, temperature=0.8)
print(pred_amzn+"\n")
print(pred_imdb)<jupyter_output><empty_output><jupyter_text>Well, these reviews are not the best ever, but we can see how both know that Harry Potter are books so... I guess I'll take that. Copy/paste this bit of code and experiment a bit. Giving a longer starting point will make the prediction a little bit better. You can test and see what happens when we give it a long input or an empty string.<jupyter_code>stem = " "
pred_amzn = amzn_learn.predict(stem, n_words, temperature=0.8)
pred_imdb = imdb_learn.predict(stem, n_words, temperature=0.8)
print(pred_amzn+"\n")
print(pred_imdb)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08537504711647192, 'fraction_numerical': 0.012815680361854504, 'mean_word_length': 5.448359659781288, 'pattern_counts': {'":': 0, '<': 46, '<?xml version=': 0, '>': 47, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '283422', 'license': 'Apache-2.0', 'n_tokens_mistral': 1721, 'n_tokens_neox': 1643, 'n_words': 593, 'path': '01_homework.ipynb', 'repo_name': 'BorjaRequena/ML_applications', 'size': '5306'}