<jupyter_start><jupyter_text>ULMFiT ExampleBased on tutorials from:* https://docs.fast.ai/tutorial.text.html* https://github.com/floleuerer/fastai_ulmfit* https://github.com/fastai/course-v3/blob/master/nbs/dl1/lesson3-imdb.ipynb This will be a basic tutorial of how to use ULMFiT on the IMBD dataset. For further exploration into the capabilities of ULMFiT, I highly recommend the third linked website above.  Environment Setup Before running the notebook code, be sure to switch the runtime to GPU. (Edit > Notebook Settings > Hardware Accelerator = GPU)<jupyter_code># if you are using google colab, you will need to run this line & restart runtime
# before importing fastai below
!pip install fastai --upgrade
from fastai.text.all import *<jupyter_output><empty_output><jupyter_text>Read in IMDB dataset and store in a `DataLoaders`<jupyter_code>path = untar_data(URLs.IMDB)
#path.ls() # see information stored in this folder
dls = TextDataLoaders.from_folder(path, valid='test', bs=8)
dls.show_batch()<jupyter_output><empty_output><jupyter_text>Above, we see that our data has been processed and tokenized, as evidence by the special tokens added:* xxbos: beginning of text* xxmaj: next work is capitalized  Fine-tune language model Recall the steps of ULMFiT: 1. pretrain large language model2. fine-tune on domain data3. add classification task headThe first step is already done for us on an AWD_LSTM model with Wiki-text data. The step we start with is 2, fine-tune on domain data.Below, we initialize a `Learner` with our data, the AWD_LSTM model, the metrics, and a default weight decay of 0.1 and drop_mult of 0.1<jupyter_code>learn_lm = language_model_learner(dls, AWD_LSTM,
                                  metrics=[accuracy, Perplexity()],
                                  path=path,
                                  wd=0.1, drop_mult=0.1).to_fp16()<jupyter_output><empty_output><jupyter_text>To begin fine-tuning, all we need to do is set the number of epochs and the intial learning rate. NOte that the pretrained `Learner` is in a frozen state and only the head of the model will be trained. For the sake of time, I will only be training 4 epochs. More will be needed for better performance.<jupyter_code>learn_lm.unfreeze()
learn_lm.fit_one_cycle(4, 1e-3)<jupyter_output><empty_output><jupyter_text>Once this is done, we save all of our model except the final layer that converts activations to probabilities of picking each token in our vocabulary.We save this final layer with `save_encoder`<jupyter_code>learn_lm.save('finetuned_lm')
learn_lm.save_encoder('finetuned_lm_enc')<jupyter_output><empty_output><jupyter_text>Fine-tune text classifier Like before, we gather data again. This time, we need to include only vocab that was included in the fine-tuned model.<jupyter_code>dls_class = TextDataLoaders.from_folder(path, valid='test', text_vocab=dls.vocab)<jupyter_output><empty_output><jupyter_text>Once again, we create a learner...<jupyter_code>learn_class = text_classifier_learner(dls_class, AWD_LSTM, drop_mult=0.5, metrics=accuracy)<jupyter_output><empty_output><jupyter_text>...and we load the previous encoder that we just saved!<jupyter_code>learn_class = learn_class.load_encoder('finetuned_lm_enc')<jupyter_output><empty_output><jupyter_text>Next, we fine-tune by freezing all but a few layers at a time until we unfreeze all. This is the gradual unfreezing step discussed in the paper. Note that we start with an initial learning rate here.<jupyter_code>learn_class.fit_one_cycle(1, 2e-2)<jupyter_output><empty_output><jupyter_text>Freeze all but last two layer. Note that we are changing the learning rate for these layers compared to the previous by making them smaller! This is discriminative fine-tuning.<jupyter_code>learn_class.freeze_to(-2)
learn_class.fit_one_cycle(1, slice(1e-2/(2.6**4),1e-2))<jupyter_output><empty_output><jupyter_text>Unfreeze all but last three layers<jupyter_code>learn_class.freeze_to(-3)
learn_class.fit_one_cycle(1, slice(5e-3/(2.6**4),5e-3))<jupyter_output><empty_output><jupyter_text>Finally, do entire model<jupyter_code>learn_class.unfreeze()
learn_class.fit_one_cycle(2, slice(1e-3/(2.6**4),1e-3))<jupyter_output><empty_output><jupyter_text>And save!<jupyter_code>learn_class.save('learn_class_unfreezed_final')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0875, 'fraction_numerical': 0.012268518518518519, 'mean_word_length': 5.678516228748068, 'pattern_counts': {'":': 0, '<': 53, '<?xml version=': 0, '>': 55, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '232093', 'license': 'MIT', 'n_tokens_mistral': 1453, 'n_tokens_neox': 1405, 'n_words': 426, 'path': 'ULMFiT_example.ipynb', 'repo_name': 'amandakonet/ulmfit', 'size': '4320'}