<jupyter_start><jupyter_text>Transfer learning in text> How to fine-tune a language model and train a classifier  Finetune a pretrained Language Model First we get our data and tokenize it.<jupyter_code>path = untar_data(URLs.IMDB_SAMPLE)
df = pd.read_csv(path/'texts.csv')
df_tok,count = tokenize_df(df, 'text')<jupyter_output><empty_output><jupyter_text>Then we put it in a `DataSource`. For a language model, we don't have targets, so there is only one transform to numericalize the texts. Note that `tokenize_df` returns the count of the words in the corpus to make it easy to create a vocabulary.<jupyter_code>splits = RandomSplitter()(range_of(df_tok))
vocab = make_vocab(count)
dsrc = DataSource(df_tok, [[attrgetter("text"), Numericalize(vocab)]], splits=splits, dl_type=LMDataLoader)<jupyter_output><empty_output><jupyter_text>Then we use that `DataSource` to create a `DataBunch`. Here the class of `TfmdDL` we need to use is `LMDataLoader` which will concatenate all the texts in a source (with a shuffle at each epoch for the training set), split it in `bs` chunks then read continuously through it.<jupyter_code>dbunch = dsrc.databunch(bs=64, seq_len=72, after_batch=Cuda)
dbunch.show_batch()<jupyter_output><empty_output><jupyter_text>Then we have a convenience method to directly grab a `Learner` from it, using the `AWD_LSTM` architecture.<jupyter_code>learn = language_model_learner(dbunch, AWD_LSTM, vocab, metrics=[accuracy, Perplexity()], path=path, opt_func = partial(Adam, wd=0.1)).to_fp16()
learn.freeze()
learn.fit_one_cycle(1, 1e-2, moms=(0.8,0.7,0.8))
learn.unfreeze()
learn.fit_one_cycle(4, 1e-2, moms=(0.8,0.7,0.8))<jupyter_output><empty_output><jupyter_text>Once we have fine-tuned the pretrained language model to this corpus, we save the encoder since we will use it for the classifier.<jupyter_code>learn.show_results()
learn.save_encoder('enc1')<jupyter_output><empty_output><jupyter_text>Use it to train a classifier For classification, we need to use two set of transforms: one to numericalize the texts and the other to encode the labels as categories.<jupyter_code>splits = RandomSplitter()(range_of(df_tok))
dsrc = DataSource(df_tok, splits=splits, tfms=[
    [attrgetter("text"), Numericalize(vocab)],
    [attrgetter("label"), Categorize()]], dl_type=SortedDL)<jupyter_output><empty_output><jupyter_text>We opnce again use a subclass of `TfmdDL` for the dataloaders, since we want to sort the texts (sortish for the training set) by order of lengths. We also use `pad_collate` to create batches form texts of different lengths.<jupyter_code>dbunch = dsrc.databunch(before_batch=pad_input, after_batch=Cuda)
dbunch.show_batch(max_n=2)<jupyter_output><empty_output><jupyter_text>Then we once again have a convenience function to create a classifier from this `DataBunch` with the `AWD_LSTM` architecture.<jupyter_code>learn = text_classifier_learner(dbunch, AWD_LSTM, vocab, metrics=[accuracy], path=path, opt_func=Adam, drop_mult=0.5)
learn = learn.load_encoder('enc1')<jupyter_output><empty_output><jupyter_text>Then we can train with gradual unfreezing and differential learning rates.<jupyter_code>learn.fit_one_cycle(4, moms=(0.8,0.7,0.8))
learn.unfreeze()
learn.opt = learn.create_opt()
learn.fit_one_cycle(8, slice(1e-5,1e-3), moms=(0.8,0.7,0.8))
learn.show_results(max_n=5)
from local.interpret import *
interp = Interpretation.from_learner(learn)
interp.plot_top_losses(6)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1090171064076544, 'fraction_numerical': 0.014786894752102058, 'mean_word_length': 7.43520782396088, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 38, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '17184', 'license': 'Apache-2.0', 'n_tokens_mistral': 1230, 'n_tokens_neox': 1193, 'n_words': 325, 'path': 'dev/38_tutorial_ulmfit.ipynb', 'repo_name': 'BenjaminDev/fastai_dev', 'size': '3449'}