<jupyter_start><jupyter_text>[[ch02]]<jupyter_code># Transformers and Transfer Learning
Now that you've been introduced to the field of natural language processing, there's something imporant you need to understand. It's not actually a very long journey from where you start to state of the art.

Eventually, we _will_ return to the basics, discuss the fundamentals, and understand all the details, of course. But we're going to show you the promised land before we venture on the long and hard journey to get there.
    
One of the most important ideas to implement if you want to get deep learning working in the real world is transfer learning, which is the process of taking a model that has already been trained on another dataset and fine-tuning it to fit your new dataset. For example, if you're training a language model to generate compelling short stories in the style of Hemingway, you could fine-tune a model trained on a wide variety of books instead of training on just the text samples of Hemingway, of which there may not be many.

.Who's That Pokémon? Language Models
> Tip: A language model is a function that takes in a sequence of words and returns a probility distribution over all the possible next words in that sequence. This task is considered one of the most important in NLP because, as the reasoning goes, to predict the next word in a sentence, you **must** have a good understanding of the language. Language models learn the features and characteristics of language in order to guess what the next word should be after any given prior phrase or sentence. Language models are the backbone of NLP today because they do not require explicit annotations (labels) and can be trained on massive corpuses without any material data preparation. Once they learn the properties of language well, language models can be fine-tuned to perform more specific NLP tasks such as text classification, which is exactly what we're going to do in this chapter.

> Note: When we refer to pretrained models throughout this book, we are generally referring to large, pretrained *language* models that have been trained to perform language modeling on large corpuses.

A nice analogy in object-oriented programming is the concept of inheritance in classes. Suppose we're making some sort of zoo management video game, where each animal is represented by a class. The animals have properties like weight and height, as well as functions like eat and sleep. In theory, we could just create a new class for each animal and replicate those shared functions, but in practice, we usually refactor our code so that we have a superclass for a generic animal and a subclass for each species to avoid duplication in our code, making it easier to read.

By training on the larger dataset, the model essentially inherits a large amount of extra knowledge, which it can use to perform better on the task you care about. From a practical standpoint, transfer learning helps you get better performing models faster since fine-tuning, if done correctly, is often computationally cheaper than training from scratch.footnote:[Assuming that the original dataset you're transferring *from* is much larger than the dataset you're using for fine-tuning. If your fine-tuning dataset is larger, perhaps you should be applying transfer learning the other way around! But in practice, it's very hard to natural language text datasets that are of comparable size to the ones used for pretraining.]

The other big advancement we'll discuss is the use of a new kind of model architecture called the transformer. Training transformers can be complicated and does not always work well without some fine-tuning. So, instead of traning it from scratch, we'll show you the pretraining technique on another architecure, and the use a popular pre-trained transformer to perform inference.

For this chapter, it's important that you have your compute environment set up since we'll be training models. Check out our [Github page](https://github.com/nlpbook/nlpbook/) for more info on how to do this.
## Training with fastai
The first thing we'll look at is an idea called transfer leaning. We're going to fine-tune a language model and then transform it into a text classifier that categorizes text based on sentiment. We'll start with the simplest working implementation, and progressively train our network using the [ULMFit](https://arxiv.org/abs/1801.06146) technique. This particular example was 

The dataset we're going to use here is the IMDB movie review datset. It's not very fun, but it's simple and small, which is what we want when starting off.<jupyter_output><empty_output><jupyter_text>from fastai.text.all import *<jupyter_code>### Using the high-level fastai API
`fastai` is more more than your standard deep learning library. It includes tools that help you solve the problem at hand end-to-end as fast as possible. One of those tools is a built-in set of common datasets that can be easily downloaded.<jupyter_output><empty_output><jupyter_text>path = untar_data(URLs.IMDB)path.ls() (path/'train').ls()<jupyter_code>This particular instance of the IMDB dataset is organized just like ImageNet is (i.e. one directory per class). So in this case, the positive reviews are saved under `pos` and the negative reviews are saved under `neg`.

We can set up set up our dataset and prepare for training by using the `TextDataLoaders.from_folder` method built into `fastai`. The only thing we need to specify is the name of the validation folder, which is "test" (and not the default "valid").<jupyter_output><empty_output><jupyter_text>dls = TextDataLoaders.from_folder(untar_data(URLs.IMDB), valid='test')<jupyter_code>We can then have a look at the data with the `show_batch` method:<jupyter_output><empty_output><jupyter_text>dls.show_batch()<jupyter_code>We can see that the library automatically processed all the texts to split then in *tokens*, adding some special tokens like:

- `xxbos` to indicate the beginning of a text
- `xxmaj` to indicate the next word was capitalized

`fastai` uses an object called a `Learner` for doing pretty much everything. We can construct one for text classification in one line of code:<jupyter_output><empty_output><jupyter_text>learn = text_classifier_learner(dls, AWD_LSTM, drop_mult=0.5, metrics=accuracy)<jupyter_code>Instead of the transformer model that we've been raving about (and will continue to dicuss) throughout a vast majority of the book, we're going to use the [AWD LSTM](https://arxiv.org/abs/1708.02182) architecture instead for now, since it's easier and faster to train.

There are a few other details: `drop_mult` is a parameter that controls the magnitude of all dropouts in that model, and we use `accuracy` to track down how well we are doing. But you don't need to worry too much about hyperparameters just yet.

With the `Learner` defined, we can now fine-tune our pretrained model, using a method with an unsurprising name:<jupyter_output><empty_output><jupyter_text>learn.fine_tune(4, 1e-2) learn.fine_tune(4, 1e-2)<jupyter_code>93% accuracy look good! But let's see how well it's actually doing...<jupyter_output><empty_output><jupyter_text>learn.show_results()<jupyter_code>We can also run prediction on individual sentences one at a time:<jupyter_output><empty_output><jupyter_text>learn.predict("That movie was wicked cool!")<jupyter_code>Here we can see the model has considered the review to be positive. The second part of the result is the index of "pos" in our data vocabulary and the last part is the probabilities attributed to each class (99.1% for "pos" and 0.9% for "neg"). 
#### Building a Dataset with fastai's DataBlock API
We can also use the `fastai` data block API to get our data in a `DataLoaders`. This is a bit more advanced, so fell free to skip this part if you are not comfortable with `fastai` just yet. This approach will give us the same results in the end.

A datablock is built by giving the fastai library a bunch of information:

- the types used, through an argument called `blocks`: here we have images and categories, so we pass `TextBlock` and `CategoryBlock`. To inform the library our texts are files in a folder, we use the `from_folder` class method.
- how to get the raw items, here our function `get_text_files`.
- how to label those items, here with the parent folder.
- how to split those items, here with the grandparent folder.<jupyter_output><empty_output><jupyter_text>imdb = DataBlock(blocks=(TextBlock.from_folder(path), CategoryBlock),                 get_items=get_text_files,                 get_y=parent_label,                 splitter=GrandparentSplitter(valid_name='test'))<jupyter_code>This only gives a blueprint on how to assemble the data. To actually create it, we need to use the `dataloaders` method:<jupyter_output><empty_output><jupyter_text>dls = imdb.dataloaders(path)<jupyter_code>### ULMFiT for Transfer Learning
The pretrained model we used in the previous section is called a language model. It was trained to guess the next word on a set of Wikipedia articles after reading all the words before. We got great results by directly fine-tuning this language model to a movie review classifier, but with one extra step, we can do even better.

The Wikipedia English is slightly different from the IMDb English. So instead of jumping directly to the classifier, we could fine-tune our pretrained language model to the IMDb dataset and then use *that* as the base for our classifier instead of the Wikipedia language model.

This intuitivly makes sense - if you, as a literate human being, get some context on what movie review generally sound like, you'd probably do a better job of classifying them. It's kind of like getting the passage to read a few days in advance before you take the SAT. Only here, we won't call the language model out for cheating, since we're friends footnote:[See, I said it right here. Please don't eat me, robot overlords in the future.].

But beyond that, another very important reason this is useful is because we often have more data for our than we have *labelled* data. Labelling is expensive and generally requires human time and effort, so it's not uncommon to have a large database of text record where only a small subset of them are used for say, document tagging. But with this fine-tuning approach, we can still use the unlabelled data to fine-tune the *language model* even before we train the 

At the risk of dragging on a flawed analogy, this is almost like getting access to years of previous SAT passages. None of them will show up on the test *exactly*, but practicing them will help get a sense of what the SAT is like.

This approach is called ULMFiT, introducted by <NAME> footnote:[Who also happends to be the creator of fastai!] and <NAME> in 2018. The process is summarized in [[ulmfit]]

![ULMFit](images/ulmfit.png)
Arrows and circles make everything so much simpler, don't they?

Since we already have the pretrained Wikipedia language model, we can start with step 2 of the piple in [[ulmfit]] - fine-tuning the IMDB language model.
### Fine-tuning a language model on IMDb
We can get our texts in a `DataLoaders` suitable for language modeling very easily:<jupyter_output><empty_output><jupyter_text>dls_lm = TextDataLoaders.from_folder(path, is_lm=True, valid_pct=0.1)<jupyter_code>We need to pass something for `valid_pct` otherwise this method will try to split the data by using the grandparent folder names. By passing `valid_pct=0.1`, we tell it to get a random 10% of those reviews for the validation set.

We can have a look at our data using `show_batch`. Here the task is to guess the next word, so we can see the targets have all shifted one word to the right.<jupyter_output><empty_output><jupyter_text>dls_lm.show_batch(max_n=5)<jupyter_code>Then we have a convenience method to directly grab a `Learner` from it, using the `AWD_LSTM` architecture like before. We use accuracy and perplexity as metrics (the later is the exponential of the loss) and we set a default weight decay of 0.1. `to_fp16` puts the `Learner` in mixed precision, which is going to help speed up training on GPUs that have Tensor Cores.<jupyter_output><empty_output><jupyter_text>learn = language_model_learner(    dls_lm, AWD_LSTM, metrics=[accuracy, Perplexity()],    path=path, wd=0.1).to_fp16()<jupyter_code>By default, a pretrained `Learner` is in a frozen state, meaning that only the head of the model will train while the body stays frozen. We show you what is behind the fine_tune method here and use a fit_one_cycle method to fit the model:<jupyter_output><empty_output><jupyter_text>learn.fit_one_cycle(1, 1e-2)<jupyter_code>This model takes a while to train, so it's a good opportunity to talk about saving intermediary results. 
You can easily save the state of your model like so:<jupyter_output><empty_output><jupyter_text>learn.save('1epoch')<jupyter_code>It will create a file in `learn.path/models/` named "1epoch.pth". If you want to load your model on another machine after creating your `Learner` the same way, or resume training later, you can load the content of this file with:<jupyter_output><empty_output><jupyter_text>learn = learn.load('1epoch')<jupyter_code>We can them fine-tune the model after unfreezing:<jupyter_output><empty_output><jupyter_text>learn.unfreeze()learn.fit_one_cycle(10, 1e-3)<jupyter_code>Once this is done, we save all of our model except the final layer that converts activations to probabilities of picking each token in our vocabulary. The model not including the final layer is called the *encoder*. We can save it with `save_encoder`:<jupyter_output><empty_output><jupyter_text>learn.save_encoder('finetuned')<jupyter_code>.Who's That Pokémon?
> Tip: The encoder is the model not including the task-specific final layer(s). It means much the same thing as *body* when applied to vision CNNs, but tends to be more used for NLP and generative models.
Before using this to fine-tune a classifier on the reviews, we can use our model to generate random reviews: since it's trained to guess what the next word of the sentence is, we can use it to write new reviews:<jupyter_output><empty_output><jupyter_text>TEXT = "I liked this movie because"N_WORDS = 40N_SENTENCES = 2preds = [learn.predict(TEXT, N_WORDS, temperature=0.75)          for _ in range(N_SENTENCES)] print("\n".join(preds))<jupyter_code>With the language model fine-tuned on movie review, we can now modify it to *classify* movie reviews. The idea is that at this point, if the model is "smart enough" to predict the next word, it *must* be able to a simple positive/negative classification.
### Training a text classifier
We can gather our data for text classification almost exactly like before:<jupyter_output><empty_output><jupyter_text>dls_clas = TextDataLoaders.from_folder(    untar_data(URLs.IMDB), valid='test',    text_vocab=dls_lm.vocab)<jupyter_code>The main difference is that we have to use the exact same vocabulary as when we were fine-tuning our language model, or the weights learned won't make any sense. We pass that vocabulary with `text_vocab`.

Then we can define our text classifier like before:<jupyter_output><empty_output><jupyter_text>learn = text_classifier_learner(dls, AWD_LSTM, drop_mult=0.5, metrics=accuracy)<jupyter_code>The difference is that before training it, we load the previous encoder:<jupyter_output><empty_output><jupyter_text>learn = learn.load_encoder('finetuned')<jupyter_code>The last step is to train with discriminative learning rates and *gradual unfreezing*. In computer vision, we often unfreeze the model all at once, but for NLP classifiers, we find that unfreezing a few layers at a time makes a real difference.<jupyter_output><empty_output><jupyter_text>learn.fit_one_cycle(1, 2e-2)<jupyter_code>In just one epoch we get the same result as our training in the first section, not too bad! We can pass `-2` to `freeze_to` to freeze all except the last two parameter groups:<jupyter_output><empty_output><jupyter_text>learn.freeze_to(-2)learn.fit_one_cycle(1, slice(1e-2/(2.6**4),1e-2))<jupyter_code>Then we can unfreeze a bit more, and continue training:<jupyter_output><empty_output><jupyter_text>learn.freeze_to(-3)learn.fit_one_cycle(1, slice(5e-3/(2.6**4),5e-3))<jupyter_code>Finally, we can unfreeze the entire model, and let it train all the layers to get a final boost in accuracy.<jupyter_output><empty_output><jupyter_text>learn.unfreeze()learn.fit_one_cycle(2, slice(1e-3/(2.6**4),1e-3))<jupyter_code>Now, you have a text classification model that can accuractely predict if a movie review has positive or negative sentiment based on the raw text content of the review alone. With an understanding of the `fastai` APIs, you should now be able to implement your own text classifier on a dataset of your choice.

While IMDB itself was fairly simple, many NLP problems today can be formulated a text classification problems. Some of the things you can do with text classifcation include:

- Predicting the programming language of some source code
- Building a simple email spam classifier
- Improving the functionality of an automated content moderation bot for online chats or forums
- Categorize documents based on their language footnote:[To do this well, you need a powerful tokenizer that can recognize text encoding in many languages]

One of the best parts about text classification is that there's a single, simple, interpretable metric to optimize - accuracy. So not only can you solve these tasks, but you can also know how well you're doing using statistics that many people are familiar with.

While the IMDB model we built just now does a wonderful job, it's perhaps not super impressive. We've had spam classifiers that do pretty good since the dawn of the Dinosaurs, so binary predictions on text is not something you might associate with the glorious future we sold you on in [[ch01]]. But it turns out that this idea of a language model is so powerful that it has become the poster child for NLP today.

To illustrate this, let's give a language model on it's own, with no additional training or fine-tuning, a change to flex it's muscles.
## Inference with Huggingface
Now that we know how to train language models, we could conceptually train a very, very large one on a lot of data, and get it to produce very accurate sounding text. Here, we'll use the huggingface library to get prediction samples from a language model that was trained using a procedure similar to the one we used above.<jupyter_output><empty_output><jupyter_text>import torchfrom transformers import GPT2Tokenizer, GPT2LMHeadModel Load pre-trained model tokenizer (vocabulary)tokenizer = GPT2Tokenizer.from_pretrained('gpt2') Encode a text inputstext = "With great power comes great "indexed_tokens = tokenizer.encode(text) Convert indexed tokens in a PyTorch tensortokens_tensor = torch.tensor([indexed_tokens])<jupyter_code>The code snippet above initializes a tokenizer, which is function thats strings as input and returns arrays of numbers that are easier for the model to interpret. We'll be covering tokenizers in much more detail in the next chapter, but for now, if you want a quick look into what our model sees, try printing `tokens_tensor`.<jupyter_output><empty_output><jupyter_text>print(tokens_tensor)<jupyter_code>Now, let's do the actual inference, which is, again, just a few lines of code thanks to the amazing huggingface transformers library.<jupyter_output><empty_output><jupyter_text>Load pre-trained model (weights)model = GPT2LMHeadModel.from_pretrained('gpt2') Set the model in evaluation mode to deactivate the DropOut modules This is IMPORTANT to have reproducible results during evaluation!model.eval() Predict all tokenswith torch.no_grad():    outputs = model(tokens_tensor)    predictions = outputs[0] get the predicted next sub-wordpredicted_index = torch.argmax(predictions[0, -1, :]).item()predicted_text = tokenizer.decode(indexed_tokens + [predicted_index])print(predicted_text)<jupyter_code>Nice! It looks like whatever we just ran, in just a few lines of code, was able to recreate the wisdom of uncle Ben!footnote[A character from the Spider-Man comics book series, who once said "with great power comes great responsibility," just like our language model did!]

And to be clear, this wasn't just some simple lookup, database search, or something like that. This was an actual state-of-the-art neural network that after reading large amounts of text on the internet, is able to complete senences based in the "knowledge" it gained. Pretty cool, huh?

But without context, this is all just a black box that you throw sentences into. So now, let's break down each line code in the block we just ran to really get a good idea of what's going on.
### Loading Models
First, we load a pretrained model. This is the single most important step for transfer learning. It downloads the model that we're going to use to make predictions from somewhere on the internet and loads it in the right format into  an object in our code. All of that functionality is thankfully packed into this one line of code:<jupyter_output><empty_output><jupyter_text>model = GPT2LMHeadModel.from_pretrained('gpt2')<jupyter_code>Most deep learning libraries package this model loading functionality neatly into a simple function. It's the last thing you'll have to worry about.

The specific model we're loading here is unimportant at this stage, but just so you know, it's called "GPT2", which was really revolutionary when it came out and basically broke the internet. You can read more about it from [an article](https://blog.floydhub.com/gpt2/) that Ajay wrote in 2019, but we'll talk about in this book as well, in [[ch09]].
.Loading Models
> **Note:** Loading models to a variable named +model+, regardless of the task or domain is something that's extremely common in deep learning, so keep that in mind when you're browsing notebooks or code samples online.
Next we run this little line of code, which tells our model that we're not training now and are instead going to make predictions (i.e., perform inference). There are a few things that change internally in the `model` object when we call this linefootmote:[Primarily, we disable the DropOut and BatchNorm layers, which are only useful during training], allowing us to generate predictions from the `model`. Again, this is not the most important line for what we're doing now, but make sure that you call this function whenever you would like to generate predictions. Running this line in a notebook will also print out all the layers of the model in the standard PyTorch format, so maybe scroll through that if you're feeling curious.<jupyter_output><empty_output><jupyter_text>model.eval()<jupyter_code>With the weights downloaded, modedl loaded into memory, and the `model` object set into evaluation mode, It's time to crank out some output from our lean footnote:[Ok, maybe this phrase is not applicable to GPT-2 specifically, but when we all have computers that are 300 times faster than what we have today, this adjective will be accurate.], mean, text generating machine.
### Generating Predictions
We're going to group the next three lines together, since they work as a block.<jupyter_output><empty_output><jupyter_text>with torch.no_grad():    outputs = model(tokens_tensor)    predictions = outputs[0]<jupyter_code>The first line, `with torch.no_grad():` tells PyTorch to run the the lines in that indent block in the `torch.no_grad()` context, which means PyTorch won't calculate the gradients, or backward pass, for the model. If you're not familiar with backpropogation, or not entirely clear why gradients are calculated in the first place, refer to the resources we have in the introduction. Strictly speaking, we don't _need_ to turn off gradient computation, but this saves time, memory, and compute, and makes the inference run faster.

In the `torch.no_grad()` context, we then run a forward pass. As always, PyTorch makes this extremely simple. Just call `model` as a function, with the `tokens_tensor` we prepared above as the input.
But wait, wasn't +model+ an object with the pretrained weights that we loaded above? How is it also a function?
.Python Dunder Methods
****

In Python, you can actually do this! You have to define `__call__` method in your class, which is a special function called a dunder method. Python has a lot of these cool dunder functions, some of which you've likely encounter before, like `__init__`, which let's you set up a constructor for your class, and `__len__`, which let's you define a "length" property for your objects which you can access via the `len()` function. Python dunder methods allow you to define a lot of cool functionality for your custom classes, such as addition, equality, and more.

If you define a function called `__call__()` in your Python class, you can then treat instances of your class as functions, and the `__call__()` function will be invoked everytime you do so. We'll soon talk about PyTorch `nn.Module` objects, which are the building blocks for neural nets. The `nn.Module` class implements the `__call__` function by default. Therefore, every PyTorch model (and submodule) can also be called as a function, which can make your code very neat and tidy. This is why we can both define the +model+ variable and call it as we would for a function at the same time.

If you're interested in learning more about python dunder methods, check out this [tutorial](https://rszalski.github.io/magicmethods/) or read more online (there are plenty of great resources one search away).

****
Calling `model(input_tensor)`, in general, will return a `torch.tensor` object with the predictions. But in this case, the huggingface library actually gives us a lot of other items as well. In this case, `model(tokens_tensor)` will return a tuple, where the first element is the predictions tensor. Let's quickly confirm alll of this by checking a few lengths and shapes.<jupyter_output><empty_output><jupyter_text>len(outputs)<jupyter_code>outputs[0].shape<jupyter_output><empty_output><jupyter_text>This checks out, because according to the huggingface transformers documentation, the predictions tensor is supposed to have shape `(batch_size, sequence_length, config.vocab_size)`. Here, the batch size is 1, since we're only passing in one sentence. The sequence length should be 5, which makes sense if you take a look at the line where we define the input sentence, which had five words (space delimitted substrings) in the string:```text = "With great power comes great "```The value of 50257 for the vocabulary size seems accurate, but this is something we could always double-check by going through the documentation for this model.> **Tip:** We can't emphasize enough how much this technique of checking the size, shape, and dimensions of `torch.tensors` is. It's one of the most effective ways of debugging your model. Hopefully, as you start training more complex models and building your own architectures from scratch, this will come naturally. But until then, always remember to try check the size with `.size` and reason through what's going on in your model. Since it seems like `outputs[0]` is what we want, we'll assign to the variable `predictions`. Putting these together and wrapping them in the `torch.no_grad()` context gives us that mini-block of code that we had above.<jupyter_code>with torch.no_grad():
    outputs = model(tokens_tensor)
    predictions = outputs[0]<jupyter_output><empty_output><jupyter_text>`predictions` is a `torch.tensor` that has values that describe the probability of each word. Remember, one of the dimensions of this `torch.tensor` is the size of the vocabulary (i.e. the number of possible words that the model could predict). What we want now is the word that is mostly likey to come next in out sentence, We grab this by using the `argmax` function, which gets the index of the largest value in the array.<jupyter_code>predicted_index = torch.argmax(predictions[0, -1, :]).item()<jupyter_output><empty_output><jupyter_text>To ensure that we're absolutely clear on the what exactly we're doing, let's also quickly break down the way we index `predictions`. It's a three dimensional tensor, so we specify 3 indices. The first, along the batch dimension, is `0`. Since we're not running batch predictions, there's only one element in this axis, so it's what we pick. Along the sequence length dimension, we pick the last element. This is because we want to predict the last word in the sentence we passed in. The last index is `:`, which means we want to grab everything. We need all the elements along the vocabulary dimension to calculate which one is most likely.Finally, we decode the index we got into a word using the +tokenizer.decode()+ funtion. This is just a simple lookup.<jupyter_code>predicted_text = tokenizer.decode(indexed_tokens + [predicted_index])
print(predicted_text)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.05861701029153616, 'fraction_numerical': 0.004922039032113724, 'mean_word_length': 5.176445578231292, 'pattern_counts': {'":': 0, '<': 151, '<?xml version=': 0, '>': 156, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '37', 'dir': 'jupyter-structured-clean-dedup', 'id': '279730', 'license': 'MIT', 'n_tokens_mistral': 7814, 'n_tokens_neox': 7509, 'n_words': 4143, 'path': 'ch02.ipynb', 'repo_name': 'Gippers/nlpbook', 'size': '29069'}