<jupyter_start><jupyter_text>Big Models Hate This One Weird Trick! (Quantization, T5, & PyTorch 1.4)As we know, models can be big lumbering beasts, comprised of millions of parameters (both weights and activations) that require lots of matrix multiplications to take an input and arrive at an answer. And for most of our work so far, that's been fine! We have mighty GPUs that can handle these burdens with ease.But what if we didn’t? We often package a model up for production inference usage so that it only runs on the CPU. And what if we wanted to run our model on a smaller embedded platform? Suddenly, both the size of the model and all those floating-point operations become a little more problematic. Thankfully, there’s a trick we can perform that makes our model smaller _and_ faster, normally with the trade off with some accuracy. Even better, PyTorch allows us to perform this one weird trick with just one line of code, with some other approaches for squeezing even more performance. Let’s have a quick look at _quantization_.  QuantizationEvery parameter in our model is a 32-bit floating point number, taking up 4 bytes of memory. That’s not a lot, but it can soon add up. Let's have a look at Google’s recent T5 transformer-based model, which has a `t5-small` variant that’s available in the `transformers` library.<jupyter_code>import torch
from transformers import pipeline, T5ForConditionalGeneration
		
def count_parameters(model):
  return sum(p.numel() for p in model.parameters())
	    
base_model = T5ForConditionalGeneration.from_pretrained("t5-small")

param_count = count_parameters(base_model)

memory = (param_count * 4) / (1024 *1024)
memory<jupyter_output><empty_output><jupyter_text>Even with the smallest pre-trained T5 weights, our model is roughly 60m parameters and weighs in at a whopping 230Mb!  However, what if we decided that we didn’t need the full precision of our floating-point parameters? If our parameters could be restricted to within a certain range of values, then we could use a smaller type of number representation to store the parameters. This _quantization_ is the key to speeding up our inference time and reducing the memory footprint of our models. What we tend to aim for is to quantize down from a 32-bit floating point to an 8-bit integer. The basic idea is: $x_{int8} = (\frac{x_{float32}}{x_{scale}} + x_{offset})$Which is essentially just fitting the potential values of the parameters of a network to a line of $y = mx + c$, although due to the reduced resolution of the 8-bit integer, there's only so many values a parameter now may take instead of the huge amount that a `float32` value could be. PyTorch does its quantizing in a slightly more complicated affair that ensures that zero is always zero, but the basic idea is the same - we have a range of values that our parameters can take, and then find an appropriate pair $x_{scale}$ and $x_{offset}$ to provide 256 graduations to represent that range - or 255 if you think about PyTorch always keeping zero around. At the moment (PyTorch 1.5), quantized layers are best supported with `CNN` and `Linear` layers. Thankfully, if we have a look at the model structure of T5, we can see a happy coincidence:<jupyter_code>base_model<jupyter_output><empty_output><jupyter_text>Yes, that’s right, look at all those `Linear` layers! We should be able to get some benefit out of quantizing this model.   One Weird Trick — Dynamic Quantization<jupyter_code>import torch.quantization
	
quantized_model = torch.quantization.quantize_dynamic(base_model, {torch.nn.Linear}, dtype=torch.qint8)<jupyter_output><empty_output><jupyter_text>No, really, that’s it. Chapter done. Bye!Oh, okay, if you really insist. But honestly, there’s not much more to it. Okay, firstly, a caveat in that `quantize_dynamic` will only quantize the weights, not the activations in our parameters. But all we need to do is pass in the `model` we wish to quantize and a dict of layers that we wish to replace with our quantized versions, in this case `Linear`. The function returns a new model, though you could run with the optional parameter `inplace=True` to mutate the original model rather than make a copy. Let’s save the model and take a look at the quantized size:<jupyter_code>!mkdir t5
quantized_model.save_pretrained("t5")
!du -m t5<jupyter_output>mkdir: cannot create directory ‘t5’: File exists
121	t5<jupyter_text>Almost a 50% reduction in size! We can’t get down to 4 times smaller due to not being able to store the activations as 8-bit integers, but we’ve done pretty well for one line of code. Let's do a very simple microbenchmark using both models in the `transformers` library summarization pipeline.<jupyter_code>base_summarizer = pipeline("summarization", model=base_model, tokenizer="t5-small")
quantized_summarizer = pipeline("summarization", model=quantized_model, tokenizer="t5-small")
%timeit base_summarizer("From the very beginning, Regan was seen as having series potential. After the television film scored highly in the ratings, work began on the development of the series proper. <NAME>'s idea was for the series to be mainly studio-based, with more dialogue and less action, but producer <NAME> disagreed, and in consequence I<NAME> parted company with the project. Childs produced it on 16mm film, a format that allowed for a much smaller film unit than videotape at that time. This made it possible to shoot almost entirely on location which helped give the series a startling degree of realism and to use film editing techniques which enabled him to give the show a heavy bias toward action sequences. The television play and the subsequent series were commissioned by Thames Television and produced by its film division Euston Films. It was originally broadcast on ITV between 2 January 1975 and 28 December 1978 at 21:00–22:00 on weekdays (usually Mondays), with repeated screenings at the same time until the early 1980s. The writers were given strict guidelines to follow: \"Each show will have an overall screen time (minus titles) of 48 minutes 40 seconds. Each film will open with a teaser of up to 3 minutes, which will be followed by the opening titles. The story will be played across three acts, each being no more than 19 minutes and no less than 8 minutes in length. Regan will appear in every episode, Carter in approximately 10 out of 13 episodes. In addition to these main characters, scripts should be based around three major speaking parts, with up to ten minor speaking parts")
%timeit quantized_summarizer("From the very beginning, Regan was seen as having series potential. After the television film scored highly in the ratings, work began on the development of the series proper. <NAME>'s idea was for the series to be mainly studio-based, with more dialogue and less action, but producer <NAME> disagreed, and in consequence <NAME> parted company with the project. Childs produced it on 16mm film, a format that allowed for a much smaller film unit than videotape at that time. This made it possible to shoot almost entirely on location which helped give the series a startling degree of realism and to use film editing techniques which enabled him to give the show a heavy bias toward action sequences. The television play and the subsequent series were commissioned by Thames Television and produced by its film division Euston Films. It was originally broadcast on ITV between 2 January 1975 and 28 December 1978 at 21:00–22:00 on weekdays (usually Mondays), with repeated screenings at the same time until the early 1980s. The writers were given strict guidelines to follow: \"Each show will have an overall screen time (minus titles) of 48 minutes 40 seconds. Each film will open with a teaser of up to 3 minutes, which will be followed by the opening titles. The story will be played across three acts, each being no more than 19 minutes and no less than 8 minutes in length. Regan will appear in every episode, Carter in approximately 10 out of 13 episodes. In addition to these main characters, scripts should be based around three major speaking parts, with up to ten minor speaking parts")<jupyter_output>1 loop, best of 3: 16.6 s per loop
{'fraction_non_alphanumeric': 0.046491763270286764, 'fraction_numerical': 0.01683953630262355, 'mean_word_length': 4.952069716775599, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '332618', 'license': 'MIT', 'n_tokens_mistral': 2144, 'n_tokens_neox': 1992, 'n_words': 1261, 'path': 'chapter7/Chapter_7_5_Quantizing_Models.ipynb', 'repo_name': 'trackitaero/beginners-pytorch-deep-learning', 'size': '8250'}