<jupyter_start><jupyter_text>Please put down your weapon. > You have twenty seconds to comply.- toc: true - badges: true- comments: true- categories: [jupyter]- image: images/posts/2020-12-17-Put-Down-Your-Weapon/Put-Down-Your-Weapon.jpg<jupyter_code>#hide
!pip install -Uqq fastbook
import fastbook
fastbook.setup_book()
#hide
from fastbook import *<jupyter_output><empty_output><jupyter_text>Image Classification Up to this point we have created a Neural Network using only 4 lines of code with fastai's high level API and we have created a Neural Network from scratch using PyTorch only for its tensors and back-propogation functions. Now we are going to start trying to coalesce these two methodologies and attempt to meet somewhere in the middle. In this chapter we are looking at multi-image classification, it is quite a step up from the work we did in Chapter 2 where we identified Star Wars Robot or the simple binary classification in Chapter 4 where we attempted to disambiguate between hand written characters of 3s and 7s. Here we are going to attempt to identify breeds of cats and dogs with 37 different catagories. It is no mean feat.  Also if you are interested in the name of this blog post, its one of my favourite scenes from Robocop, where ED-209 cannot identify whether or not Kinney has reliquished his firearm. Let's hope we can do a better job here!  From Dogs and Cats to Pet Breeds Thankfully the fantastic team at fastai have provided a dataset for us, moving forward and looking at our own problems we will need to create them ourselves, but for now we can simply run `untar_data` and pass in the `URLs.PETS` url to download, extract and return a local path to the data<jupyter_code>from fastai.vision.all import *
path = untar_data(URLs.PETS)
#hide
Path.BASE_PATH = path<jupyter_output><empty_output><jupyter_text>Again here we can use the `ls` function to look inside our directory to what our dataset folder contains<jupyter_code>path.ls()<jupyter_output><empty_output><jupyter_text>Lets take a look inside the `images` folder. Here we can see that we have 7393 images and the first ten paths. Similarly to our example in Chapter 1, file names that start with a capital letter are `Cat` breeds and file names beginning with lowercase letters are `Dog` breeds. But we need to differtiate them even further in this example, into the specific breeds.<jupyter_code>(path/"images").ls()<jupyter_output><empty_output><jupyter_text>We can do that using a Regular Expression. The format of the file names are: {ANIMAL_BREED}\_{INTEGER}.{EXTENSION} Regular expressions are a fantastic utility that can parse strings are return specific data from them. There is a great resource that can be found as part of the [fastai nlp course](https://github.com/fastai/course-nlp/blob/master/4-regex.ipynb). First lets store first image path into a variable.<jupyter_code>fname = (path/"images").ls()[0]<jupyter_output><empty_output><jupyter_text>Here by using our regular expression `r'(.+)_\d+.jpg$'` we are able to extract the name of our first breed. `(.+)` this part of the expression is looking for the breed using `re.findall()` and it will return that value. This represents the {ANIMAL_BREED} part of the name format shown above.  `_\d+.jpg$'` this part of the expression is looking for the underscore `_` the {INTEGER} `\d+` and the .{EXTENSION} `.jpg`  Here we can see it worked! By passing in our first image path our Regular Expression returns `Ragdoll`<jupyter_code>re.findall(r'(.+)_\d+.jpg$', fname.name)<jupyter_output><empty_output><jupyter_text>So now, lets create a DataBlock from our dataset. We define that our Dependent Variable is of type `ImageBlock` and our Independent Variable is of type `CatagoryBlock`.Here we are using a new function and a new class. Here we are passing `RegexLabeller` object and the `name` variable to fastai's `using_attr` function. This will perform our regex segmentation on every files `name` attribute to return our image labels into the DataBlocks `get_y` argument.<jupyter_code>pets = DataBlock(blocks = (ImageBlock, CategoryBlock),
                 get_items=get_image_files, 
                 splitter=RandomSplitter(seed=42),
                 get_y=using_attr(RegexLabeller(r'(.+)_\d+.jpg$'), 'name'),
                 item_tfms=Resize(460),
                 batch_tfms=aug_transforms(size=224, min_scale=0.75))
dls = pets.dataloaders(path/"images")<jupyter_output><empty_output><jupyter_text>We also have two new lines that perform resizing operations. Once on a per item level and one at a batch level, this is a process the team at fastai call `Presizing`  Presizing ![](2020-12-17-Put-Down-Your-Weapon/Presizing.jpg) - Resize images to relatively "large" dimensions—that is, dimensions significantly larger than the target training dimensions.- Compose all of the common augmentation operations (including a resize to the final target size) into one, and perform the combined operation on the GPU only once at the end of processing, rather than performing the operations individually and interpolating multiple times.The first step, the resize, creates images large enough that they have spare margin to allow further augmentation transforms on their inner regions without creating empty zones. This transformation works by resizing to a square, using a large crop size. On the training set, the crop area is chosen randomly, and the size of the crop is selected to cover the entire width or height of the image, whichever is smaller.In the second step, the GPU is used for all data augmentation, and all of the potentially destructive operations are done together, with a single interpolation at the end.<jupyter_code>dblock1 = DataBlock(blocks=(ImageBlock(), CategoryBlock()),
                   get_y=parent_label,
                   item_tfms=Resize(460))
dls1 = dblock1.dataloaders([(Path.cwd().parent/'images'/'grizzly.jpg')]*100, bs=8)
dls1.train.get_idxs = lambda: Inf.ones
x,y = dls1.valid.one_batch()
_,axs = subplots(1, 2)

x1 = TensorImage(x.clone())
x1 = x1.affine_coord(sz=224)
x1 = x1.rotate(draw=30, p=1.)
x1 = x1.zoom(draw=1.2, p=1.)
x1 = x1.warp(draw_x=-0.2, draw_y=0.2, p=1.)

tfms = setup_aug_tfms([Rotate(draw=30, p=1, size=224), Zoom(draw=1.2, p=1., size=224),
                       Warp(draw_x=-0.2, draw_y=0.2, p=1., size=224)])
x = Pipeline(tfms)(x)
#x.affine_coord(coord_tfm=coord_tfm, sz=size, mode=mode, pad_mode=pad_mode)
TensorImage(x[0]).show(ctx=axs[0])
TensorImage(x1[0]).show(ctx=axs[1]);<jupyter_output><empty_output><jupyter_text>As you can see this `Presizing` process gives significantly better quality results  Checking and Debugging a DataBlock So now we can check out our `DataBlock` by using `show_batch`<jupyter_code>dls.show_batch(nrows=1, ncols=3)<jupyter_output><empty_output><jupyter_text>This example shows us a summary of a `DataBlock` if we didn't utilise our `item_tfms` and `batch_tfms`.- Sets up transforms pipeline- Builds one sample- Collects the items- Builds one batchBefore it falls over because our images are of different dimensions. `summary` is really useful to see what has happened in building your `DataBlock` and image tranformation pipeline and gives detailed explaination of the the errors<jupyter_code>pets1 = DataBlock(blocks = (ImageBlock, CategoryBlock),
                 get_items=get_image_files, 
                 splitter=RandomSplitter(seed=42),
                 get_y=using_attr(RegexLabeller(r'(.+)_\d+.jpg$'), 'name'))
pets1.summary(path/"images")<jupyter_output><empty_output><jupyter_text>Setting-up type transforms pipelinesCollecting items from /storage/data/oxford-iiit-pet/imagesFound 7390 items2 datasets of sizes 5912,1478Setting up Pipeline: PILBase.createSetting up Pipeline: partial -> Categorize -- {'vocab': None, 'sort': True, 'add_na': False}Building one sample  Pipeline: PILBase.create    starting from      /storage/data/oxford-iiit-pet/images/english_setter_53.jpg    applying PILBase.create gives      PILImage mode=RGB size=500x400  Pipeline: partial -> Categorize -- {'vocab': None, 'sort': True, 'add_na': False}    starting from      /storage/data/oxford-iiit-pet/images/english_setter_53.jpg    applying partial gives      english_setter    applying Categorize -- {'vocab': None, 'sort': True, 'add_na': False} gives      TensorCategory(19)Final sample: (PILImage mode=RGB size=500x400, TensorCategory(19))Collecting items from /storage/data/oxford-iiit-pet/imagesFound 7390 items2 datasets of sizes 5912,1478Setting up Pipeline: PILBase.createSetting up Pipeline: partial -> Categorize -- {'vocab': None, 'sort': True, 'add_na': False}Setting up after_item: Pipeline: ToTensorSetting up before_batch: Pipeline: Setting up after_batch: Pipeline: IntToFloatTensor -- {'div': 255.0, 'div_mask': 1}Building one batchApplying item_tfms to the first sample:  Pipeline: ToTensor    starting from      (PILImage mode=RGB size=500x400, TensorCategory(19))    applying ToTensor gives      (TensorImage of size 3x400x500, TensorCategory(19))Adding the next 3 samplesNo before_batch transform to applyCollating items in a batchError! It's not possible to collate your items in a batchCould not collate the 0-th members of your tuples because got the following shapes(3, 400, 500),(3, 225, 300),(3, 225, 300),(3, 332, 500) Now lets create a `cnn_learner` pass in our `dataloader` along with the `resnet34` architecture and train the last 2 layers of the network.<jupyter_code>learn = cnn_learner(dls, resnet34, metrics=error_rate)
learn.fine_tune(2)<jupyter_output><empty_output><jupyter_text>But we didn't specify a loss function, thankfully fastai knows which loss fucntions are useful for each type of learner. So what did it pick?<jupyter_code>learn.loss_func<jupyter_output><empty_output><jupyter_text>So what is Cross-Entropy Loss?  Cross-Entropy Loss  Viewing Activations and Labels A sensible place to start is taking a look at our data, here we can destructure `one_batch` of from our data loader and see our independent variable `y`<jupyter_code>x,y = dls.one_batch()
y<jupyter_output><empty_output><jupyter_text>Here we can see all of our classifications, each index value represents one of the breeds within our dataset. We can see what that index value pertains to by using the `vocab` function, but it isn't entirely relevant in this example. Let's now make a prediction using `get_preds`<jupyter_code>preds,_ = learn.get_preds(dl=[(x,y)])
preds[0]<jupyter_output><empty_output><jupyter_text>Here we can see that the highest value prediction is `9.3534e-01` and it is significantly higher than any of the other predictions. There is also another attribute of these predictions<jupyter_code>len(preds[0]),preds[0].sum()<jupyter_output><empty_output><jupyter_text>They all sum to 1.  Softmax Lets start with a simple example from Chapter 4, lets draw a graph of the `sigmoid` function<jupyter_code>plot_function(torch.sigmoid, min=-4,max=4)
#hide
torch.random.manual_seed(42);<jupyter_output><empty_output><jupyter_text>If we remember correctly by passing in extremely high numbers the result with tend towards 1, and extremely low numbers will tend towards 0. This will aid us in normalizing our predicted results. If we create some simple example data that could be representative of the predictions of whether an input to the network is either a 3 or a 7 we might get a tensor similar to our data below<jupyter_code>acts = torch.randn((6,2))*2
acts<jupyter_output><empty_output><jupyter_text>If we pass these values through our `sigmoid` function we get two new values, but they aren't particularly helpful in giving us a concrete predicition on whether or not our inputs are 3s or 7s, so how can we remedy this?<jupyter_code>acts.sigmoid()<jupyter_output><empty_output><jupyter_text>Maybe instead of looking at the values independently we want to pass the difference of the predictions to the sigmoid fucntion and have `1 - the sigmoid of the difference` being the resultant prediction<jupyter_code>(acts[:,0]-acts[:,1]).sigmoid()<jupyter_output><empty_output><jupyter_text>The additional benefit is that the results are guaranteed to sum to 1 effectively giving us a percentage. - **60.25%** chance of being a **3** vs **39.75%** chance of being a **7**<jupyter_code>sm_acts = torch.softmax(acts, dim=1)
sm_acts<jupyter_output><empty_output><jupyter_text>Note: The softmax function looks like this:<jupyter_code>def softmax(x): return exp(x) / exp(x).sum(dim=1, keepdim=True)<jupyter_output><empty_output><jupyter_text>Exponential function (exp): Literally defined as e**x, where e is a special number approximately equal to 2.718. It is the inverse of the natural logarithm function. Note that exp is always positive, and it increases very rapidly! If we have three output activations, such as in our droid classifier, calculating softmax for a single droid image would then look like something like this: ![](2020-12-17-Put-Down-Your-Weapon/Comparison.jpg)  Log Likelihood So now that we have our predictions how do calculate our loss? If we remeber back in Chapter 4, we created our `mnist_loss` function<jupyter_code>def mnist_loss(inputs, targets):
    inputs = inputs.sigmoid()
    return torch.where(targets==1, 1-inputs, inputs).mean()<jupyter_output><empty_output><jupyter_text>Here we determined that if the target was 1 we would return 1 - our input else the input value, this would tell us how far our prediction was from the actual target.  But this will only work in the binary case, what if we would like to solve for more classes? We can simply index into our softmax activations! Sounds to good to be true. Lets take a look at our toy example above<jupyter_code>targ = tensor([0,1,0,1,1,0])<jupyter_output><empty_output><jupyter_text>Here we have defined our target indices in a tensor<jupyter_code>sm_acts<jupyter_output><empty_output><jupyter_text>Our softmax activations are the same as they are above But by looking into each row and retrieving only the column indexed in that row we can determine our loss<jupyter_code>idx = range(6)
sm_acts[idx, targ]
from IPython.display import HTML
df = pd.DataFrame(sm_acts, columns=["3","7"])
df['targ'] = targ
df['idx'] = idx
df['loss'] = sm_acts[range(6), targ]
t = df.style.hide_index()
#To have html code compatible with our script
html = t._repr_html_().split('</style>')[1]
html = re.sub(r'<table id="([^"]+)"\s*>', r'<table >', html)
display(HTML(html))<jupyter_output><empty_output><jupyter_text>The nice thing about this method is that it scales to any number of catagories. If we were doing the full MNIST classification, identifying all ten digits, we would be indexing into a column of ten values instead of the two in this example<jupyter_code>-sm_acts[idx, targ]
F.nll_loss(sm_acts, targ, reduction='none')<jupyter_output><empty_output><jupyter_text>By making our softmax activations negative, we are getting the identical results as the PyTorch function `nll_loss` or `Negative Log Likelihood`, here we can see that the function provided by PyTorch and fastai are simple shortcuts to work we can do ourselves warning: Confusing Name, Beware: The nll in nll_loss stands for "negative log likelihood," but it doesn't actually take the log at all! It assumes you have already taken the log. PyTorch has a function called log_softmax that combines log and softmax in a fast and accurate way. nll_loss is designed to be used after log_softmax.  Taking the Log y = b**aa = log(y,b)In this case, we're assuming that log(y,b) returns log y base b. However, PyTorch actually doesn't define log this way: log in Python uses the special number e (2.718...) as the base.Perhaps a logarithm is something that you have not thought about for the last 20 years or so. But it's a mathematical idea that is going to be really critical for many things in deep learning, so now would be a great time to refresh your memory. The key thing to know about logarithms is this relationship:log(a*b) = log(a)+log(b)<jupyter_code>plot_function(torch.log, min=0,max=4)
loss_func = nn.CrossEntropyLoss()
loss_func(acts, targ)
F.cross_entropy(acts, targ)
nn.CrossEntropyLoss(reduction='none')(acts, targ)<jupyter_output><empty_output><jupyter_text>An interesting feature about cross-entropy loss appears when we consider its gradient. The gradient of cross_entropy(a,b) is just softmax(a)-b. Since softmax(a) is just the final activation of the model, that means that the gradient is proportional to the difference between the prediction and the target. This is the same as mean squared error in regression (assuming there's no final activation function such as that added by y_range), since the gradient of (a-b)**2 is 2*(a-b). Because the gradient is linear, that means we won't see sudden jumps or exponential increases in gradients, which should lead to smoother training of models.  Model Interpretation<jupyter_code>interp = ClassificationInterpretation.from_learner(learn)
interp.plot_confusion_matrix(figsize=(12,12), dpi=60)
interp.most_confused(min_val=5)<jupyter_output><empty_output><jupyter_text>Improving Our Model  The Learning Rate Finder<jupyter_code>learn = cnn_learner(dls, resnet34, metrics=error_rate)
learn.fine_tune(1, base_lr=0.1)
learn = cnn_learner(dls, resnet34, metrics=error_rate)
lr_min,lr_steep = learn.lr_find()
print(f"Minimum/10: {lr_min:.2e}, steepest point: {lr_steep:.2e}")
learn = cnn_learner(dls, resnet34, metrics=error_rate)
learn.fine_tune(2, base_lr=3e-3)<jupyter_output><empty_output><jupyter_text>Unfreezing and Transfer Learning<jupyter_code>learn.fine_tune??<jupyter_output><empty_output><jupyter_text>Signature:learn.fine_tune(    epochs,    base_lr=0.002,    freeze_epochs=1,    lr_mult=100,    pct_start=0.3,    div=5.0,    lr_max=None,    div_final=100000.0,    wd=None,    moms=None,    cbs=None,    reset_opt=False,)Source:   @patch@delegates(Learner.fit_one_cycle)def fine_tune(self:Learner, epochs, base_lr=2e-3, freeze_epochs=1, lr_mult=100,              pct_start=0.3, div=5.0, **kwargs):    "Fine tune with `freeze` for `freeze_epochs` then with `unfreeze` from `epochs` using discriminative LR"    self.freeze()    self.fit_one_cycle(freeze_epochs, slice(base_lr), pct_start=0.99, **kwargs)    base_lr /= 2    self.unfreeze()    self.fit_one_cycle(epochs, slice(base_lr/lr_mult, base_lr), pct_start=pct_start, div=div, **kwargs)File:      /opt/conda/envs/fastai/lib/python3.8/site-packages/fastai/callback/schedule.pyType:      method<jupyter_code>learn = cnn_learner(dls, resnet34, metrics=error_rate)
learn.fit_one_cycle(3, 3e-3)
learn.unfreeze()
learn.lr_find()
learn.fit_one_cycle(6, lr_max=1e-5)<jupyter_output><empty_output><jupyter_text>Discriminative Learning Rates<jupyter_code>learn = cnn_learner(dls, resnet34, metrics=error_rate)
learn.fit_one_cycle(3, 3e-3)
learn.unfreeze()
learn.fit_one_cycle(12, lr_max=slice(1e-6,1e-4))
learn.recorder.plot_loss()<jupyter_output><empty_output><jupyter_text>Selecting the Number of Epochs  Deeper Architectures<jupyter_code>from fastai.callback.fp16 import *
learn = cnn_learner(dls, resnet50, metrics=error_rate).to_fp16()
learn.fine_tune(6, freeze_epochs=3)<jupyter_output>Downloading: "https://download.pytorch.org/models/resnet50-19c8e357.pth" to /root/.cache/torch/hub/checkpoints/resnet50-19c8e357.pth
{'fraction_non_alphanumeric': 0.0895250961238699, 'fraction_numerical': 0.020575704042398422, 'mean_word_length': 5.471755211835911, 'pattern_counts': {'":': 0, '<': 135, '<?xml version=': 0, '>': 139, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '33', 'dir': 'jupyter-structured-clean-dedup', 'id': '563852', 'license': 'Apache-2.0', 'n_tokens_mistral': 6227, 'n_tokens_neox': 5905, 'n_words': 2182, 'path': '_notebooks/2020-12-17-Put-Down-Your-Weapon.ipynb', 'repo_name': 'bailey-deep-learning/im_sorry_dave', 'size': '19246'}