<jupyter_start><jupyter_text>Chapter 5 - Image Classification> Deep Learning For Coders with fastai & Pytorch - Image Classification, In this notebook I followed both <NAME>'s Lesson on fast.ai and Weigh and Biases reading group videos. Lots of notes added, some cell's order changed some are added to make the topic more understandable for me. (Check Manual calculation `log_softmax` + `nll_loss`).  Click `open in colab` button at the right side to view as notebook.- toc: true - badges: true- comments: true- categories: [fastbook]- image: images/cyberman.png *** ![](images/chapter-05/cyberman.png) >  I'm a Doctor Who fan and this is my cyberman coffee cup, as I remember got it from Manchester Science Museum.***<jupyter_code>#!pip install -Uqq fastbook
import fastbook
fastbook.setup_book()
%config Completer.use_jedi = False
from fastbook import *<jupyter_output><empty_output><jupyter_text>[[chapter_pet_breeds]]  PLAYING WITH THE DATASET<jupyter_code>from fastai.vision.all import *
path = untar_data(URLs.PETS)<jupyter_output><empty_output><jupyter_text>> Note: __With `untar` we download the data. This data originally come from Oxford University [Visual Geomety Group](https://www.robots.ox.ac.uk/~vgg/data/) and our dataset is [here:](https://www.robots.ox.ac.uk/~vgg/data/pets/)__<jupyter_code>path<jupyter_output><empty_output><jupyter_text>> Note: __This is the local download path for my computer.__<jupyter_code>Path.BASE_PATH = path<jupyter_output><empty_output><jupyter_text>> Tip: **This is a trick to get the relative path, check above and below**<jupyter_code>path<jupyter_output><empty_output><jupyter_text>Now the `path` is looks different.<jupyter_code>path.ls()<jupyter_output><empty_output><jupyter_text>> Note: __`2` is number of item in the list. `annotations` represents target variables of this datasets but we do not use them at this time instead we create our own labels.__<jupyter_code>(path/"images").ls()
fname = (path/"images").ls()[0]
fname<jupyter_output><empty_output><jupyter_text>> Note: __The first image in the `path` list.__<jupyter_code>re.findall(r'(.+)_\d+.jpg$', fname.name)<jupyter_output><empty_output><jupyter_text>> Note: __Since we don't use the annonations in the Dataset we need to find a way to get breeds form the filename. This is regex `findall` method, Check `geeksforgeeks.org` tutorial [here](https://www.geeksforgeeks.org/python-regex-re-search-vs-re-findall/)__<jupyter_code>pets = DataBlock(blocks = (ImageBlock, CategoryBlock),
                 get_items=get_image_files, 
                 splitter=RandomSplitter(seed=42),
                 get_y=using_attr(RegexLabeller(r'(.+)_\d+.jpg$'), 'name'),
                 item_tfms=Resize(460),
                 batch_tfms=aug_transforms(size=224, min_scale=0.75))
dls = pets.dataloaders(path/"images")<jupyter_output>/home/niyazi/anaconda3/envs/fastbook/lib/python3.8/site-packages/torch/_tensor.py:1023: UserWarning: torch.solve is deprecated in favor of torch.linalg.solveand will be removed in a future PyTorch release.
torch.linalg.solve has its arguments reversed and does not return the LU factorization.
To get the LU factorization see torch.lu, which can be used with torch.lu_solve or torch.lu_unpack.
X = torch.solve(B, A).solution
should be replaced with
X = torch.linalg.solve(A, B) (Triggered internally at  /opt/conda/conda-bld/pytorch_1623448278899/work/aten/src/ATen/native/BatchLinearAlgebra.cpp:760.)
  ret = func(*args, **kwargs)<jupyter_text>> Note: __now find all names with RegexLabeller. The `item_tmsf` and `batch_transfdrms` may look a bit meaningless. Check below to find out why.__ *** PRESIZING As a summary FastAi gives a chance to augment our images in a smarter way (`presizing`) such that provide much more detail and information for the training. First, we presize images with `item_tfms` then push them to GPU and use augmentation.[check the original document for the whole idea](https://colab.research.google.com/github/fastai/fastbook/blob/master/05_pet_breeds.ipynb)<jupyter_code>#id interpolations
#caption A comparison of fastai's data augmentation strategy (left) and the traditional approach (right).
dblock1 = DataBlock(blocks=(ImageBlock(), CategoryBlock()),
                   get_y=parent_label,
                   item_tfms=Resize(460))
# Place an image in the 'images/grizzly.jpg' subfolder where this notebook is located before running this
dls1 = dblock1.dataloaders([(Path.cwd()/'images'/'chapter-05'/'grizzly.jpg')]*100, bs=8)
dls1.train.get_idxs = lambda: Inf.ones
x,y = dls1.valid.one_batch()
_,axs = subplots(1, 2)

x1 = TensorImage(x.clone())
x1 = x1.affine_coord(sz=224)
x1 = x1.rotate(draw=30, p=1.)
x1 = x1.zoom(draw=1.2, p=1.)
x1 = x1.warp(draw_x=-0.2, draw_y=0.2, p=1.)

tfms = setup_aug_tfms([Rotate(draw=30, p=1, size=224), Zoom(draw=1.2, p=1., size=224),
                       Warp(draw_x=-0.2, draw_y=0.2, p=1., size=224)])
x = Pipeline(tfms)(x)
#x.affine_coord(coord_tfm=coord_tfm, sz=size, mode=mode, pad_mode=pad_mode)
TensorImage(x[0]).show(ctx=axs[0])
TensorImage(x1[0]).show(ctx=axs[1]);
dls.show_batch(nrows=3, ncols=3)
pets1 = DataBlock(blocks = (ImageBlock, CategoryBlock),
                 get_items=get_image_files, 
                 splitter=RandomSplitter(seed=42),
                 get_y=using_attr(RegexLabeller(r'(.+)_\d+.jpg$'), 'name'))
pets1.summary(path/"images")<jupyter_output>Setting-up type transforms pipelines
Collecting items from /home/niyazi/.fastai/data/oxford-iiit-pet/images
Found 7390 items
2 datasets of sizes 5912,1478
Setting up Pipeline: PILBase.create
Setting up Pipeline: partial -> Categorize -- {'vocab': None, 'sort': True, 'add_na': False}

Building one sample
  Pipeline: PILBase.create
    starting from
      /home/niyazi/.fastai/data/oxford-iiit-pet/images/British_Shorthair_110.jpg
    applying PILBase.create gives
      PILImage mode=RGB size=500x333
  Pipeline: partial -> Categorize -- {'vocab': None, 'sort': True, 'add_na': False}
    starting from
      /home/niyazi/.fastai/data/oxford-iiit-pet/images/British_Shorthair_110.jpg
    applying partial gives
      British_Shorthair
    applying Categorize -- {'vocab': None, 'sort': True, 'add_na': False} gives
      TensorCategory(4)

Final sample: (PILImage mode=RGB size=500x333, TensorCategory(4))


Collecting items from /home/niyazi/.fastai/data/oxford-iiit-pet/images
Found 7390 items
2 d[...]<jupyter_text>> Note: __It is alway good to get a quick summary. `pets1.summary(path/"images")` Check the summary above, it has lots of details. It is natural to get an error in this example because we are trying the put diffent size images into the same `DataBlock`.__ *** BASELINE MODEL For every project, just start with a Baseline. Baseline is a good point to think about the project/domain/problem at the same time, then start improve and make experiments about architecture, hyperparameters etc.<jupyter_code>learn = cnn_learner(dls, resnet34, metrics=error_rate)
learn.fine_tune(2)<jupyter_output>/home/niyazi/anaconda3/envs/fastbook/lib/python3.8/site-packages/torch/nn/functional.py:718: UserWarning: Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (Triggered internally at  /opt/conda/conda-bld/pytorch_1623448278899/work/c10/core/TensorImpl.h:1156.)
  return torch.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode)<jupyter_text>> Note: __A basic run is helpful as baseline for the beginning.__  Defaults for the baseline<jupyter_code>learn.loss_func
learn.lr<jupyter_output><empty_output><jupyter_text>> Tip: __Very easy to see default arguments for the learner. Above loss function `loss_func` and learning rate `lr`.__  One Batch Run<jupyter_code>first(dls.train)<jupyter_output><empty_output><jupyter_text>> Note: above and below is same<jupyter_code>x,y = dls.one_batch()<jupyter_output><empty_output><jupyter_text>*** Understanding Labels<jupyter_code>dls.vocab
dls.vocab[0]<jupyter_output><empty_output><jupyter_text>> Tip: __`vocab` gives as all labels as text.__ *** What's inside the tensors?<jupyter_code>y<jupyter_output><empty_output><jupyter_text>> Note: __Targets as coded.__<jupyter_code>x<jupyter_output><empty_output><jupyter_text>> Note: __Our stacked image tensor.__ *** Predictions of the baseline model.<jupyter_code>preds,_ = learn.get_preds(dl=[(x,y)])
preds[0]<jupyter_output><empty_output><jupyter_text>> Note: __result for first item that adds up to one. There are 37 outputs for 37 image categories and the results are in percentage for probability of each category.__<jupyter_code>_<jupyter_output><empty_output><jupyter_text>> Note: __Category codes__<jupyter_code>len(preds[0]),preds[0].sum()<jupyter_output><empty_output><jupyter_text>Prediction for 37 categories that adds up to one. *** FUNCTION FOR CLASSIFIYING MORE THAN TWO CATEGORY For classifiying more than two category, we need to employ a new function. It is not totally different than sigmoid, in fact it starts with a sigmoid function.<jupyter_code>plot_function(torch.sigmoid, min=-4,max=4)<jupyter_output>/home/niyazi/anaconda3/envs/fastbook/lib/python3.8/site-packages/fastbook/__init__.py:73: UserWarning: Not providing a value for linspace's steps is deprecated and will throw a runtime error in a future release. This warning will appear only once per process. (Triggered internally at  /opt/conda/conda-bld/pytorch_1623448278899/work/aten/src/ATen/native/RangeFactories.cpp:25.)
  x = torch.linspace(min,max)<jupyter_text>> Note: __This is how `torch.sigmoid` squishes values between 0 and 1.__<jupyter_code>torch.random.manual_seed(42);
acts = torch.randn((6,2))*2
acts<jupyter_output><empty_output><jupyter_text>> Note: __These are random numbers represent binary results of a hypothetical network. First colums represent 3's the and second is 7's standart deviation of 2. It generally shows how confident the model about the predictions.__<jupyter_code>acts.sigmoid()<jupyter_output><empty_output><jupyter_text>> Note: __If we apply the sigmoid, the result become like this(above). Obviously they aren't adds up to one. These are relative confidence over inputs. For example first row says: it's a three. But what is the probability? It is not clear.__<jupyter_code>(acts[:,0]-acts[:,1]).sigmoid()<jupyter_output><empty_output><jupyter_text>> Note: __If we take the difference between these relative confidence the results become like this above: Now we can say that for the first item, model is 0.6025 (%60.25) confident.__ this part is a bit different in the lesson video. so check the video. [1:35:20](https://youtu.be/p50s63nPq9I?t=5721)<jupyter_code>sm_acts = torch.softmax(acts, dim=1)
sm_acts<jupyter_output><empty_output><jupyter_text>> Note: __`torch.softmax` does that in one step. Now results for each item adds up to one and identical.__ *** __Log Likelihood__<jupyter_code>targ = tensor([0,1,0,1,1,0])<jupyter_output><empty_output><jupyter_text>this is our softmax activations:<jupyter_code>sm_acts
idx = range(6)
sm_acts[idx, targ]<jupyter_output><empty_output><jupyter_text>> Note: __Nice trick for getting confidence level for each item.__ lets see everything in a table:<jupyter_code>from IPython.display import HTML
df = pd.DataFrame(sm_acts, columns=["3","7"])
df['targ'] = targ
df['idx'] = idx
df['loss'] = sm_acts[range(6), targ]
t = df.style.hide_index()
#To have html code compatible with our script
html = t._repr_html_().split('</style>')[1]
html = re.sub(r'<table id="([^"]+)"\s*>', r'<table >', html)
display(HTML(html))<jupyter_output><empty_output><jupyter_text>> Warning: __I think the last label is wrong here. It must be the confidence instead.__<jupyter_code>-sm_acts[idx, targ]<jupyter_output><empty_output><jupyter_text>> Warning: __There is a caveat here. These are neg of our confidence level, not loss.__ Pytorch way of doing the same here:<jupyter_code>F.nll_loss(sm_acts, targ, reduction='none')<jupyter_output><empty_output><jupyter_text>> Note: __Anyway, numbers are still not right, that will be addresses in the `Taking the Log` section below. The reason is F.nll_loss (negative log likelihood loss) needs arguments such that log is already applied to make the calculation right.(loss)__ *** Taking the Log > Note: Directly from the book: > Important: __Confusing Name, Beware: The nll in `nll_loss` stands for "negative log likelihood," but it doesn't actually take the log at all! It assumes you have _already_ taken the log. PyTorch has a function called `log_softmax` that combines `log` and `softmax` in a fast and accurate way. `nll_loss` is designed to be used after `log_softmax`.__ When we first take the softmax, and then the log likelihood of that, that combination is called *cross-entropy loss*. In PyTorch, this is available as `nn.CrossEntropyLoss` (which, in practice, actually does `log_softmax` and then `nll_loss`): pytorch's crossEntropy:<jupyter_code>loss_func = nn.CrossEntropyLoss()
loss_func(acts, targ)<jupyter_output><empty_output><jupyter_text>or:<jupyter_code>F.cross_entropy(acts, targ)<jupyter_output><empty_output><jupyter_text>> Note: this is the mean of all losses and this is all results without taking the mean:<jupyter_code>nn.CrossEntropyLoss(reduction='none')(acts, targ)<jupyter_output><empty_output><jupyter_text>> Note:Results above are cross entrophy loss for each image in the list (of course our current numbers are fake numbers) *** Manual calculation `log_softmax` + `nll_loss` First log_softmax:<jupyter_code>log_sm_acts = torch.log_softmax(acts, dim=1)
log_sm_acts<jupyter_output><empty_output><jupyter_text>Then negative log likelihood:<jupyter_code>F.nll_loss(log_sm_acts, targ, reduction='none')<jupyter_output><empty_output><jupyter_text>>Note: __Results are identical__ *** REVISITING THE BASELINE MODEL (Model Interpretation)<jupyter_code>#width 600
interp = ClassificationInterpretation.from_learner(learn)
interp.plot_confusion_matrix(figsize=(12,12), dpi=60)
interp.most_confused(min_val=5)<jupyter_output><empty_output><jupyter_text>this is our baseline we can start improveing from this point. *** IMPROVING THE MODEL  Fine Tune Fine tune the model with default arguments:<jupyter_code>learn = cnn_learner(dls, resnet34, metrics=error_rate)
learn.fine_tune(1, base_lr=0.1)<jupyter_output><empty_output><jupyter_text>> Note: __This is where we overshot. Our loss just increase over second epoch is there a better way to find a learning rate?__ *** Learning Rate Finder<jupyter_code>learn = cnn_learner(dls, resnet34, metrics=error_rate)
suggested_lr= learn.lr_find()<jupyter_output><empty_output><jupyter_text>> Warning: There is a discrepancy between lesson and reading group notebooks. In the book we get two values from the function but in reading group, only one. I thing there was an update for this function that not reflected in the book.<jupyter_code>suggested_lr
print(f"suggested: {suggested_lr.valley:.2e}")
learn = cnn_learner(dls, resnet34, metrics=error_rate)
learn.fine_tune(2, base_lr=8.32e-04)<jupyter_output><empty_output><jupyter_text>At this time it decreases steadily  __What's under the hood of `fine_tune`__ When we create a model from a pretrained network fastai automatically freezes all of the pretrained layers for us. When we call the `fine_tune` method fastai does two things:- Trains the randomly added layers for one epoch, with all other layers frozen- Unfreezes all of the layers, and trains them all for the number of epochs requested __Lets do it manually__<jupyter_code>learn = cnn_learner(dls, resnet34, metrics=error_rate)
learn.fit_one_cycle(3, 8.32e-04)
learn.unfreeze()<jupyter_output><empty_output><jupyter_text>Run the `lr_find` again, because having more layers to train, and weights that have already been trained for three epochs, means our previously found learning rate isn't appropriate any more:<jupyter_code>learn.lr_find()<jupyter_output><empty_output><jupyter_text>Train again with the new lr.<jupyter_code>learn.fit_one_cycle(6, lr_max=0.0001)<jupyter_output><empty_output><jupyter_text>So far so good but there is more way to go *** Discriminative Learning Rates Basically we use variable learning rate for the model. Bigger rate for the later layers and smaller for early layers.<jupyter_code>learn = cnn_learner(dls, resnet34, metrics=error_rate)
learn.fit_one_cycle(3, 8.32e-04)# first lr
learn.unfreeze()
learn.fit_one_cycle(12, lr_max=slice(0.00005,0.0005))#second lr with a range<jupyter_output><empty_output><jupyter_text>It is better most of the times.(sometimes I don't get good results, need to arrange the `slice` values more carefully)<jupyter_code>learn.recorder.plot_loss()<jupyter_output><empty_output><jupyter_text>> Note: Directly from the book: As you can see, the training loss keeps getting better and better. But notice that eventually the validation loss improvement slows, and sometimes even gets worse! This is the point at which the model is starting to over fit. In particular, the model is becoming overconfident of its predictions. But this does not mean that it is getting less accurate, necessarily. Take a look at the table of training results per epoch, and you will often see that the accuracy continues improving, even as the validation loss gets worse. In the end what matters is your accuracy, or more generally your chosen metrics, not the loss. The loss is just the function we've given the computer to help us to optimize. > Important: I need to think about it how loss increase and accuracy stil becoming better.  Deeper Architectures In general, a bigger model has the ability to better capture the real underlying relationships in your data, and also to capture and memorize the specific details of your individual images.However, using a deeper model is going to require more GPU RAM, so you may need to lower the size of your batches to avoid an *out-of-memory error*. This happens when you try to fit too much inside your GPU and looks like:```Cuda runtime error: out of memory```You may have to restart your notebook when this happens. The way to solve it is to use a smaller batch size, which means passing smaller groups of images at any given time through your model. You can pass the batch size you want to the call creating your `DataLoaders` with `bs=`.The other downside of deeper architectures is that they take quite a bit longer to train. One technique that can speed things up a lot is *mixed-precision training*. This refers to using less-precise numbers (*half-precision floating point*, also called *fp16*) where possible during training. As we are writing these words in early 2020, nearly all current NVIDIA GPUs support a special feature called *tensor cores* that can dramatically speed up neural network training, by 2-3x. They also require a lot less GPU memory. To enable this feature in fastai, just add `to_fp16()` after your `Learner` creation (you also need to import the module).You can't really know ahead of time what the best architecture for your particular problem is—you need to try training some. So let's try a ResNet-50 now with mixed precision:<jupyter_code>from fastai.callback.fp16 import *
learn = cnn_learner(dls, resnet50, metrics=error_rate).to_fp16()
learn.fine_tune(12, freeze_epochs=3)
learn.recorder.plot_loss()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09795707266614948, 'fraction_numerical': 0.017636410654253944, 'mean_word_length': 5.794096978215038, 'pattern_counts': {'":': 0, '<': 181, '<?xml version=': 0, '>': 219, 'https://': 5, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '45', 'dir': 'jupyter-structured-clean-dedup', 'id': '28627', 'license': 'Apache-2.0', 'n_tokens_mistral': 6431, 'n_tokens_neox': 6128, 'n_words': 2034, 'path': '_notebooks/2021-07-22-chapter-5.ipynb', 'repo_name': 'niyazikemer/blog', 'size': '19342'}