<jupyter_start><jupyter_text>Reproducing the [notebook](https://github.com/muellerzr/Conference_Posters/blob/master/Talks/The%20fastai%20API%20and%20You/01_Image_Classification.ipynb) created by [muellerzr](https://github.com/muellerzr) and also collect all the different techniques into one notebook**Techniques**- Fit OneCycle with resnet34 - LearningRate(LR) Finder, Discriminative LR and Mixed Precision- MixUp during training by adding it as a callback  `learn.fine_tune(1, cbs=[MixUp()]`- Apply labelsmoothing as regularization- Mixup using callbacks- Test Time Augmentation- ranger as an optimizer, a mix of LookAhead and RAdam instead of the default AdamW**Results**Experiment | Accuracy -----------| ------resnet34 with CrossEntropy |  0.944515resnet50 with LR + Discriminative, MP | 0.93889resnet34 with LabelSmooting | resnet34 with MixUp | resnet34 with CrossEntropy + TTA |resnet34 with Ranger + Fit Flat Cosine + Discrim LR | 0.930resnet34 with CrossEntropy + NativeMP | 0.93991  Performing image classification on [ImageWoof](https://github.com/fastai/imagenette) dataset. This is a subset of ImageNet containing 10 different dog breeds.  Outline- Install & Import the library- Applying DataBlock - Create a baseline using Resnet34 model- Showing examples using techniques such as MixUp and Test Time-Augmentation  Install and Import the library With pytorch, different releases are published are based on the computation backends such as (CPU, CUDA). One has to install appropriate pytorch version based on CUDA version. light-the-torch package simplifies the problem.<jupyter_code>%%capture
!pip install light-the-torch
!ltt install torch torchvision
!pip install fastai==2.1.4 fastcore==1.3.2 nbdev --upgrade

#Ensure that you restart the runtime if it still shows the old version.
#Enable this variable if you want to run fine_tune
FINE_TUNE=False<jupyter_output><empty_output><jupyter_text>We will use vision module in the fastai library.<jupyter_code>import fastai
fastai.__version__
#export
from fastai.vision.all import *
from nbdev.showdoc import *<jupyter_output><empty_output><jupyter_text>Applying the DataBlock<jupyter_code>#Download the data
path = untar_data(URLs.IMAGEWOOF)<jupyter_output><empty_output><jupyter_text>Let's look at how this dataset is structured.<jupyter_code>path.ls()<jupyter_output><empty_output><jupyter_text>There are two directories containing training and validation files. The training directory contains a list of directories containinig images corresponding to a particular breed.<jupyter_code>(path/'train').ls()[:2]<jupyter_output><empty_output><jupyter_text>The directory names contain the labels.<jupyter_code># Defining the datablock to prepare for modeling
dblk = DataBlock(
    blocks = (ImageBlock, CategoryBlock),
    get_items = get_image_files,
    splitter = GrandparentSplitter(train_name='train', valid_name='val'),
    get_y = parent_label,
    item_tfms = [Resize(size=448, method='crop')], #data augmentation see more here https://docs.fast.ai/vision.augment.html
    batch_tfms = [Normalize.from_stats(*imagenet_stats)]
)
#Let's pass a single directory 
dblk.summary((path/'train').ls()[0])<jupyter_output>Setting-up type transforms pipelines
Collecting items from /root/.fastai/data/imagewoof2/train/n02087394
Found 942 items
2 datasets of sizes 942,0
Setting up Pipeline: PILBase.create
Setting up Pipeline: parent_label -> Categorize -- {'vocab': None, 'sort': True, 'add_na': False}

Building one sample
  Pipeline: PILBase.create
    starting from
      /root/.fastai/data/imagewoof2/train/n02087394/n02087394_46.JPEG
    applying PILBase.create gives
      PILImage mode=RGB size=500x451
  Pipeline: parent_label -> Categorize -- {'vocab': None, 'sort': True, 'add_na': False}
    starting from
      /root/.fastai/data/imagewoof2/train/n02087394/n02087394_46.JPEG
    applying parent_label gives
      n02087394
    applying Categorize -- {'vocab': None, 'sort': True, 'add_na': False} gives
      TensorCategory(0)

Final sample: (PILImage mode=RGB size=500x451, TensorCategory(0))


Setting up after_item: Pipeline: Resize -- {'size': (448, 448), 'method': 'crop', 'pad_mode': 'reflection', 'resam[...]<jupyter_text>From the above, we can observe it sets a pipeline for X and y. And to perform a transformation on y it first applies `parent_label`(get the name of directory containing this file) followed by `Categorize` and converting it to TensorCategory.``` Pipeline: parent_label -> Categorize -- {'vocab': None, 'sort': True, 'add_na': False}        starting from      /root/.fastai/data/imagewoof2/train/n02096294/n02096294_1539.JPEG    applying parent_label gives      n02096294    applying Categorize -- {'vocab': None, 'sort': True, 'add_na': False} gives      TensorCategory(0)``` fastai provides **Transform** pipeline construct that allows not only to augment the data but also allows one to change the folder name into a more readable one.<jupyter_code>label_dict = dict(
    n02086240= 'Shih-Tzu',
    n02087394= 'Rhodesian ridgeback',
    n02088364= 'Beagle',
    n02089973= 'English foxhound',
    n02093754= 'Australian terrier',
    n02096294= 'Border terrier',
    n02099601= 'Golden retriever',
    n02105641= 'Old English sheepdog',
    n02111889= 'Samoyed',
    n02115641= 'Dingo'
)
label_dict
item = (path/'train').ls()[0]
item.name, label_dict[item.name], label_dict.__getitem__(item.name)<jupyter_output><empty_output><jupyter_text>Adding this as part of transformation to the target using get_y<jupyter_code># Pipeline is composed of functions/transforms for encoding/decoding purpose.
get_y = Pipeline([parent_label, label_dict.__getitem__])<jupyter_output><empty_output><jupyter_text>Let's rewrite the datablock and run the summary if it transform the y as we expected.<jupyter_code>dblk = DataBlock(
    blocks = (ImageBlock, CategoryBlock),
    get_items = get_image_files,
    splitter = GrandparentSplitter(train_name='train', valid_name='val'),
    get_y = get_y,
    item_tfms = [Resize(size=448, method='crop')], #data augmentation see more here https://docs.fast.ai/vision.augment.html
    batch_tfms = [Normalize.from_stats(*imagenet_stats)]
)
dblk.summary((path/'train').ls()[0])<jupyter_output>Setting-up type transforms pipelines
Collecting items from /root/.fastai/data/imagewoof2/train/n02087394
Found 942 items
2 datasets of sizes 942,0
Setting up Pipeline: PILBase.create
Setting up Pipeline: parent_label -> dict.__getitem__ -> Categorize -- {'vocab': None, 'sort': True, 'add_na': False}

Building one sample
  Pipeline: PILBase.create
    starting from
      /root/.fastai/data/imagewoof2/train/n02087394/n02087394_46.JPEG
    applying PILBase.create gives
      PILImage mode=RGB size=500x451
  Pipeline: parent_label -> dict.__getitem__ -> Categorize -- {'vocab': None, 'sort': True, 'add_na': False}
    starting from
      /root/.fastai/data/imagewoof2/train/n02087394/n02087394_46.JPEG
    applying parent_label gives
      n02087394
    applying dict.__getitem__ gives
      Rhodesian ridgeback
    applying Categorize -- {'vocab': None, 'sort': True, 'add_na': False} gives
      TensorCategory(0)

Final sample: (PILImage mode=RGB size=500x451, TensorCategory(0))


Setting up a[...]<jupyter_text>As we can see, we have introduced an additional transform and converted the y from n02096294 to Border terrier```Pipeline: parent_label -> dict.__getitem__ -> Categorize -- {'vocab': None, 'sort': True, 'add_na': False}    starting from      /root/.fastai/data/imagewoof2/train/n02096294/n02096294_1539.JPEG    applying parent_label gives      n02096294    applying dict.__getitem__ gives      Border terrier    applying Categorize -- {'vocab': None, 'sort': True, 'add_na': False} gives      TensorCategory(0)```<jupyter_code>aug_transforms(size=224, min_scale=0.75)
#If you want more fine grained control,explictly define your batch tfms

# batch_tfms = [FlipItem(), RandomResizedCrop(128, min_scale=0.35),
#                   Normalize.from_stats(*imagenet_stats)
#                   ]

# Or go with the defaults
#batch_tfms = [aug_transforms(size=128, min_scale=0.75)]
# Defining the datablock to prepare for modeling
dblk = DataBlock(
    blocks = (ImageBlock, CategoryBlock),
    get_items = get_image_files,
    splitter = GrandparentSplitter(train_name='train', valid_name='val'),
    get_y = get_y,
    #Resize images to relatively “large” dimensions
    item_tfms = [Resize(size=448, method='crop')], 
    #data augmentation see more here https://docs.fast.ai/vision.augment.html
    batch_tfms = [FlipItem(), RandomResizedCrop(128, min_scale=0.35),
                  Normalize.from_stats(*imagenet_stats)
                  ]
)
dblk.summary(path)
dls = dblk.dataloaders(path, bs=64)
#dls.show_batch()
b = dls.one_batch()
len(b), b[0].shape, b[0][0].shape<jupyter_output><empty_output><jupyter_text>Baseline using Resnet34 We can leverage Transfer Learning to speed up the training and also improve the metrics.fastai provides Learner object which wraps the dataloaders, architecture, hyperparameters, metrics etc.For images, we can use cnn_learner(Convolutional Neural Network)<jupyter_code>learn = cnn_learner(dls, arch=resnet34, metrics=accuracy, loss_func=CrossEntropyLossFlat())<jupyter_output>Downloading: "https://download.pytorch.org/models/resnet34-333f7ec4.pth" to /root/.cache/torch/hub/checkpoints/resnet34-333f7ec4.pth<jupyter_text>Learner by default tries to download the pretrained weights for resnet architecture and apply custom head on top. Let's look at the model summary.<jupyter_code>learn.loss_func<jupyter_output><empty_output><jupyter_text>Experiment 1 - Baseline<jupyter_code>if FINE_TUNE:
  learn.fine_tune(1)<jupyter_output><empty_output><jupyter_text>Experiment 2 - LR Finder, Discriminative LR, Mixed Precision, Resnet 50<jupyter_code>learn = cnn_learner(dls, arch=resnet50, metrics=accuracy, loss_func=CrossEntropyLossFlat()).to_fp16()
learn.lr_find()
if FINE_TUNE:
  learn.fine_tune(3, 1e-2)
if FINE_TUNE:
  learn.lr_find()
if FINE_TUNE:
  learn.fit_one_cycle(3, slice(1e-5, 5e-4))<jupyter_output><empty_output><jupyter_text>The performance seemse to be much lower than the resnet34.  Experiment 2a - LabelSmoothingChange the loss function with label smoothing applied<jupyter_code>doc(LabelSmoothingCrossEntropyFlat)<jupyter_output><empty_output><jupyter_text>From https://rickwierenga.com/blog/machine%20learning/LabelSmoothing.html> Label smoothing is a variation on one hot encoding where the negative labels have a value slightly higher than 0, say 0.1  and the positive labels have a value slightly lower than 1, say 0.9```class LabelSmoothingCrossEntropy(Module):    y_int = True    def __init__(self, eps:float=0.1, reduction='mean'): self.eps,self.reduction = eps,reduction    def forward(self, output, target):        c = output.size()[-1]        log_preds = F.log_softmax(output, dim=-1)        if self.reduction=='sum': loss = -log_preds.sum()        else:            loss = -log_preds.sum(dim=-1) We divide by that size at the return line so sum and not mean            if self.reduction=='mean':  loss = loss.mean()        return loss*self.eps/c + (1-self.eps) * F.nll_loss(log_preds, target.long(), reduction=self.reduction)    def activation(self, out): return F.softmax(out, dim=-1)    def decodes(self, out):    return out.argmax(dim=-1)``````@delegates()class LabelSmoothingCrossEntropyFlat(BaseLoss):    "Same as `LabelSmoothingCrossEntropy`, but flattens input and target."    y_int = True    @use_kwargs_dict(keep=True, eps=0.1, reduction='mean')    def __init__(self, *args, axis=-1, **kwargs): super().__init__(LabelSmoothingCrossEntropy, *args, axis=axis, **kwargs)    def activation(self, out): return F.softmax(out, dim=-1)    def decodes(self, out):    return out.argmax(dim=-1)```<jupyter_code>learn = cnn_learner(dls, arch=resnet34, metrics=accuracy, loss_func=LabelSmoothingCrossEntropyFlat())
learn.loss_func
if FINE_TUNE:
  learn.fine_tune(1)<jupyter_output><empty_output><jupyter_text>Experiment 3 - Mixup Source: https://towardsdatascience.com/2-reasons-to-use-mixup-when-training-yor-deep-learning-models-58728f15c559Mixup is a form of regularization technique. The idea is instead of passing raw images to your model, you pass a blend of 2 images from your training set. newImage = alpha * image1 + (1-alpha) * image2But<jupyter_code>doc(MixUp)
learn = cnn_learner(dls, arch=resnet34, metrics=accuracy, loss_func=CrossEntropyLossFlat())
learn.loss_func
learn.fine_tune(1, cbs=MixUp())<jupyter_output><empty_output><jupyter_text>Experiment 4 - Test Time Augmentation  This is a type of data augmentation. By selecting a number of areas to crop from the original rectangular image, pass each of them through our model, and take the **maximum** or **average** of the predictions. - `preds, _ = learn.tta(dl=test_dl, beta=0)`<jupyter_code>show_doc(Learner.tta)
tta_preds = learn.tta(ds_idx=1)
preds = learn.get_preds(ds_idx = 1)
doc(accuracy)
len(preds), len(preds[0]), len(preds[1])
preds[0][0], preds[1][0]
accuracy(*preds)
accuracy(*tta_preds)<jupyter_output><empty_output><jupyter_text>Experiment 5 - Native Mixed Precision- Native Mixed Precision `to_native_fp16()` during training and back to `to_native_fp32` before export<jupyter_code>learn = cnn_learner(dls, arch=resnet34, metrics=accuracy, loss_func=CrossEntropyLossFlat()).to_native_fp16()
learn.lr_find()
learn.fine_tune(5, 1e-2)<jupyter_output><empty_output><jupyter_text>We may need to convert back full precision before exporting the model.<jupyter_code>learn = learn.to_native_fp32()<jupyter_output><empty_output><jupyter_text>Experiment 6 - Optimizers (LookAhead + RAdam) Source: https://www.kaggle.com/muellerzr/cassava-fastai-starter<jupyter_code>show_doc(ranger)
learn = cnn_learner(dls, arch=resnet34, metrics=accuracy, opt_func=ranger)
learn.lr_find()<jupyter_output><empty_output><jupyter_text>> Learner.fine_tune??```def fine_tune(self:Learner, epochs, base_lr=2e-3, freeze_epochs=1, lr_mult=100,              pct_start=0.3, div=5.0, **kwargs):    "Fine tune with `freeze` for `freeze_epochs` then with `unfreeze` from `epochs` using discriminative LR"    self.freeze()    self.fit_one_cycle(freeze_epochs, slice(base_lr), pct_start=0.99, **kwargs)    base_lr /= 2    self.unfreeze()    self.fit_one_cycle(epochs, slice(base_lr/lr_mult, base_lr), pct_start=pct_start, div=div, **kwargs)```<jupyter_code>show_doc(Learner.fit_flat_cos)<jupyter_output><empty_output><jupyter_text>The same can be rewritten using flat cosine annealing. @patch functionality ties it with the Learner. Read more about [patch](https://fastcore.fast.ai/basics.htmlpatch) . patch decorator adds the decorated function to the class defined in first parameter.<jupyter_code>@patch
def fine_tune_flat_cos(self: Learner, epochs, base_lr=2e-3, freeze_epochs=1, lr_mult=100, pct_start=0.3, div=5.0, **kwargs):
  '''Fine tune with `freeze` for `freeze_epochs` then with `unfreeze` from `epochs` using discriminative LR using flat cosine annealing'''
  self.freeze()
  self.fit_flat_cos(freeze_epochs, slice(base_lr), pct_start=0.99, **kwargs)
  base_lr /= 2
  self.unfreeze()
  self.fit_flat_cos(epochs, slice(base_lr/lr_mult, base_lr), pct_start=pct_start, div=div, **kwargs)
if FINE_TUNE:
  learn.fine_tune_flat_cos(5, 2e-2)
learn.validate(dl=dls.valid)<jupyter_output><empty_output><jupyter_text>Inference<jupyter_code>item = get_image_files(path)[0]
learn.predict(item)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11129804581338165, 'fraction_numerical': 0.0355894913938139, 'mean_word_length': 5.482802013422819, 'pattern_counts': {'":': 0, '<': 94, '<?xml version=': 0, '>': 105, 'https://': 11, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '24', 'dir': 'jupyter-structured-clean-dedup', 'id': '190108', 'license': 'Apache-2.0', 'n_tokens_mistral': 5573, 'n_tokens_neox': 5189, 'n_words': 1304, 'path': '02_imagewoof.ipynb', 'repo_name': 'manisnesan/fastchai', 'size': '15454'}