<jupyter_start><jupyter_text>PLEASE READ BEFORE STARTING1. **Don't edit this file, make a copy first:**  * Click on File -> Save a copy in Drive2. Also do the following:  * Click on Runtime -> Change runtime type -> Make sure hardware accelerator is set to GPU  FastAI TutorialNeural networks can be used for a whole host of tasks.The most basic of these though is image classification.Hence, today we'll go through using supervised learning to predict an images class/label from the CIFAR-10 dataset.We'll be using a library called FastAI which is built ontop of PyTorch to abstract away all the nitty-gritty details.Instead we can focus on comparing, contrarsting and understanding the different concepts we are able to use (like pretraining).By the end of this you should be roughly familiar with all the major concepts and theory behind basic neural nets!There's a lot going on here, so let's get going!  Importing LibrariesBefore we dive into the image classification task for this workshop, it is important the relevant libraries.Note here that we're importing everything from fastai's vision package, which is useful here but not in practice.<jupyter_code>!pip install fastai --upgrade --quiet
from fastai.vision.all import *<jupyter_output><empty_output><jupyter_text>Dataset SetupThe first step for anything data related is to read a dataset and split the data into seperate training and validation partitions.In this tutorial we will be using the CIFAR-10 dataset, which includes 60,000 images, each belonging to one of ten categories.The dataset was collected by <NAME>, <NAME>, and <NAME>. More information on the dataset can be found here: https://www.cs.toronto.edu/~kriz/cifar.html Initialising the Data into a pipelineTo start, we will download the CIFAR-10 dataset and extract all the images from it.<jupyter_code>path = untar_data(URLs.CIFAR) # Downloads url and unzips to folder destination<jupyter_output><empty_output><jupyter_text>Next we will initialise an instance of the 'DataBlock' class from the FastAI library. Which is a generic container that allows us to build a "smart" dataset called a dataloader, that contains the images seperated into training and validation sets, with their class attached to them. The purpose of the dataloader is that it specified a pipeline in which the model will receive data.We use the function ImageDataLoaders() to establish the dataloader by passing the path to the dataset, and the proportion of the dataset we wish to dedicate to the validation set.The validation dataset provides a way to check whether we are actually learning how to classify images or just overfitting the data (i.e. the model has just memorised which image belongs to which class).To do this we can create a validation dataset which the model doesn't train with, but insntead is used to "test" how well it does "out-of-sample".In this case we've used 20% of CIFAR-10's 60,000 images for validation (but you can change this).<jupyter_code>data = ImageDataLoaders.from_folder(path=path, valid_pct=0.2)<jupyter_output><empty_output><jupyter_text>Validating the datasetBefore we proceed, it is important to validate the data to ensure that we do not have an incomplete dataset, a dateset with incorrect preprocessing, and more importantly to understand the data we are working in before we begin training.We know that CIFAR-10 has 60,000 images, lets start by verifying that we correctly set 20% of the dataset to the validation set.<jupyter_code>print("Training Set Size = " + str(len(data.train_ds)))
print("Validiation Set Size = " + str(len(data.valid_ds)))
print("Total Dataset Size = " + str(len(data.train_ds) + len(data.valid_ds)))<jupyter_output><empty_output><jupyter_text>We can also validate the dataset by checking a random batch to view the images with their respective labels.Notice that the images are blurry, this is because the CIFAR dataset is used to train neural nets to identify far away objects that are often pixilated. The images have been taken while the camera setting was zoomed in to maximum.Hence, this image classification is a real life example of where AI can be applied.<jupyter_code>data.show_batch(figsize=(10,10))<jupyter_output><empty_output><jupyter_text>Model Setup Transfer LearningTransfer learning allows us to repurpose a model trained for one task to many others.This means our research and practical testing of image classification techniques using neural nets on CIFAR-10 can also be used for other datasets (and even other broad domains like Natural Language Processing).This is why transfer learning is one of the most fundemental aspect of deep learning.We can find several examples of this, including:- A model trained for the ImageNet competition can be repurposed to recognise between different dog breeds- A model trained on ImageNet can be repurposed to help us classify whether an image is a 'plane' or a 'dog'- A language model trained on Spanish could be adapted and repurposed for French/ItalianThe primary benefit of transfer learning is that we can use the same "toolkit" or "basic techniques" for a very wide variety of complex problems.Hence, we have solid foundations which we can build up upon.This cuts down on training time *substantially*.In this example we're using a pretrained *ResNet-34* (more advanced model) as our base architecture, then retraining it to adapt it to classify our data.<jupyter_code>learn = cnn_learner(data, resnet34, metrics=accuracy)<jupyter_output><empty_output><jupyter_text>Training Determining the Learning RateAn important thing to figgure out is what learning rate to use.It's a hard problem to solve, but we can nudge ourselves in the right direction by finding out how changing our learning rate effects the loss initially.We see this with FastAI's `lr_find()`.It provides the minimum learning rate (divided by 10) and the point of steepest descent.<jupyter_code>learn.lr_find()<jupyter_output><empty_output><jupyter_text>You can try and pick the best learning rate and put it into the training method below (in the next section) to see how it changes your results.If you try a few different values, you'll soon see that your choice greatly effects the results!  Training Time!We are now ready to begin training!To start, we establish our base learning rate which we can choose from our previous graph. For this experiment, you will need to assign the variable `base_lr` with a learning rate of your choice.After selecting your learning rate, execute the code block below before you continue reading as it may take some time to train the model.After the learning rate is defined, we freeze the lower layers by calling the `freeze()` method. This concept is taken from Transfer Learning, as we have previously spoken about, and it will allow us to 'custom fit' the *ResNet-34* to the dataset, as the network is already pretrained on a similar problem. The actual 'freezing' occurs by preventing any weights in the lower layers from being modified until we unfreeze, allowing us to change the final fully connected layers.Now we can train our fully connected layers using the `fit_one_cycle()` method. This method takes in how many epochs we want to train and the learning rate at which we want to train our network at. We have also inputted an optional argument which is slightly out of scope for this workshop.After one epoch, we unfreeze the lower layers so that ALL layers can now have their weights updated according to the loss function. We can then train for another 3 epochs and evaluate the results.<jupyter_code>base_lr = #<Replace with desired learning rate>
learn.freeze()
learn.fit_one_cycle(1, base_lr, pct_start=0.99)
base_lr /= 2
learn.unfreeze()
learn.fit_one_cycle(3, base_lr, pct_start=0.3)<jupyter_output><empty_output><jupyter_text>While it is training, you may hopefully notice that the training loss and validation loss get lower over time, and the accuracy may increase as the model learns.  Saving Model WeightsOnce training has completed, we should save our models weights (so we can use or pretrain from it later).This could also allow:- Reverting to a previous model- Tracking a models progress through special version-controlModel weights can be simply reloaded with `learner.load('some-name')`<jupyter_code>learn.save('trained-lr-default')<jupyter_output><empty_output><jupyter_text>EvaluationOnce the training has been completed, we need to gauge how good/bad our model is. Sample PredictionsWe can view some predictions with our newly trained model with the `show_results()` method. Activate the block below to see how it went!The text at the top indicates the actual class of the image, the bottom text indicates the predicted class, if they're green, our model successfully predicted correctly.<jupyter_code>learn.show_results()<jupyter_output><empty_output><jupyter_text>Viewing a sample batch of predictions is visually appealing, however there are more appropriate metrics to validate the model. Model ValidationIt is important to validate our models with appropriate metrics to determine how well the model is performing, and whether or not further investigation is required. Today we will be doing using a Confusion Matrix and viewing our top losses. We will start by creating an instance of the `ClassificationInterpretation` class from our model in order to begin.<jupyter_code>interp = ClassificationInterpretation.from_learner(learn)<jupyter_output><empty_output><jupyter_text>Confusion MatrixA Confusion Matrix can be used to determine where our model is producing false positives or false negatives. Click below to see what happened!<jupyter_code>interp.plot_confusion_matrix()<jupyter_output><empty_output><jupyter_text>Top LossesWe can also produce a set of images that fastai considers 'top losses' with the `plot_top_losses()` method. The images are considered 'top losses' based on the probability that the prediction was correct. The images with a probability of 1 technically don't have a probability of 1, it's an softmax bug within FastAI.<jupyter_code>interp.plot_top_losses(8, figsize=(15,11))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.04964889724062902, 'fraction_numerical': 0.006230837701513204, 'mean_word_length': 5.482051282051282, 'pattern_counts': {'":': 0, '<': 57, '<?xml version=': 0, '>': 60, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '482956', 'license': 'MIT', 'n_tokens_mistral': 2593, 'n_tokens_neox': 2487, 'n_words': 1406, 'path': 'Workshops/Workshop 1 - Intro to Deep Learning/FastAI Tutorial.ipynb', 'repo_name': 'traffichaos01/Training', 'size': '10133'}