<jupyter_start><jupyter_text>Fashion-MNIST Classifier4 fundamental steps:1. Prepare data2. Build the model3. Train the model4. Analyze the model's resultWe start off with the first method of preparing the data using concept of ETL (Extract-Transform-Load)<jupyter_code># import relevant libraries
import torch   # top level package
import torchvision  # provides access to popular datasets and transformations
import torchvision.transforms as transforms   # interface which gives access to common image transformations
from torch.utils.data import DataLoader, Dataset, TensorDataset
import matplotlib.pyplot as plt
import numpy as np
import torch.nn as nn
import cv2
torch.set_printoptions(linewidth=120)   # sets the line width for pytorch console<jupyter_output><empty_output><jupyter_text>1. extract: Get the Fashion-MNIST image data2. transform: Transform the data into tensor form3. load: Put our data into object form which is easily accessibleFor this PyTorch provides 2 classes, **Dataset** class and **Dataloader**. Dataset and Dataloaders with an example of custom dataset**Dataset** is an abstract class for representing a dataset. An **abstract** class is a Python class that has methods which we must implement, so that we can create a custom dataset by creating a subclass.<jupyter_code>class subclass(Dataset):
    def __init__(self, data):
        self.data = data
        
    def __getitem__(self, index):
        sample = self.data[:,:-2]
        label = self.data[:,-1]
        return sample, label
    
    def __len__(self):
        return len(self.data)<jupyter_output><empty_output><jupyter_text>An abstract class named subclass is implemented by inheriting the abstract class Dataset. The abstract class has __init__(), __getitem__() and __len__() functions.These functions can be customized to load our custom dataset.   Dataset transformationtorchvision.datasets contains functions to create and transform a custom dataset or any of the inbuilt torch provided datasets.<jupyter_code>train_set = torchvision.datasets.FashionMNIST(root="/home/nikunjlad/data/FashionMNIST",
                                             train=True, download=True,
                                             transform=transforms.Compose([transforms.ToTensor()]))<jupyter_output><empty_output><jupyter_text>Let's explore the arguments:1. root: the directory on disk to save the data2. train: it tells us we want the training data3. download: it tells the class to download the data4. transform: we pass a composition of transformations. since we want tensor format, we have ToTensor() transformation<jupyter_code>train_set<jupyter_output><empty_output><jupyter_text>The train_set variable as seen above contains information on the dataset which we just extracted and transformed. I has various parameters and information into it which helps us gain more knowledge about this variable. Let's explore the various functionality provided by this dataset variable. You can access these functionality or additional metadata using **.(dot)** after the variable name as below:<jupyter_code># this maps class labels to integer indices
print(train_set.class_to_idx)
print("---------------------")
print(train_set.classes)   # print all the classes present in this dataset
# contains the actual training data, we can also get shape as below.
# dont use test_data and train_data. instead use data. see warnings
train_set.data
print(train_set.train_data.shape)
print(train_set.test_data.shape)
print(train_set.data.shape)
# gives download information of the dataset like number of datapoints, location of the data on disk, transformations applied, etc
print(train_set.download)
print("-----------------")
print(train_set.extra_repr)
# location of the folder where processed amd raw images are stored in .pt format (pytorch format)
print(train_set.processed_folder)
print(train_set.raw_folder)
# gives the URLs from where the resource are fetched from. 
train_set.resources
# root data directory where both the processed and raw data lie
train_set.root
# target corresponding to 60000 images in the dataset
# dont use test_labels and train_labels. instead use targets. see warnings
print(train_set.targets)
print(train_set.train_labels)
print(train_set.test_labels)
# name of the test file and training file
print(train_set.test_file)
print(train_set.training_file)
# is this training data?
train_set.train
# which kind of transformations did we apply on the data
print(train_set.transforms)
# both results are kind of similar
print(train_set.transform)<jupyter_output>StandardTransform
Transform: Compose(
               ToTensor()
           )
Compose(
    ToTensor()
)<jupyter_text>Dataloader class and its propertiesDataloader is used to load the extracted and transformed data generated by Dataset class. This creates a dataloader object which is a form suitable for training, since data is batched, sampled, shuffled, etc here<jupyter_code>train_loader = DataLoader(train_set)
train_loader<jupyter_output><empty_output><jupyter_text>As we can see, once the dataset is loaded, we get a pytorch dataloader object. This object too has many options which gives various information about the dataloader object. Let's, explore them all<jupyter_code>print(train_loader.batch_sampler.batch_size)  # returns default batch_size
print(train_loader.batch_size)   # use this preferably
# this is used when loading map-style of datasets. (need to explore iterable stype vs map style)
train_loader.collate_fn   
# information of the source from where data generated
print(train_loader.dataset)
print("--------------------")
print(train_loader.batch_sampler.sampler.data_source)   
# drops the last incomplete batch of data. If say 100 data and batch size is 30, the last batch of 10 is dropped if True
print(train_loader.batch_sampler.drop_last)  # set to false, i.e no dropping 
print(train_loader.drop_last)  # use this preferably
# number of parallel processes to spawn for loading data. Usually configured to be equal to cores present on a machine
train_loader.num_workers
# if True, dataloaders will COPY tensors to CUDA pinned memory locations
train_loader.pin_memory
# timeout value for collecting batch of samples from workers
train_loader.timeout
# generally used with multi processing.
train_loader.worker_init_fn<jupyter_output><empty_output><jupyter_text>We took a detour, so let's get back to loading the data using dataloaders  Continuing with our data preparation...<jupyter_code>train_set = torchvision.datasets.FashionMNIST(root="/home/nikunjlad/data/FashionMNIST",
                                             train=True, download=True,
                                             transform=transforms.Compose([transforms.ToTensor()]))
train_loader = DataLoader(train_set, batch_size=64)
len(train_set)   # no of training samples
train_set.targets # labels
# how many samples are there in the 10 classes of Fashion-MNIST. We have
# balanced data of 6000 samples. If there is imbalance, then over sampling
# will help. This is done by repeating uncommon occurences of data and balance our dataset

# Oversampling is said to be the best method
train_set.targets.bincount()<jupyter_output><empty_output><jupyter_text>OVER SAMPLING is best method of balancing imbalanced data[arXiv publication](https://arxiv.org/abs/1710.05381) for oversampling<jupyter_code>sample = next(iter(train_set))   # create an iterator of the training data
len(sample)   # get length of our sample. we have image-label pairs, hence 2
type(sample)   # create type of our sample
image, label = sample  # sequence unpacking (also called deconstruction of object)<jupyter_output><empty_output><jupyter_text>**iter()** function creates an iterator object representing a **stream** of data. This object essentially holds a pointer and does not load all the data in memory. This pointer then iterates over all the data elements on invocation of **next()** function. This is analogous to the behaviour of how **generators** work in Python.  Displaying some images to understand what our data contains<jupyter_code>image.shape
label
plt.imshow(image.squeeze(), cmap='gray')
print("label", label)
batch = next(iter(train_loader))
len(batch)  # batches still contain images and labels, but now a list instead of single image-label pair
type(batch)  # its a list, instead of a tuple
images, labels = batch   # we get first batch
images.shape   # rank 4 tensor of 64 gray scale images of shape 28x28
labels.shape   # rank 1 having 64 labels corresponding to above data<jupyter_output><empty_output><jupyter_text>Just like for training set which had data which could be accessed individually, the training dataloader has batches of data configured so that it can be accessed efficiently. This helps us in doing essentially the same thing as we did before but now on batches of data<jupyter_code># of the 64 images in the batches, we select the first 10 and display it 
# in a grid using torchvision.utils make_grid() function
grid = torchvision.utils.make_grid(images[:10], nrow=10)

plt.figure(figsize=(15,15))
plt.imshow(np.transpose(grid, (1,2,0)))

print("labels", labels[:10])<jupyter_output>labels tensor([9, 0, 0, 3, 0, 2, 7, 2, 5, 5])<jupyter_text>Building our Neural Network ModelWe need to use, **torch.nn.Module** class. Class is like a blueprint, where people / things of similar nature / functionality exist.  Methods and AttributesObjects are like the actual people / things which exist in a class. There can be multiple objects of a class, just like multiple students exist in a class. Those objects can perform all functions mentioned in the class. Simply put, all students can write, read and listen / understand to things taught in class. Yet each student reads / writes / listens in a different manner based on the inputs it receives or their own indifferences. When an object is created, we call it an **Instance** of a class. The Instance usually has 2 properties:1. **Methods** : represent the code or behavior of the object. For example, a student can run, sing, write, read, eat, etc. These things are behaviour methods of a class. 2. **Attributes** : they represent the data of the object. They hold the data which is to be used by the methods. Inshort they help us know the objects (student's) characteristics. For example, gender (male/female), age, grade level, athlete (yes/no), etc. They define characteristics of the object.Recall, that a class is a **BluePrint** of the object. It does not contain actual information, but is customizable based on user information. **Methods** are standalone and consistent across the class, but the **Attributes** are customizable parameters. EncapsulationAn object has it's own set of attributes and can access all the methods of the class. An object therefore holds both information of attributes and methods, since it can access both of them uniquely. This packaging of attributes and methods into a single class is called **Encapsulation**.<jupyter_code>class Students:
    
    def __init__(self, name, height):
        self.name = name
        self.height = height
        
    def play(self):
        
        if self.height >= 6:
            print("{} can play basketball!".format(self.name))
        else:
            print("{} cannot play basketball".format(self.name))
    
s = Students("Jack", 5)
s.play()
s.name = "Mike"
s.height = 6
s.play()<jupyter_output>Jack cannot play basketball
Mike can play basketball!<jupyter_text>InheritancePyTorch has an inbuilt neural network library which is named torch.nnThere is a base class called **Module** which has all the functionality. i.e it has all the neural networks modules for constructing our model. This means all layers in PyTorch **extends** the nn.Module class. This concept of a class extending another class to gain access of it's attributes and methods is called **Inheritance**.We inherit the **nn.Module** class to get all the functionality of the neural networks into our code. Inshort, if we imagine neural networks to be one large function, then layers are functions and our network is a **composition** of all those functions. Layers and networks are kind of similar in nature due to above analogy. Understanding forward propagationA tensor passes through our network via each layer and undergoes transformation performed on it by each layer. This process of tensor flowing forward through the network is known as **forward pass** or **forward propagation**. Each layer has its own transformations and overall forward pass transformation of the network is the collection of all the transformations from these individual layers.Every **nn.Module** class has a **forward()** method which needs to be implemented in order to provide forward pass information. We need to implement the forward() method everytime we need to design our model.When we implement the **forward()** function, we typically call the **nn.Functional** module. It provides packages for implementing our forward function.<jupyter_code>class Net(nn.Module):  # inheriting the nn.Module class
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(in_channels=1, out_channels=6, kernel_size=5)
        self.conv2 = nn.Conv2d(in_channels=6, out_channels=12, kernel_size=5)
        
        self.fc1 = nn.Linear(in_features=12*4*4, out_features=120)
        self.fc2 = nn.Linear(in_features=120, out_features=60)
        self.out = nn.Linear(in_features=60, out_features=10)
        
    """
    Implementation of forward function takes in a tensor t and transforms it using the dummy layer.
    """
    def forward(self, t):
        t = self.layer(t)
        return t<jupyter_output><empty_output><jupyter_text>The Module class is keeping track of the networks weights which are contained within each layer. Linear layers are also called **Dense** layers<jupyter_code># we get the network architecture here.
net = Net()
net<jupyter_output><empty_output><jupyter_text>We get a good string representation of our network object above. This is because we have extended the nn.Module class which prints this. If we remove it, then we get just an object reference of the Net() class like below:<jupyter_code>class Net():  # inheriting the nn.Module class
    def __init__(self):
#         super(Net, self).__init__()
        self.conv1 = nn.Conv2d(in_channels=1, out_channels=6, kernel_size=5)
        self.conv2 = nn.Conv2d(in_channels=6, out_channels=12, kernel_size=5)
        
        self.fc1 = nn.Linear(in_features=12*4*4, out_features=120)
        self.fc2 = nn.Linear(in_features=120, out_features=60)
        self.out = nn.Linear(in_features=60, out_features=10)
        
    """
    Implementation of forward function takes in a tensor t and transforms it using the dummy layer.
    """
    def forward(self, t):
        t = self.layer(t)
        return t

## This is just used for illustration purposes. 
net = Net()
net<jupyter_output><empty_output><jupyter_text>We use **__repr__** we can have a valuable message to be displayed when we print an object. Special OOP methods in python are called **Dunder methods**. They are called Dunder because of **Double Underscore**.Stride helps us define how much do we need to shift the kernel to the right and below during convolution operation.  Argument vs ParametersParameters are used inside function definition. They act as placeholders. They are like local variables.Arguments are actual values which are passed into the functions. Values comes from outside the function to inside. 2 types of parameters enter the network:1. hyperparameter2. data dependent hyperparameters.When we contruct a layer we pass parameters and values to the layers. For convolutional layers we have 3 parameters and for FC layers we have 2 parameters.**Hyperparameters** values are chosen manually or arbitrarily. Manually choosing hyperparameters include **kernel_size**, **out_channels** and **out_features**, etc.Data dependent hyperparameters include, **input channels** which is equal to depth of the image (either 1 or 3) and **output features** (i.e number of classes in the dataset)  Filters vs KernelsA kernel is a small nxn matrix which traverses over the image. They are usually odd numbered (preferably, like 3x3, 5x5, 7x7, etc) but can be even numbered too like 2x2 or 4x4.A filter is basically a collection of kernels. **Output channels** are equal to ** of filters**. They are commonly known as **Feature Maps**. If we are dealing with linear layers we don't call them feature maps, since they are a rank 1 tensor  Learnable parametersLearnable parameters are parameters whose values are trained during training process. We typically start with an arbitrary set of values and they get updated iteratively as the network learns.Our objective is to train the network so that it can **learn** the best set of these parameters in order to **minimize** the error it makes (i.e loss function)These parameters are commonly called **weights**.<jupyter_code>class Net(nn.Module):  # inheriting the nn.Module class
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(in_channels=3, out_channels=6, kernel_size=5)
        self.conv2 = nn.Conv2d(in_channels=6, out_channels=12, kernel_size=5)
        
        self.fc1 = nn.Linear(in_features=12*4*4, out_features=120)
        self.fc2 = nn.Linear(in_features=120, out_features=60)
        self.out = nn.Linear(in_features=60, out_features=10)
        
    """
    Implementation of forward function takes in a tensor t and transforms it using the dummy layer.
    """
    def forward(self, t):
        t = self.layer(t)
        return t
    
net = Net()
net
net.conv1
net.conv2
net.fc1
net.fc2
net.out<jupyter_output><empty_output><jupyter_text>Just like the network object returns a string **representation(__repr__)** of the object, we can get individual layer informations as well using the **dot(.)** notation. Lets look at the initialized weights inside this functions<jupyter_code>net.conv1.weight<jupyter_output><empty_output><jupyter_text>Above output proves that each filter has different kernels. We can see, each filter has 3 kernels (each unique in nature) of size 5x5. They are randomly initialized. These 3 kernels together make up a filter. Since a filter has 3 kernels and there are 6 filters, we have total of 18 kernels. Since image is a color image, a filter has 3 kernels each corresponding to an RGB channel. If grayscale, then each filter will have only 1 kernel.<jupyter_code>net.conv1.weight.shape
net.conv2.weight.shape
net.fc1.weight.shape
net.fc2.weight.shape
net.out.weight.shape
net.conv1.weight[0].shape<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07302980043871382, 'fraction_numerical': 0.011663367396072976, 'mean_word_length': 4.417971014492753, 'pattern_counts': {'":': 0, '<': 70, '<?xml version=': 0, '>': 71, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 14, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '191950', 'license': 'MIT', 'n_tokens_mistral': 5114, 'n_tokens_neox': 4894, 'n_words': 2235, 'path': 'pytorch/Fashion-MNIST Classifier.ipynb', 'repo_name': 'nikunjlad/Coding-Playground', 'size': '18691'}