<jupyter_start><jupyter_text>MNIST<jupyter_code>import torch
from torchvision import datasets, transforms
import numpy as np
%matplotlib inline
from matplotlib import pyplot as plt<jupyter_output><empty_output><jupyter_text>Description Classification of hand-written digits (MNIST dataset) using a simple multi-layer perceptorn architecture implemented using only `torch.tensor`s. The `relu` activation function, the `softmax` function and the `cross_entropy_loss` loss function are manually implemented as well as the weights initialisation.  Content 1. Loading a training dataset    1. Apply transformations    2. Creating a data loader2. Definition of functions    1. ReLU activation function    2. Softmax function    3. Cross-entropy loss function3. Manually defining a MLP model    1. Forward pass    2. Weight and biases initialisation4. Traning  Data Set  MNIST The MNIST data set is a large database of handwritten digits, often used as toy model to validate machine learning and deep learning algorithms in image recognition and classification.  Loading the Data Set Let's start by loading the MNIST dataset. Since we want to use the data (both images and associated labels) with PyTorch we have to load the dataset into `torch.tensor`s. Additionally we want our data to be normalised in order to avoid large variations in the data; we can compose the two transformation and apply them directly when loading the dataset.<jupyter_code>transform = transforms.Compose(
    [
        transforms.ToTensor(), # Array to PyTorch tensor
        transforms.Normalize((0.5,), (0.5,)), # Normalisation (mean and std)
    ]
)

trainset = datasets.MNIST('data', download=True, train=True, transform=transform)<jupyter_output><empty_output><jupyter_text>*Note*: some `torchvision.transforms` apply directly to the loaded image and therefore go before `transforms.ToTensor()` while other `torchvision.transforms` are applied directly to `torch.tensors` and therefore go after `transforms.ToTensor()`. Once the dataset is loaded and transformed, we can define a `DataLoader`. This is an utility class that allows to split the dataset in *minibatches*, used for training.<jupyter_code>trainloader = torch.utils.data.DataLoader(trainset, batch_size=64, shuffle=True, drop_last=True)<jupyter_output><empty_output><jupyter_text>Visualizing Images and Labels Let's look at the first batch of images:<jupyter_code>dataiter = iter(trainloader) # Create an iterator from trainloader

images, labels = next(dataiter)

fig = plt.figure(figsize=(8,8))
for idx in range(64):
    ax = fig.add_subplot(8, 8, idx + 1, xticks=[], yticks=[])
    
    # Un-normalize image
    img = images[idx].numpy().squeeze() * 0.5 + 0.5
           
    plt.imshow(img, cmap='Greys')
    
    ax.set_title(labels[idx].item(), fontdict={"fontsize": 12})

plt.tight_layout()
plt.show()<jupyter_output><empty_output><jupyter_text>Neural Network with `torch.tensor` [PyTorch](https://pytorch.org/) is a modern an powerful deep learning framework that provides all the building blocks needed to define state-of-the-art deep learning architectures. However, PyTorch's "magic" is based on `torch.tensor` and `torch.autograd`: tensors are used for fast computation (possibly GPU-accelerated) while `torch.autograd` keeps track of such computations and make it easy to compute gradients via backpropagation. Here we will try to implement a simple multi-layer perceptron (MLP) using only `torch.tensor`.  Define the Neural Network In order to define a multi-layer perceptron we need an activation function. Here we use the rectified linear unit (ReLU) activation function defined by$$    \text{ReLU}(x) = \begin{cases} 0 \quad x \le 0 \\ x \quad x > 0 \end{cases}$$<jupyter_code>def relu(x):
    z = torch.zeros_like(x)
    
    return torch.max(x, z)<jupyter_output><empty_output><jupyter_text>This is equivalent to `nn.ReLU`. With the activation function we can finally define our model, which in this case is a multi-layer perceptron:<jupyter_code>def nnmodel(x, weights, biases):
    for w, b in zip(weights[:-1], biases[:-1]):
        x = torch.matmul(x, w) + b # Linear unit
        x = relu(x) # 
        
    return torch.matmul(x, weights[-1]) + biases[-1]<jupyter_output><empty_output><jupyter_text>The previous model outputs *logits*, the vector of raw predictions of out classification model. We can transform the output into class probabilities using the *softmax* function:$$    \text{softmax}(\mathbf{x}) = \frac{e^\mathbf{x}}{\sum_i e^{x_i}}$$<jupyter_code>def softmax(x):
    # The first dimension is batch_size
    # To sum over classes we have to sum over dim=1
    return torch.exp(x) / torch.sum(torch.exp(x), dim=1).view(-1, 1)<jupyter_output><empty_output><jupyter_text>This is equivalent to `nn.Softmax`.  Weights and Biases Initialization Weights and baises initialisation is a delicate business and care should be taken when initialising the parameters of a model. The general consensus is that the weighs should be initialised from a uniform random distributions on the interval $[-y, y]$ where $y = 1/\sqrt{n}$ for a layer with $n$ inputs. The following utility function defines a random `np.ndarray` if given size, with random number uniformely distributed over $[-y, y]$:<jupyter_code>def uniform(*size):
    n = size[0] # Input size
    
    isn = 1 / np.sqrt(n)
    
    samples = np.random.uniform(-isn, isn, size=size)
    
    return samples<jupyter_output><empty_output><jupyter_text>*Note*: A normali distribution $\mathcal{N}(0, y)$ would have similar performance. See [Understanding the difficulty of training deep feedforward neural networks](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf) for more informations. With the `uniform` function we can finally declare the weights and biases of the multi-layer perceptron and collect them in lists that will be passed to `nnmodel` during the forward propagation:<jupyter_code>w1 = torch.tensor(uniform(784, 256), requires_grad=True, dtype=torch.float)
b1 = torch.tensor(uniform(256), requires_grad=True, dtype=torch.float)
w2 = torch.tensor(uniform(256, 128), requires_grad=True, dtype=torch.float)
b2 = torch.tensor(uniform(128), requires_grad=True, dtype=torch.float)
w3 = torch.tensor(uniform(128, 10), requires_grad=True, dtype=torch.float)
b3 = torch.tensor(uniform(10), requires_grad=True, dtype=torch.float)

weights = [w1, w2, w3]
biases = [b1, b2, b3]<jupyter_output><empty_output><jupyter_text>Test Forward Pass In order to check that the model is defined correctly we can perfom a forward pass with a batch of images. We can also see how the model performs before training  by plotting predicting class probabilities.<jupyter_code># Get one batch of data
images, labels = next(dataiter)

# Define a wrapper to nnmodel
# This incorporated weights and biases
def model(x):
    return nnmodel(x, weights, biases)

def showclassp(images, labels, model):
    """
    Plot class probabilities for a batch of images and labels.
    """
    
    # Defint a figure
    fig = plt.figure(figsize=(12,12))
    
    # Flatten image for forward pass
    images = images.view(images.shape[0], -1)

    # Compute predictions
    with torch.no_grad(): # Do not track gradients
        # Perform forward pass
        out = model(images)
            
        # Compute class probabilities
        p = softmax(out).numpy()
    
    # Loop over images and labels in a batch
    for idx in range(64):
        
        # Create subplot
        ax = fig.add_subplot(8, 8, idx + 1, xticks=range(10), yticks=[0, 1])

        # Plot all class probabilities for given image
        for i in range(10):
            if labels[idx] == i:
                if labels[idx] == np.argmax(p[idx]):
                    plt.bar(i, p[idx,i], color="g")
                else:
                    plt.bar(i, p[idx,i], color="r")
            else:
                plt.bar(i, p[idx,i], color="k")
        plt.ylim([0,1.25])

        ax.set_title(labels[idx].item(), fontdict={"fontsize": 12})

    plt.tight_layout()
    plt.show()
    
showclassp(images, labels, model)<jupyter_output><empty_output><jupyter_text>Since the weights of the network are initialised at random, every class has a similar probability close to $\frac{1}{10}$.  Training To train our model we need to define a loss function. For this multi-class classification problem we can use the cross-entropy loss function defined by$$    \mathcal{L}(y, \hat{y}) = -\frac{1}{N}\sum_{n}^N \left[y_n \log(\hat{y}_n) + (1 - y_n) \log(1 - \hat{y}_n) \right]$$<jupyter_code>def cross_entropy_loss(p, labels):
    
    n = len(labels)
    
    # One-hot encoding
    l = torch.zeros((n, 10))
    l[range(n),labels] = 1
    
    # Compute cross-entropy loss
    return -torch.sum(l * torch.log(p), dim=1).mean()<jupyter_output><empty_output><jupyter_text>This is equivalent to `nn.CrossEntropyLoss`. Now that everything is in place, we can finally train our model. For the optimisation we use the simple gradient descent algorithm.<jupyter_code>lr = 0.03
epochs = 10

for epoch in range(epochs):
    
    epoch_loss = 0
    
    for images, labels in trainloader:
        
        inputs = images.view(images.shape[0], -1)
        
        # Forward pass
        out = nnmodel(inputs, weights, biases)
        
        # Compute class-probabilities
        p = softmax(out)
        
        # Compute loss
        loss = cross_entropy_loss(p, labels)
        
        # Compute gradients with backpropagation
        loss.backward()
        
        # sub_ and zero_ perform in-place
        for w, b in zip(weights, biases):
            # Perform gradient descent step
            w.data.sub_(lr * w.grad.data)
            b.data.sub_(lr * b.grad.data)
            
            # Zero the gradients
            w.grad.data.zero_()
            b.grad.data.zero_()
            
        epoch_loss += loss.item()
    else:
        print(f"Loss #{epoch}: {epoch_loss/len(trainloader)}")<jupyter_output>Loss #0: 0.5790785283104715
Loss #1: 0.26857152758533953
Loss #2: 0.20366187446741754
Loss #3: 0.16235460102430913
Loss #4: 0.13466131538470755
Loss #5: 0.11456972584744624
Loss #6: 0.0990276684241097
Loss #7: 0.08665569945414459
Loss #8: 0.07682453568531807
Loss #9: 0.06851563683393926<jupyter_text>The training loop above changes the weights and biases of the networs using gradient descent in order to minimize the loss function (`cross_entropy_loss`). We can finally evaluate the performance of the model:<jupyter_code>images, labels = next(dataiter)

showclassp(images, labels, model)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0999905580209612, 'fraction_numerical': 0.029836653762628645, 'mean_word_length': 3.7497757847533633, 'pattern_counts': {'":': 2, '<': 52, '<?xml version=': 0, '>': 53, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '342646', 'license': 'MIT', 'n_tokens_mistral': 3441, 'n_tokens_neox': 3195, 'n_words': 1098, 'path': '00_Deep_Learning_PyTorch/05_intro-to-pytorch/02-MNIST-MLP-tensor.ipynb', 'repo_name': 'RMeli/BertelsmannAI', 'size': '10591'}