<jupyter_start><jupyter_text>Multi-Layer Perceptron, MNIST---In this notebook, we will train an MLP to classify images from the [MNIST database](http://yann.lecun.com/exdb/mnist/) hand-written digit database.The process will be broken down into the following steps:>1. Load and visualize the data2. Define a neural network3. Train the model4. Evaluate the performance of our trained model on a test dataset!Before we begin, we have to import the necessary libraries for working with data and PyTorch.<jupyter_code>#import libraries
import torch
import numpy as np<jupyter_output><empty_output><jupyter_text>--- Load and Visualize the [Data](http://pytorch.org/docs/stable/torchvision/datasets.html)Downloading may take a few moments, and you should see your progress as the data is loading. You may also choose to change the `batch_size` if you want to load more data at a time.This cell will create DataLoaders for each of our datasets.<jupyter_code>from torchvision import datasets
import torchvision.transforms as transforms

#number of subprocess used for loading data
num_workers=0
#how many samples per batch to load
batch_size=20

#convert data to torch.FloatTensor
transform=transforms.ToTensor()

#choose the training and test datasets
train_data= datasets.MNIST(root='data',train=True,download=True,
                           transform=transform)
test_data=datasets.MNIST(root='data',train=False,download=True,
                         transform=transform)

#prepare data loader
train_loader=torch.utils.data.DataLoader(train_data,batch_size=batch_size,
                                         num_workers=num_workers)
test_loader=torch.utils.data.DataLoader(test_data,batch_size=batch_size,
                                        num_workers=num_workers)<jupyter_output>Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz to data/MNIST/raw/train-images-idx3-ubyte.gz<jupyter_text>Visualize a Batch of Training DataThe first step in a classification task is to take a look at the data, make sure it is loaded in correctly, then make any initial observations about patterns in that data.<jupyter_code>import matplotlib.pyplot as plt
%matplotlib inline

#obtain one batch of training images
dataiter=iter(train_loader)
images,labels=dataiter.next()
images=images.numpy()
# plot the images in the batch, along with the corresponding labels
fig=plt.figure(figsize=(25,4))

for idx in np.arange(20):
  ax=fig.add_subplot(2,20/2,idx+1,xticks=[],yticks=[])
  ax.imshow(np.squeeze(images[idx]),cmap='gray')
  #print ourt the correct label for each image
  #.item() gets the value contained in a Tensor
  ax.set_title(str(labels[idx].item()))<jupyter_output><empty_output><jupyter_text>View an Image in More Detail<jupyter_code>img=np.squeeze(images[1])

fig= plt.figure(figsize=(12,12))
ax=fig.add_subplot(111)
ax.imshow(img,cmap='gray')

width,height =img.shape
thresh = img.max()/2.5

for x in range(width):
  for y in range(height):
    val=round(img[x][y],2) if img[x][y]!=0 else 0
    ax.annotate(str(val),xy=(y,x),
                horizontalalignment='center',
                verticalalignment='center',
                color='white'if img[x][y]<thresh else 'black')<jupyter_output><empty_output><jupyter_text>--- Define the Network [Architecture](http://pytorch.org/docs/stable/nn.html)The architecture will be responsible for seeing as input a 784-dim Tensor of pixel values for each image, and producing a Tensor of length 10 (our number of classes) that indicates the class scores for an input image. This particular example uses two hidden layers and dropout to avoid overfitting.<jupyter_code>import torch.nn as nn
import torch.nn.functional as F

#Defining the Neural Network (NN) Architecture
class Net(nn.Module):
  def __init__(self):
    super(Net,self).__init__()

    #no of hidden nodes in each hidden layer (512)
    hidden_1=512
    hidden_2=512
    #linear layer (784 to hidden1)
    self.fc1=nn.Linear(28*28,hidden_1)
    
    #linear layer (hidden1 to hidden2)
    self.fc2=nn.Linear(hidden_1,hidden_2)

    #linear layer(hidden 2 to final(i.e 10))
    self.fc3=nn.Linear(hidden_2,10)

    #dropout layer(p=0.2)
    #Droup out prevents overfitting data
    self.dropout = nn.Dropout(0.2)
  def forward(self,x):
    #flatten image input
    x =x.view(-1,28*28)
    # add hidden layer with relu Activation function
    x=F.relu(self.fc1(x))
    # add dropout layer
    x= self.dropout(x)
    # add hidden layer , with relu activation function
    x=F.relu(self.fc2(x))
    # add dropout layer
    x= self.dropout(x)
    # add output layer
    x= self.fc3(x)
    return x
# initialize the Neural Network
model=Net()
print(model)<jupyter_output>Net(
  (fc1): Linear(in_features=784, out_features=512, bias=True)
  (fc2): Linear(in_features=512, out_features=512, bias=True)
  (fc3): Linear(in_features=512, out_features=10, bias=True)
  (dropout): Dropout(p=0.2, inplace=False)
)<jupyter_text>Specify [Loss Function](http://pytorch.org/docs/stable/nn.htmlloss-functions) and [Optimizer](http://pytorch.org/docs/stable/optim.html)It's recommended that you use cross-entropy loss for classification. If you look at the documentation (linked above), you can see that PyTorch's cross entropy function applies a softmax funtion to the output layer *and* then calculates the log loss.<jupyter_code># specify loss function (categorical cross-entropy)
criterion = nn.CrossEntropyLoss()
# specify optimizer (stochastic gradient descent) 
#learning rate = 0.01
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)<jupyter_output><empty_output><jupyter_text>--- Train the NetworkThe steps for training/learning from a batch of data are described in the comments below:1. Clear the gradients of all optimized variables2. Forward pass: compute predicted outputs by passing inputs to the model3. Calculate the loss4. Backward pass: compute gradient of the loss with respect to model parameters5. Perform a single optimization step (parameter update)6. Update average training lossThe following loop trains for 50 epochs; take a look at how the values for the training loss decrease over time. We want it to decrease while also avoiding overfitting the training data.<jupyter_code># no of epochs to train the model
n_epochs=50

#prepare model for training
model.train()

for epoch in range(n_epochs):
  #to monitor the training loss
  train_loss=0.0
  
  ###############################
  #         Train The Model     #
  ################################

  for data, target in train_loader:
    #clear the gradient of all optimized variables
    optimizer.zero_grad()
    # forward pass :compute predicted outputs by passing inputs to the model
    output =model(data)
    #calculate the loss
    loss=criterion(output,target)
    # backward pass: compute gradient of the loss with respect to model parameters
    loss.backward()
    # perform a single optimization step (parameter update)
    optimizer.step()
    # update running training loss
    train_loss+=loss.item()*data.size(0)
  
  # print train statistics
  #calculate average loss over an epoch
  train_loss=train_loss/len(train_loader.dataset)
  print('Epoch: {} \tTraining Loss: {:.6f}'.format(
        epoch+1, 
        train_loss
        ))<jupyter_output>Epoch: 1 	Training Loss: 0.830495
Epoch: 2 	Training Loss: 0.320684
Epoch: 3 	Training Loss: 0.248325
Epoch: 4 	Training Loss: 0.200470
Epoch: 5 	Training Loss: 0.168571
Epoch: 6 	Training Loss: 0.145481
Epoch: 7 	Training Loss: 0.128156
Epoch: 8 	Training Loss: 0.113390
Epoch: 9 	Training Loss: 0.102459
Epoch: 10 	Training Loss: 0.092343
Epoch: 11 	Training Loss: 0.085404
Epoch: 12 	Training Loss: 0.079393
Epoch: 13 	Training Loss: 0.073015
Epoch: 14 	Training Loss: 0.068202
Epoch: 15 	Training Loss: 0.062383
Epoch: 16 	Training Loss: 0.058107
Epoch: 17 	Training Loss: 0.055073
Epoch: 18 	Training Loss: 0.051843
Epoch: 19 	Training Loss: 0.047442
Epoch: 20 	Training Loss: 0.045712
Epoch: 21 	Training Loss: 0.042775
Epoch: 22 	Training Loss: 0.040399
Epoch: 23 	Training Loss: 0.038401
Epoch: 24 	Training Loss: 0.036254
Epoch: 25 	Training Loss: 0.034061
Epoch: 26 	Training Loss: 0.032745
Epoch: 27 	Training Loss: 0.031158
Epoch: 28 	Training Loss: 0.028857
Epoch: 29 	Training Loss: 0.0[...]<jupyter_text>--- Test the Trained NetworkFinally, we test our best model on previously unseen **test data** and evaluate it's performance. Testing on unseen data is a good way to check that our model generalizes well. It may also be useful to be granular in this analysis and take a look at how this model performs on each class as well as looking at its overall loss and accuracy.<jupyter_code># initialize lists to monitor test loss and accuracy
test_loss=0.0
class_correct=list(0. for i in range(10))
class_total=list(0. for i in range(10))

#prepare model for taining
model.eval()

for data,target in test_loader:
  # forward pass: compute predicted output by passing inputs to the model
  output=model(data)
  #calculate the loss
  loss=criterion(output,target)

  #update the test loss
  test_loss+= loss.item()*data.size(0)
  
  #convert output probabilites to predicted class
  _, pred= torch.max(output,1)

  #compare prediction to true label
  correct=np.squeeze(pred.eq(target.data.view_as(pred)))

  #calculate test accuracy for each object class
  for i in range(batch_size):
    label=target.data[i]
    class_correct[label]+= correct[i].item()
    class_total[label]+=1
#calculate and print avg test loss
test_loss =test_loss/len(test_loader.dataset)
print('Test Loss: {:.6f}\n'.format(test_loss))

for i in range(10):
  if class_total[i] > 0:
    print('Test Accuracy of %5s: %2d%% (%2d/%2d)' % (
        str(i), 100 * class_correct[i] / class_total[i],
        np.sum(class_correct[i]), np.sum(class_total[i])))
  else:
      print('Test Accuracy of %5s: N/A (no training examples)' % (classes[i]))

print('\nTest Accuracy (Overall): %2d%% (%2d/%2d)' % (
    100. * np.sum(class_correct) / np.sum(class_total),
    np.sum(class_correct), np.sum(class_total)))<jupyter_output>Test Loss: 0.054274

Test Accuracy of     0: 99% (973/980)
Test Accuracy of     1: 99% (1127/1135)
Test Accuracy of     2: 97% (1011/1032)
Test Accuracy of     3: 98% (994/1010)
Test Accuracy of     4: 98% (967/982)
Test Accuracy of     5: 98% (876/892)
Test Accuracy of     6: 98% (940/958)
Test Accuracy of     7: 97% (1006/1028)
Test Accuracy of     8: 97% (949/974)
Test Accuracy of     9: 97% (987/1009)

Test Accuracy (Overall): 98% (9830/10000)<jupyter_text>Visualize Sample Test ResultsThis cell displays test images and their labels in this format: `predicted (ground-truth)`. The text will be green for accurately classified examples and red for incorrect predictions.<jupyter_code># obtain one batch of test images
dataiter = iter(test_loader)
images, labels = dataiter.next()

# get sample outputs
output = model(images)
# convert output probabilities to predicted class
_, preds = torch.max(output, 1)
# prep images for display
images = images.numpy()

# plot the images in the batch, along with predicted and true labels
fig = plt.figure(figsize=(25, 4))
for idx in np.arange(20):
    ax = fig.add_subplot(2, 20/2, idx+1, xticks=[], yticks=[])
    ax.imshow(np.squeeze(images[idx]), cmap='gray')
    ax.set_title("{} ({})".format(str(preds[idx].item()), str(labels[idx].item())),
                 color=("green" if preds[idx]==labels[idx] else "red"))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10713350213917751, 'fraction_numerical': 0.0461014581332402, 'mean_word_length': 4.28808864265928, 'pattern_counts': {'":': 0, '<': 34, '<?xml version=': 0, '>': 35, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '117310', 'license': 'MIT', 'n_tokens_mistral': 4079, 'n_tokens_neox': 3670, 'n_words': 1214, 'path': 'MLP_MNIST1.ipynb', 'repo_name': 'souravgopal25/DeepLearnigNanoDegree', 'size': '11453'}