<jupyter_start><jupyter_text>Classifying Fashion-MNISTNow it's your turn to build and train a neural network. You'll be using the [Fashion-MNIST dataset](https://github.com/zalandoresearch/fashion-mnist), a drop-in replacement for the MNIST dataset. MNIST is actually quite trivial with neural networks where you can easily achieve better than 97% accuracy. Fashion-MNIST is a set of 28x28 greyscale images of clothes. It's more complex than MNIST, so it's a better representation of the actual performance of your network, and a better representation of datasets you'll use in the real world.In this notebook, you'll build your own neural network. For the most part, you could just copy and paste the code from Part 3, but you wouldn't be learning. It's important for you to write the code yourself and get it to work. Feel free to consult the previous notebooks though as you work through this.First off, let's load the dataset through torchvision.<jupyter_code>%matplotlib inline
import matplotlib.pyplot as plt
import torch
from torchvision import datasets
import torchvision.transforms as T
import helper
import torch.nn as nn
import torch.nn.functional as F
from tqdm.notebook import tqdm

transform = T.Compose([ T.ToTensor(), 
                        T.Normalize((0.5,),(0.5,)) ])

trainset = datasets.FashionMNIST('~/.pytorch/F_MNIST_data/', train=True, download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=64, shuffle=True)

testset = datasets.FashionMNIST('~/.pytorch/F_MNIST_data/', train=False, download=True, transform=transform)
testloader = torch.utils.data.DataLoader(testset, batch_size=len(testset), shuffle=False)<jupyter_output>Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-images-idx3-ubyte.gz
Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-images-idx3-ubyte.gz to /home/testuser/.pytorch/F_MNIST_data/FashionMNIST/raw/train-images-idx3-ubyte.gz<jupyter_text>0 T-shirt/top  1 Trouser  2 Pullover  3 Dress  4 Coat  5 Sandal  6 Shirt  7 Sneaker  8 Bag  9 Ankle boot<jupyter_code>image, label = next(iter(trainloader))
plt.imshow(image[0,:].permute(1,2,0).squeeze().numpy(), cmap='Greys_r');
helper.imshow(image[0,:]);<jupyter_output><empty_output><jupyter_text>Here we can see one of the images.  Building the networkHere you should define your network. As with MNIST, each image is 28x28 which is a total of 784 pixels, and there are 10 classes. You should include at least one hidden layer. We suggest you use ReLU activations for the layers and to return the logits or log-softmax from the forward pass. It's up to you how many layers you add and the size of those layers.<jupyter_code># TODO: Define your network architecture here

class fmnist(nn.Module):
    def __init__(self, layers, n_input=784, n_classes = 10, dropout_prob=0):
        
        def fmnist_block(f_in, f_out, dropout_prob=0):
            return nn.Sequential(*[nn.Dropout(p=dropout_prob), nn.Linear(f_in, f_out), nn.ReLU()])
        
        super().__init__()
        self.dropout_prob = dropout_prob
        #add first and last dimensions
        layers.insert(0, n_input)
        layers.append(n_classes)
        
        net = [fmnist_block(layers[i-1], layers[i], dropout_prob=self.dropout_prob) for i in range(1, len(layers))]
        self.net = nn.Sequential(*net)
                         
    def forward(self, x):
        return F.log_softmax(self.net(x.view(x.shape[0],-1)), dim=1)  
# TODO: Define your network architecture here

class fmnist_listed(nn.Module):
    def __init__(self, layers, n_input=784, n_classes = 10, dropout_prob=0):
        
        def fmnist_block(f_in, f_out, dropout_prob=0):
            return [nn.Dropout(p=dropout_prob), nn.Linear(f_in, f_out), nn.ReLU()]
        
        super().__init__()
        self.dropout_prob = dropout_prob
        #add first and last dimensions
        layers.insert(0, n_input)
        layers.append(n_classes)
        self.net = nn.ModuleList()
        
        for f_in, f_out in zip(layers[:-1], layers[1:]):
            self.net.extend(fmnist_block(f_in, f_out, self.dropout_prob))
                             
    def forward(self, x):
        out = x.view(x.shape[0],-1)
        for layer in self.net:
            out = layer(out)
        return F.log_softmax(out, dim=1)  

criterion = nn.NLLLoss()
#net = fmnist([128,64,32])
#net = fmnist([256,128,64], dropout_prob=0.5)
net = fmnist_listed([256,128,64], dropout_prob=0.5)
device = 'cuda:0' if torch.cuda.is_available() else 'cpu'
display(device)
#device = 'cpu'
net.to(device);
image.shape
image.view(64,-1).shape
# sanity check
with torch.no_grad():
    out = net(image.to(device))
torch.exp(out).sum()
out.shape<jupyter_output><empty_output><jupyter_text>Train the networkNow you should create your network and train it. First you'll want to define [the criterion](http://pytorch.org/docs/master/nn.htmlloss-functions) ( something like `nn.CrossEntropyLoss`) and [the optimizer](http://pytorch.org/docs/master/optim.html) (typically `optim.SGD` or `optim.Adam`).Then write the training code. Remember the training pass is a fairly straightforward process:* Make a forward pass through the network to get the logits * Use the logits to calculate the loss* Perform a backward pass through the network with `loss.backward()` to calculate the gradients* Take a step with the optimizer to update the weightsBy adjusting the hyperparameters (hidden units, learning rate, etc), you should be able to get the training loss below 0.4.<jupyter_code># TODO: Create the network, define the criterion and optimizer
# done before
def init_weights(m):
    """
    initialize weights with Kaming 
    https://towardsdatascience.com/weight-initialization-in-neural-networks-a-journey-from-the-basics-to-kaiming-954fb9b47c79
    """
    if isinstance(m, nn.ConvTranspose2d) or isinstance(m, nn.Conv2d) or isinstance(m, nn.Linear):
        nn.init.kaiming_normal_(m.weight)
        #torch.nn.init.kaiming_normal_(m.bias)
net.apply(init_weights);<jupyter_output><empty_output><jupyter_text>Requires Grad block<jupyter_code>net.net[0][0].weight.requires_grad=False
net.net[0][0].bias.requires_grad=False
net.net[0][0].weight.requires_grad, net.net[0][0].bias.requires_grad
def check_grads(n):
    '''
    check if all submodules have requires grads
    '''
    if isinstance(n, nn.Linear):
        print(n)
        for name, par in n.named_parameters():
            print(name, par.requires_grad)
        print('---')
net.apply(check_grads)
net.requires_grad_(True)
net.apply(check_grads)
net.net[0].requires_grad_(False)
net.apply(check_grads)
%%time
# TODO: Train the network here
N_EPOCH = 30

# return back to requires grad
net.requires_grad_(True);

# create an optimizer
optimizer = torch.optim.Adam(net.parameters())
criterion = nn.NLLLoss()
net.train()
net.to(device)

train_loss = []
test_loss = []
for epoch in tqdm(range(N_EPOCH)):
    running_loss = 0
    net.train()
    for data, labels in trainloader:
        optimizer.zero_grad()
        out = net(data.to(device))
        loss_item = criterion(out, labels.to(device))
        loss_item.backward()
        optimizer.step()
        running_loss +=loss_item.item()
    train_loss.append(running_loss/len(trainloader))
    print(f'Epoch {epoch}, loss = {train_loss[-1]}')
    
    
    net.eval()
    with torch.no_grad():
        # don't need a loop cos all in one batch
        data, labels = next(iter(testloader))
        ps = net(data.to(device))
        test_loss.append(criterion(ps, labels.to(device)))
        ps = torch.exp(ps)
        _, predicted_classes = ps.max(dim=1)
        predicted_classes = predicted_classes.to('cpu')
        accuracy_score = sum(predicted_classes == labels).item()/len(labels)
plt.plot(list(range(N_EPOCH)), test_loss, label='test')
plt.plot(list(range(N_EPOCH)), train_loss, label='train')
plt.legend(loc='best')
print(f'Accuracy score for the network is {accuracy_score}')
plt.show()<jupyter_output>Accuracy score for the network is 0.8508<jupyter_text>Calculate the accuracy<jupyter_code>with torch.no_grad():
    net.to(device)
    data, labels = next(iter(testloader))
    predictions = torch.exp(net(data.to(device)))

top_p, top_class = predictions.topk(1,dim=1)
equals = top_class == labels.view(top_class.shape).to(device)
accuracy_score = torch.mean(equals.type(torch.FloatTensor)).item()
print(f'accuracy score = {accuracy_score:.2f}')
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score
for i in range(len(trainset.classes)):
    print(f'{trainset.classes[i]:>11} : precision = {precision_score(predicted_classes==i, labels==i):.2f} \
recall = {recall_score(predicted_classes==i, labels==i):.2f}')
%matplotlib inline
%config InlineBackend.figure_format = 'retina'

import helper

# Test out your network!

dataiter = iter(testloader)
images, labels = dataiter.next()
img = images[0]

# TODO: Calculate the class probabilities (softmax) for img
net.eval()
with torch.no_grad():
    ps = torch.exp(net(img.to(device)))

# Plot the image and probabilities
helper.view_classify(img.resize_(1, 28, 28), ps.to('cpu'), version='Fashion')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11907602212341395, 'fraction_numerical': 0.015833423706756316, 'mean_word_length': 4.126181211784325, 'pattern_counts': {'":': 0, '<': 23, '<?xml version=': 0, '>': 24, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '223228', 'license': 'MIT', 'n_tokens_mistral': 3150, 'n_tokens_neox': 3033, 'n_words': 784, 'path': 'intro-to-pytorch/Part 4 - Fashion-MNIST (Exercises).ipynb', 'repo_name': 'chorus12/deep-learning-v2-pytorch', 'size': '9221'}