<jupyter_start><jupyter_text>Classifying Fashion-MNISTNow it's your turn to build and train a neural network. You'll be using the [Fashion-MNIST dataset](https://github.com/zalandoresearch/fashion-mnist), a drop-in replacement for the MNIST dataset. MNIST is actually quite trivial with neural networks where you can easily achieve better than 97% accuracy. Fashion-MNIST is a set of 28x28 greyscale images of clothes. It's more complex than MNIST, so it's a better representation of the actual performance of your network, and a better representation of datasets you'll use in the real world.In this notebook, you'll build your own neural network. For the most part, you could just copy and paste the code from Part 3, but you wouldn't be learning. It's important for you to write the code yourself and get it to work. Feel free to consult the previous notebook though as you work through this.First off, let's load the dataset through torchvision.<jupyter_code>import torch
from torchvision import datasets, transforms
import helper

# Define a transform to normalize the data
# If the img has three channels, you should have three number for mean, for example, 
# img is RGB, mean is [0.5, 0.5, 0.5], the normalize result is R * 0.5, G * 0.5, B * 0.5. 
# If img is grey type that only one channel, so mean should be [0.5], the normalize result is R * 0.5
transform = transforms.Compose([transforms.ToTensor(),
                                transforms.Normalize([0.5], [0.5])])
# Download and load the training data
trainset = datasets.FashionMNIST('F_MNIST_data/', download=True, train=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=128, shuffle=True)

# Download and load the test data
testset = datasets.FashionMNIST('F_MNIST_data/', download=True, train=False, transform=transform)
n_test_set = len(testset)
n_val_set = int(n_test_set/2)
n_test_set = n_test_set - n_val_set

print("n_val_set", n_val_set)
print("n_test_set", n_test_set)

# Randomly split the test_set into test_set and val_set
testset, valset = torch.utils.data.random_split(testset, [n_test_set, n_val_set])
testloader = torch.utils.data.DataLoader(testset, batch_size=128, shuffle=True)
valloader = torch.utils.data.DataLoader(valset, batch_size=128, shuffle=True)<jupyter_output>n_val_set 5000
n_test_set 5000<jupyter_text>Here we can see one of the images.<jupyter_code>image, label = next(iter(trainloader))
print("image shape:", image.shape)
print("label shape:", label.shape)
helper.imshow(image[0,:]);<jupyter_output>image shape: torch.Size([128, 1, 28, 28])
label shape: torch.Size([128])<jupyter_text>With the data loaded, it's time to import the necessary packages.<jupyter_code>%matplotlib inline
%config InlineBackend.figure_format = 'retina'

import matplotlib.pyplot as plt
import numpy as np
import time

import torch
from torch import nn
from torch import optim
import torch.nn.functional as F
from torchvision import datasets, transforms

import helper<jupyter_output><empty_output><jupyter_text>Building the networkHere you should define your network. As with MNIST, each image is 28x28 which is a total of 784 pixels, and there are 10 classes. You should include at least one hidden layer. We suggest you use ReLU activations for the layers and to return the logits from the forward pass. It's up to you how many layers you add and the size of those layers.<jupyter_code># Define network architecture
class ResidualBlock(nn.Module):
    def __init__(self, in_channels, out_channels, stride=1):
        super(ResidualBlock, self).__init__()
        
        # Conv Layer 1
        self.conv1 = nn.Conv2d(kernel_size=(3, 3), in_channels=in_channels, out_channels=out_channels, 
                               stride=stride, padding=1, bias=False)
        self.batch_norm1 = nn.BatchNorm2d(out_channels)
        
        # Conv Layer 2
        self.conv2 = nn.Conv2d(kernel_size=(3, 3), in_channels=out_channels, out_channels=out_channels, 
                               stride=1, padding=1, bias=False)
        self.batch_norm2 = nn.BatchNorm2d(out_channels)
        
        # Shortcut connection to downsample residual
        # In case the output dimensions of the residual block is not the same 
        # as it's input, have a convolutional layer downsample the layer 
        # being bought forward by approporate striding and filters
        self.shortcut = nn.Sequential()
        if in_channels != out_channels and stride != 1:
            self.shortcut = nn.Sequential(
                nn.Conv2d(kernel_size=(1, 1), in_channels=in_channels, out_channels=out_channels, 
                          stride=stride, bias=False),
                nn.BatchNorm2d(out_channels)
            )
    
    def forward(self, x):
        out = nn.ReLU()(self.batch_norm1(self.conv1(x)))
        out = self.batch_norm2(self.conv2(out))
        out += self.shortcut(x)
        out = nn.ReLU()(out)
        return out
            
class ResNet(nn.Module):
    def __init__(self, num_classes=10, input_channels=3):
        super(ResNet, self).__init__()
        
        self.conv1 = nn.Conv2d(
            kernel_size=(3, 3), in_channels=input_channels, out_channels=64, stride=1, padding=1, bias=False)
        self.batch_norm1 = nn.BatchNorm2d(64)
        
        self.block1 = self._create_block(64, 64, stride=1)
        self.block2 = self._create_block(64, 128, stride=2)
#         self.block3 = self._create_block(128, 256, stride=2)
#         self.block4 = self._create_block(256, 512, stride=2)
        self.linear = nn.Linear(1152, num_classes)
        
    def _create_block(self, in_channels, out_channels, stride):
        return nn.Sequential(ResidualBlock(in_channels, out_channels, stride),
                             ResidualBlock(out_channels, out_channels, 1))
    
    def forward(self, x):
        out = nn.ReLU()(self.batch_norm1(self.conv1(x)))
        out = self.block1(out)
        out = self.block2(out)
#         out = self.block3(out)
#         out = self.block4(out)
        out = nn.AvgPool2d(4)(out)
        out = out.view(out.size(0), -1)
        out = self.linear(out)
        return out<jupyter_output><empty_output><jupyter_text>Train the networkNow you should create your network and train it. First you'll want to define [the criterion](http://pytorch.org/docs/master/nn.htmlloss-functions) ( something like `nn.CrossEntropyLoss`) and [the optimizer](http://pytorch.org/docs/master/optim.html) (typically `optim.SGD` or `optim.Adam`).Then write the training code. Remember the training pass is a fairly straightforward process:* Make a forward pass through the network to get the logits * Use the logits to calculate the loss* Perform a backward pass through the network with `loss.backward()` to calculate the gradients* Take a step with the optimizer to update the weightsBy adjusting the hyperparameters (hidden units, learning rate, etc), you should be able to get the training loss below 0.4.<jupyter_code># Create the network, define the criterion and optimizer
model = ResNet(input_channels=1)
loss_func = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=0.01)
# Train the network

model.train()
for epoch in range(1):
    
    loss_list=[]
    for batch_idx, (images, labels) in enumerate(trainloader):
        
        optimizer.zero_grad()
        outputs = model(images)
        loss = loss_func(outputs, labels)
        loss.backward()
        optimizer.step()
        
        loss_list.append(loss.item())
        
        if batch_idx % 10 == 0:
            model.eval()
            total=0
            correct = 0
            with torch.no_grad():
                for _, (images, labels) in enumerate(valloader):
                    outputs_val = model(images)
                    _, predicted = torch.max(outputs_val.data, 1)
                    total += labels.size(0)
                    correct += predicted.eq(labels.data).sum()
                    
                acc=100.*correct/total
            print("epoch:{0}, batch index:{1}, loss:{2}, acc:{3}"
                  .format(epoch, batch_idx, np.mean(loss_list), acc))
            model.train()
                
# Test out your network!

dataiter = iter(testloader)
images, labels = dataiter.next()
img = images[0]
print("img shape:", img.shape)
# Convert img to the input format in which the first dimension is batch size
img = img.resize_(1, img.shape[0], img.shape[1], img.shape[2])

# Calculate the class probabilities (softmax) for img
with torch.no_grad():
    output = model.forward(img)

# ps = torch.exp(output)
ps = F.softmax(output, dim=1)
print(ps.shape)

# Plot the image and probabilities
helper.view_classify(img.resize_(1, 28, 28), ps, version='Fashion')<jupyter_output>img shape: torch.Size([1, 28, 28])
torch.Size([1, 10])
{'fraction_non_alphanumeric': 0.09838673028857078, 'fraction_numerical': 0.021813224267211998, 'mean_word_length': 3.2691561590688654, 'pattern_counts': {'":': 0, '<': 18, '<?xml version=': 0, '>': 18, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '158285', 'license': 'MIT', 'n_tokens_mistral': 2789, 'n_tokens_neox': 2619, 'n_words': 828, 'path': 'Part 05 - Fashion-MNIST Exercise.ipynb', 'repo_name': 'yankang18/pytorch_deep_learning', 'size': '8802'}