<jupyter_start><jupyter_text>Tiny MNIST ChallengeAuthor: <NAME>Customized by: <NAME> University of TechnologyMethods of Artificial IntelligenceChallenge: https://www.kaggle.com/c/msi-tinymnist2020/overviewColab: https://colab.research.google.com/drive/11MZBLRatBPMW6PeqZMhVhwg-I43QZcFL Hand-written digits recognition 2020![alt text](https://upload.wikimedia.org/wikipedia/commons/thumb/2/27/MnistExamples.png/220px-MnistExamples.png) Introduction As a first step you have to get your kaggle *token*. Log into your kaggle account and go to *My Account* section. Click *Create new API token* and download it. Open you *kaggle.json*. You should see something like this: {`“username”:”YOUR-USERNAME”,”key”:”YOUR-VERY-LONG-KEY”}`Copy that and insert in the code below.<jupyter_code>import os
os.environ['KAGGLE_USERNAME'] = "xxxxxxxxxxxx" # username from the json file
os.environ['KAGGLE_KEY'] = "xxxxxxxxxxxxxxxx" # key from the json file<jupyter_output><empty_output><jupyter_text>Now we will install necessary libraries.*  kaggle API - useful for sending submissions  https://github.com/Kaggle/kaggle-api*   torch and torchvision - pytorch library, perfect for training neural networks https://pytorch.org/<jupyter_code>!pip install --upgrade kaggle
!pip install torch
!pip install torchvision<jupyter_output>Requirement already up-to-date: kaggle in /usr/local/lib/python3.6/dist-packages (1.5.6)
Requirement already satisfied, skipping upgrade: urllib3<1.25,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from kaggle) (1.24.3)
Requirement already satisfied, skipping upgrade: six>=1.10 in /usr/local/lib/python3.6/dist-packages (from kaggle) (1.12.0)
Requirement already satisfied, skipping upgrade: tqdm in /usr/local/lib/python3.6/dist-packages (from kaggle) (4.38.0)
Requirement already satisfied, skipping upgrade: python-slugify in /usr/local/lib/python3.6/dist-packages (from kaggle) (4.0.0)
Requirement already satisfied, skipping upgrade: python-dateutil in /usr/local/lib/python3.6/dist-packages (from kaggle) (2.8.1)
Requirement already satisfied, skipping upgrade: certifi in /usr/local/lib/python3.6/dist-packages (from kaggle) (2019.11.28)
Requirement already satisfied, skipping upgrade: requests in /usr/local/lib/python3.6/dist-packages (from kaggle) (2.21.0)
Requirement already satisf[...]<jupyter_text>Now, go to your kaggle account and download dataset. Upload it below.<jupyter_code>from google.colab import files
uploaded = files.upload()
!unzip -q TinyMNIST.zip #uzipping dataset<jupyter_output><empty_output><jupyter_text>Now we will seed everything so the results are reproducible<jupyter_code>import random
import numpy as np
import torch
def seed_everything(seed):
    random.seed(seed)
    os.environ['PYTHONHASHSEED'] = str(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed(seed)
    torch.backends.cudnn.deterministic = True
seed_everything(2020)<jupyter_output><empty_output><jupyter_text>Loading dataNow, we will preprocess and load training set<jupyter_code>NUM_EPOCHS = 40 # number of times which the entire dataset is passed throughout the model
BATCH_SIZE = 60 # the size of input data took for one iteration
lr = 1e-3 # size of step
import torch.nn as nn
import torchvision.datasets as dsets
import torchvision.transforms as transforms
from torch.autograd import Variable

train_dataset = dsets.ImageFolder(
        root='TinyMNIST/train',
        transform=transforms.Compose([transforms.Grayscale(num_output_channels=1),transforms.ToTensor()])
    )

train_set,valid_set =torch.utils.data.random_split(train_dataset, [3200, 800])

train_gen = torch.utils.data.DataLoader(dataset=train_set,
                                             batch_size = BATCH_SIZE,
                                             shuffle = True)

valid_gen = torch.utils.data.DataLoader(dataset = valid_set,
                                      batch_size = 800, 
                                      shuffle = False)


print('Loaded ', len(train_set), 'training images and ', len(valid_set), ' validation images.')
print('Classes detected: ', train_dataset.classes)<jupyter_output>Loaded  3200 training images and  800  validation images.
Classes detected:  ['0', '1', '2', '3', '4', '5', '6', '7', '8', '9']<jupyter_text>ArchitectureDefining simple net - fully connected network with 2 layers.<jupyter_code>class ConvNetwork(nn.Module):
  def __init__(self):
    super(ConvNetwork,self).__init__()

    self.relu = nn.ReLU()
    self.conv1 = nn.Conv2d(1, 20,kernel_size= 5)          # (N, 1, 28, 28) -> (N,  x, 24, 24)
    self.conv2 = nn.Conv2d(20, 40, kernel_size=5)         # (N, 1, 24, 24) -> (N,  x, 20, 20)
    self.conv3 = nn.Conv2d(40, 80, kernel_size=4)         # (N, 1, 21, 21) -> (N,  x, 17, 17)
    self.conv4 = nn.Conv2d(80, 160, kernel_size=3)        # (N, 1, 19, 19) -> (N,  x, 15, 15)
    self.conv5 = nn.Conv2d(160, 320, kernel_size=2)       # (N, 1, 17, 17) -> (N,  x, 14, 14)

    self.avg = nn.AvgPool2d(2, stride=2)      # (N, x, 16, 16) -> (N,  x, 7, 7)
    self.batch = nn.BatchNorm2d(320)
    self.fc = nn.Linear(320*7*7, 2000)
    self.fc1 = nn.Linear(2000, 10)
  def forward(self, x):
      x = self.relu(self.conv1(x))
      x = self.relu(self.conv2(x))
      nn.Dropout2d(0.2)
      x = self.relu(self.conv3(x))
      x = self.relu(self.conv4(x))
      nn.Dropout2d(0.2)
      x = self.relu(self.conv5(x))
      x = self.relu(self.avg(x))
      x = self.relu(self.batch(x))
      x = x.view(-1, 320*7*7)
      x = self.fc(x)
      x = self.fc1(x)
      return x
class ConvNetwork(nn.Module):
  def __init__(self):
    super(ConvNetwork,self).__init__()

    
    self.conv1 = nn.Conv2d(1, 20, kernel_size=6) # convolutional layer
    self.relu = nn.ReLU() # activation function: relu
    self.conv2 = nn.Conv2d(20, 40, kernel_size=5) # second convolutional layer
    self.conv3 = nn.Conv2d(40, 80, kernel_size=5) # third convolutional layer
    self.conv4 = nn.Conv2d(80, 160, kernel_size=4) # 4-th convolutional layer
    self.conv5 = nn.Conv2d(160, 320, kernel_size=4) # 5-th convolutional layer
    self.conv6 = nn.Conv2d(320, 640, kernel_size=4)
    self.fc1 = nn.Linear(320*9*9, 10000)
    self.fc2 = nn.Linear(10000, 2000)
    self.fc = nn.Linear(640*6*6, 10) # output layer
    self.batch = nn.BatchNorm2d(320)
    self.batch1 = nn.BatchNorm2d(640)
#    self.softplus = nn.Softplus()
#    self.conv2_drop = nn.Dropout2d()


  # definition how the signal goes through the network
  def forward(self,x):
    x = self.conv1(x)
    x = self.relu(x)
    x = self.conv2(x)
    x = self.relu(x)
    x = self.conv3(x)
    x = self.relu(x)
    x = self.conv4(x)
    x = self.relu(x)
    x = self.conv5(x)
    x = self.relu(x)
    x = self.conv6(x)
    x = self.relu(x)
    x = x.view(-1, 640*6*6)
    out = self.fc(x)
    return out

net = ConvNetwork()<jupyter_output><empty_output><jupyter_text>Define loss function<jupyter_code>loss_function = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(net.parameters(), lr=lr)
#optimizer = torch.optim.Adadelta(net.parameters(), lr=lr, rho=0.9, eps=1e-06, weight_decay=0)<jupyter_output><empty_output><jupyter_text>Training<jupyter_code>print('Start training') 
for epoch in range(NUM_EPOCHS):
  

  for i,(images,labels) in enumerate(train_gen):

      optimizer.zero_grad()
      outputs = net(images)

      loss = loss_function(outputs, labels)
      loss.backward()
      optimizer.step()

  print('\repoch: ',epoch + 1,"/ ",NUM_EPOCHS, loss,end="")
  if loss < 1e-8 and loss > 0 and epoch>10:
   break
 
from google.colab import drive
drive.mount('/content/drive')<jupyter_output>Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount("/content/drive", force_remount=True).<jupyter_text>Testing neural network<jupyter_code>import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sn
from sklearn.metrics import confusion_matrix

def plot_confusion_matrix(valid_gen):
  confusion_matrix = torch.zeros(10, 10)
  with torch.no_grad():
      for i, (images,labels) in enumerate(valid_gen):
          outputs = net(images)
          _, preds = torch.max(outputs, 1)
         
          for t, p in zip(labels.view(-1), preds.view(-1)):
                  confusion_matrix[t.long(), p.long()] += 1

  df_cm = pd.DataFrame(confusion_matrix.cpu().detach().numpy()/len(valid_gen.dataset)*10, range(10), range(10))
  plt.figure(figsize=(10,7))
  sn.set(font_scale=1.4) # for label size
  sn.heatmap(df_cm, annot=True, annot_kws={"size": 14}) # font size
  plt.show()

plot_confusion_matrix(valid_gen)<jupyter_output><empty_output><jupyter_text>Let's generate outputs for test data!<jupyter_code>def predict(test_gen):
  with torch.no_grad():
  
      for data, labels in test_gen:
        print(data.size())
        outputs = net(data)
        _, preds = torch.max(outputs, 1)
  return preds

test_dataset = dsets.ImageFolder(
        root='TinyMNIST/test',
        transform=transforms.Compose([transforms.Grayscale(num_output_channels=1),transforms.ToTensor()]),
    )

test_gen = torch.utils.data.DataLoader(dataset=test_dataset,
                                             batch_size = 600,
                                             shuffle = False)
preds = predict(test_gen)<jupyter_output>torch.Size([600, 1, 28, 28])<jupyter_text>Generate submission fileNow, finally we can generate submission file with our test data. We will define a new function because we might want to use it later.<jupyter_code>def generate_submission(predictions,test_dataset):
  ids = []
  labels = []
  for predictions,sample in zip(predictions,test_dataset.imgs):
    ids.append(predictions)
    sample=list(sample)[0]
    labels.append(int(sample.split("_",1)[1].split(".",1)[0]))
  ids_s,labels_s = zip(*sorted(zip(labels, ids)))
  df = pd.DataFrame()
  df['id']=ids_s
  df['category']=labels_s
  df.set_index('id',inplace = True)
  print(df.head())
  df.to_csv('test_submission.csv', header=True)

generate_submission(preds,test_dataset)<jupyter_output>category
id          
0          1
1          0
2          1
3          4
4          0<jupyter_text>Send submission using kaggle API<jupyter_code>!kaggle competitions submit -c msi-tinymnist2020 -f test_submission.csv -m "Jupyter notebook test submission"<jupyter_output>Warning: Looks like you're using an outdated API Version, please consider updating (server 1.5.6 / client 1.5.4)
100% 3.42k/3.42k [00:01<00:00, 3.00kB/s]
Successfully submitted to MSI Tiny MNIST Challenge 2020
{'fraction_non_alphanumeric': 0.12905991046766357, 'fraction_numerical': 0.042004000380988664, 'mean_word_length': 3.6854082998661313, 'pattern_counts': {'":': 1, '<': 48, '<?xml version=': 0, '>': 55, 'https://': 5, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 13, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '77124', 'license': 'MIT', 'n_tokens_mistral': 3990, 'n_tokens_neox': 3670, 'n_words': 841, 'path': 'notebooks/7_My_simple_CNN.ipynb', 'repo_name': 'Kamil-Och/tiny-mnist-challenge', 'size': '10527'}