<jupyter_start><jupyter_text>Fruit Classifier - ColourWhat if we were building a model to classify fruits, say to decide if a particular fruit is good or rotten.That was the question that got asked in one of the 1:1 this week.I built this model to show how this could be done using a Convolutional Neural Network, not dissimilar to the one that we built for project one.The major difference is that Projet 1 is a ** regression exercise **, i.e. it returns 68 x,y pairs of where the keypoints are, whilst this exercise returns wheter a fruit is good or rotten (i.e if it belongs to a specific class).As we are dealing with two classes, this is a ** binary ** classifier  ModelWe will define the model directly in the notebook this time<jupyter_code># import the usual resources
import matplotlib.pyplot as plt
import numpy as np

# watch for any changes in model.py, if it changes, re-load it automatically
%load_ext autoreload
%autoreload 2
import torch
from torch.autograd import Variable
import torch.nn as nn
import torch.nn.functional as F
# can use the below import should you choose to initialize the weights of your Net
import torch.nn.init as I


class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()
        
        ## TODO: Define all the layers of this CNN, the only requirements are:
        ## 1. This network takes in a square (same width and height), 3-colour image as input
        ## 2. It ends with a linear layer that represents the keypoints
        ## it's suggested that you make this last layer output 136 values, 2 for each of the 68 keypoint (x, y) pairs
        
        # As an example, you've been given a convolutional layer, which you may (but don't have to) change:
        # 1 input image channel (grayscale), 32 output channels/feature maps, 5x5 square convolution kernel
        
        self.conv1 = nn.Conv2d(3, 32, 5)    #  224 x 224 # this time we left the colour channels in 
        self.conv1_bn = nn.BatchNorm2d(32)
        self.conv2 = nn.Conv2d(32, 64, 3)  
        self.conv2_bn = nn.BatchNorm2d(64)
        self.conv3 = nn.Conv2d(64, 128, 3)
        self.conv3_bn = nn.BatchNorm2d(128)
        
        self.maxPool = nn.MaxPool2d(2, 2)         

        self.fc3 = nn.Linear(128 * 26 * 26 , 10)
        self.fc4 = nn.Linear(10 , 1)
        
        # sigmoid returns a value between 0 -> 1
        self.sigmoid = nn.Sigmoid()
        
    def forward(self, x):

        ## x is the input image and, as an example, here you may choose to include a pool/conv step:

        x = self.maxPool(F.relu(self.conv1_bn(self.conv1(x)))) # 220 x 220 / 2 = 110 x 110
        x = self.maxPool(F.relu(self.conv2_bn(self.conv2(x)))) # 108 x 108 / 2 = 54 x 54
        x = self.maxPool(F.relu(self.conv3_bn(self.conv3(x)))) # 52  x 52  / 2 = 26 x 26
  
        x = F.relu(self.fc3(x.view(x.size(0), -1)))
        x = self.sigmoid(self.fc4(x))

        return x
## Model is pre-defined

import torch
import torch.nn as nn
import torch.nn.functional as F

# from models import Net

net = Net()
print(net)<jupyter_output>Net(
  (conv1): Conv2d(3, 32, kernel_size=(5, 5), stride=(1, 1))
  (conv1_bn): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (conv2): Conv2d(32, 64, kernel_size=(3, 3), stride=(1, 1))
  (conv2_bn): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (conv3): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1))
  (conv3_bn): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (maxPool): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  (fc3): Linear(in_features=86528, out_features=10, bias=True)
  (fc4): Linear(in_features=10, out_features=1, bias=True)
  (sigmoid): Sigmoid()
)<jupyter_text>Reloading last saved set of weightsI have already trained this model, and saved the weights in **saved_models/best_ever.pt**<jupyter_code>model_dir = 'saved_models/'
model_name = 'colour.pt'

# net.load_state_dict(torch.load(model_dir+model_name))
net.eval()<jupyter_output><empty_output><jupyter_text>Transform the dataset I have created a transformed dataset of images and classes.<jupyter_code># !conda install opencv -y<jupyter_output><empty_output><jupyter_text>A few helpersThe functions below are used to help extract data from the **/data** folder and identify the class that the image belogs to.I use the name of the folder itself as the name of the classThis is different from P1_Facial_Keypoints - as the training data in that exercise contains a set of keypoints - and in this exercise we have two folders for training: one with good fruits and one with rotten fruits<jupyter_code>import glob
import os

# extract class name from the class folder
def getFolderClass(str):
    parts = str.split('/')
    return parts[-1]

# extract class name from a file name
def getFileClass(str):
    parts = str.split('/')
    return parts[-2]

def extractClassNames(folder):
    class_folders = glob.glob(os.path.join(folder,'*'))
    class_names = [ getFolderClass(f) for f in class_folders ]
    
    newdict = {}
    translator = {}
    for i,c in enumerate(class_names):
        newdict[c] = i
        translator[i] = c
    
    return newdict,translator
     
## testing the helper for classes
print(extractClassNames('data/training'))<jupyter_output>({'bad_berries': 0, 'good_berries': 1}, {0: 'bad_berries', 1: 'good_berries'})<jupyter_text>Look at one of the images<jupyter_code># a quick look at one of the loaded images

import matplotlib.image as mpimg
import matplotlib.pyplot as plt
import random

files = glob.glob(os.path.join('data/training','*/*'))

image_file_name = files[random.randint(800,1000)]

image = mpimg.imread(image_file_name)
print(image.max())
plt.title(getFileClass(image_file_name))
plt.imshow(image)<jupyter_output>0.552941<jupyter_text>Defining the DatasetIt is similar to the original dataset, with the differences that:* It returns image and class* it retains the color channel<jupyter_code>import glob
import os
import torch
from torch.utils.data import Dataset, DataLoader
import numpy as np
import pandas as pd
import cv2


class FruitsDataset(Dataset):
    """Fruits"""

    def __init__(self, root_dir, transform=None):
        """
        Args:
            root_dir (string): Directory with all the images.
            transform (callable, optional): Optional transform to be applied
                on a sample.
        """
        self.root_dir = root_dir
        
        # all files that need to be loaded
        self.files = glob.glob(os.path.join(root_dir,'*/*'))
        
        # classes where those files are
        self.classes,self.translator = extractClassNames(root_dir)
        
        self.transform = transform

    def __len__(self):
        
        # TODO define lenght (number of files in set)
        return len(self.files)

    def __getitem__(self, idx):
        image_name = self.files[idx]
        
        image = mpimg.imread(image_name)
        
        # if image has an alpha color channel, get rid of it
        if(image.shape[2] == 4):
             image = image[:,:,0:3]
        
        img_class = np.array([self.classes[getFileClass(image_name)]]).astype('float') # read dict with class 
        sample = {'image': image, 'class': img_class}

        if self.transform:
            sample = self.transform(sample)

        return sample<jupyter_output><empty_output><jupyter_text>Transforms<jupyter_code># tranforms

class Normalize(object):
    """Convert a color image to grayscale and normalize the color range to [0,1]."""        

    def __call__(self, sample):
        image, img_class = sample['image'], sample['class']
        
        image_copy = np.copy(image)
        img_class_copy = np.copy(img_class)

        
        # scale color range from [0, 255] to [0, 1]
        # image_copy =  image_copy.astype(np.float32) / 256.00
        
        return {'image': image_copy, 'class': img_class}


class Rescale(object):
    """Rescale the image in a sample to a given size.

    Args:
        output_size (tuple or int): Desired output size. If tuple, output is
            matched to output_size. If int, smaller of image edges is matched
            to output_size keeping aspect ratio the same.
    """

    def __init__(self, output_size):
        assert isinstance(output_size, (int, tuple))
        self.output_size = output_size

    def __call__(self, sample):
        image, img_class = sample['image'], sample['class']

        h, w = image.shape[:2]
        if isinstance(self.output_size, int):
            if h > w:
                new_h, new_w = self.output_size * h / w, self.output_size
            else:
                new_h, new_w = self.output_size, self.output_size * w / h
        else:
            new_h, new_w = self.output_size

        new_h, new_w = int(new_h), int(new_w)

        img = cv2.resize(image, (new_w, new_h))
        
        return {'image': img, 'class': img_class}


class RandomCrop(object):
    """Crop randomly the image in a sample.

    Args:
        output_size (tuple or int): Desired output size. If int, square crop
            is made.
    """

    def __init__(self, output_size):
        assert isinstance(output_size, (int, tuple))
        if isinstance(output_size, int):
            self.output_size = (output_size, output_size)
        else:
            assert len(output_size) == 2
            self.output_size = output_size

    def __call__(self, sample):
        image, img_class = sample['image'], sample['class']

        h, w = image.shape[:2]
        new_h, new_w = self.output_size

        top = np.random.randint(0, h - new_h)
        left = np.random.randint(0, w - new_w)

        img = image[top: top + new_h,
                      left: left + new_w]


        return {'image': img, 'class': img_class}


class ToTensor(object):
    """Convert ndarrays in sample to Tensors."""

    def __call__(self, sample):
        image, img_class = sample['image'], sample['class']
         
        # if image has no grayscale color channel, add one
        if(len(image.shape) == 2):
            # add that third color dim
            image = image.reshape(image.shape[0], image.shape[1], 1)
            
        # swap color axis because
        # numpy image: H x W x C
        # torch image: C X H X W
        image = image.transpose((2, 0, 1))
        
        return {'image': torch.from_numpy(image),
                'class': torch.from_numpy(img_class)}<jupyter_output><empty_output><jupyter_text>Choosing transformsI am reusing the same transforms here<jupyter_code>from torch.utils.data import Dataset, DataLoader
from torchvision import transforms, utils


## TODO: define the data_transform using transforms.Compose([all tx's, . , .])
# order matters! i.e. rescaling should come before a smaller crop
data_transform = transforms.Compose([Rescale(250),
                                     RandomCrop(224),
                                     Normalize(),
                                     ToTensor()])



# testing that you've defined a transform
# assert(data_transform is not None), 'Define a data_transform'
# create the transformed dataset
transformed_dataset = FruitsDataset(root_dir='data/training/',
                                    transform=data_transform)


print('Number of images: ', len(transformed_dataset))

print(len(transformed_dataset[0]['class']))

# iterate through the transformed dataset and print some stats about the first few samples
for i in range(4):
    sample = transformed_dataset[i]
    print(i, sample['image'].size(), sample['class'].size())<jupyter_output>Number of images:  2215
1
0 torch.Size([3, 224, 224]) torch.Size([1])
1 torch.Size([3, 224, 224]) torch.Size([1])
2 torch.Size([3, 224, 224]) torch.Size([1])
3 torch.Size([3, 224, 224]) torch.Size([1])<jupyter_text>Batching and loading dataNext, having defined the transformed dataset, we can use PyTorch's DataLoader class to load the training data in batches of whatever size as well as to shuffle the data for training the model. You can read more about the parameters of the DataLoader, in [this documentation](http://pytorch.org/docs/master/data.html). Batch sizeDecide on a good batch size for training your model. Try both small and large batch sizes and note how the loss decreases as the model trains.**Note for Windows users**: Please change the `num_workers` to 0 or you may face some issues with your DataLoader failing.<jupyter_code># load training data in batches
batch_size = 10

train_loader = DataLoader(transformed_dataset, 
                          batch_size=batch_size,
                          shuffle=True, 
                          num_workers=0)<jupyter_output><empty_output><jupyter_text>Before trainingTake a look at how this model performs before it trains. As I have already loaded a pre-trainined model, it should work really well.As an exercise, you can skip the loading of pre-trainned weights, and see how the model performs. Load in the test datasetThe test dataset is one that this model has *not* seen before, meaning it has not trained with these images. We'll load in this test data and before and after training, see how your model performs on this set!To visualize this test data, we have to go through some un-transformation steps to turn our images into python images from tensors and to turn our keypoints back into a recognizable range.<jupyter_code># load in the test data, using the dataset class
# AND apply the data_transform you defined above

# create the test dataset
test_dataset = FruitsDataset(root_dir='data/test/',
                            transform=data_transform)


# load test data in batches
batch_size = 10

test_loader = DataLoader(test_dataset, 
                          batch_size=batch_size,
                          shuffle=True, 
                          num_workers=0)<jupyter_output><empty_output><jupyter_text>Apply the model on a test sampleTo test the model on a test sample of data, you have to follow these steps:1. Extract the image and ground truth keypoints from a sample2. Make sure the image is a FloatTensor, which the model expects.3. Forward pass the image through the net to get the predicted, output keypoints.This function test how the network performs on the first batch of test data. It returns the images, the transformed images, the predicted keypoints (produced by the model), and the ground truth keypoints.<jupyter_code># test the model on a batch of test images

def net_sample_output(sample_number=0,start_from=0):
    
    all_images = all_outputs = all_classes = torch.tensor([])
 
    # read and discard images until we reach the starting point
    for i in range(start_from):
        _ = test_loader
        
    # iterate through the test dataset
    for i, sample in enumerate(test_loader):
        
        # get sample data: images and ground truth keypoints
        images = sample['image']
        classes = sample['class']

        # convert images to FloatTensors
        images = images.type(torch.FloatTensor)
        classes = classes.type(torch.FloatTensor)
        

        # forward pass to get net output
        output_classes = net(images)
        
        all_images = torch.cat((all_images,images))
        all_outputs = torch.cat((all_outputs,output_classes))
        all_classes = torch.cat((all_classes,classes))
        
        
        # break after first image is tested
        if i == sample_number:
            return all_images, all_outputs, all_classes<jupyter_output><empty_output><jupyter_text>Debugging tipsIf you get a size or dimension error here, make sure that your network outputs the expected number of keypoints! Or if you get a Tensor type error, look into changing the above code that casts the data into float types: `images = images.type(torch.FloatTensor)`.<jupyter_code># call the above function
# returns: test images, test predicted keypoints, test ground truth keypoints
test_images, test_outputs, gt_pts = net_sample_output(start_from=5)

# print out the dimensions of the data to see if they make sense
print(test_images.data.size())
print(gt_pts.data.size())
print(gt_pts.size())<jupyter_output>torch.Size([10, 3, 224, 224])
torch.Size([10, 1])
torch.Size([10, 1])<jupyter_text>Un-transformationNext, you'll see a helper function. `visualize_output` that takes in a batch of images, predicted keypoints, and ground truth keypoints and displays a set of those images and their true/predicted keypoints.This function's main role is to take batches of image and keypoint data (the input and output of your CNN), and transform them into numpy images and un-normalized keypoints (x, y) for normal display. The un-transformation process turns keypoints and images into numpy arrays from Tensors *and* it undoes the keypoint normalization done in the Normalize() transform; it's assumed that you applied these transformations when you loaded your test data.<jupyter_code># a random number
i = random.randint(0,len(test_images) - 1)

image = test_images[i].data   # get the image from its wrapper
# image = image.numpy() * 256   # convert to numpy array from a Tensor
image = np.transpose(image, (1, 2, 0))   # transpose to go from torch to numpy image

print(image.max())

plt.title('{} -> {}'.format(round(gt_pts[i].data.item()),round(test_outputs[i].data.item())))
plt.imshow(np.squeeze(image),cmap='gray')<jupyter_output>tensor(0.6701)<jupyter_text>Let us visualise the test data<jupyter_code># visualize the output
# by default this shows a batch of 10 images
def visualize_output(test_images, test_outputs, gt_pts, batch_size=10):

    no_samples = 0 
    
    running_acc = 0.0
    
    to = test_outputs.round()
    
    for i,_ in enumerate(test_images):
        plt.figure(figsize=(len(test_images)* 4, 10))
        ax = plt.subplot(1, len(test_images), i+1)

        # un-transform the image data
        image = test_images[i].data   # get the image from it's wrapper
        # image = image.numpy() * 256  # convert to numpy array from a Tensor
        image = np.transpose(image, (1, 2, 0))   # transpose to go from torch to numpy image
        plt.imshow(np.squeeze(image),cmap='gray')
        plt.title('label {} -> inferred {} '.format(gt_pts[i,0],to[i,0]))
        plt.axis('on')
        

        if gt_pts[i,0].item() == to[i,0].item():
            running_acc += 1

        no_samples = i + 1
            
    plt.show()
    
    print('Accuracy: {}%'.format(running_acc * 100 / no_samples))
# call it
visualize_output(test_images,test_outputs, gt_pts )<jupyter_output><empty_output><jupyter_text>Training Loss functionTraining a network to predict keypoints is different than training a network to predict a class; instead of outputting a distribution of classes and using cross entropy loss, you may want to choose a loss function that is suited for regression, which directly compares a predicted value and target value. Read about the various kinds of loss functions (like MSE or L1/SmoothL1 loss) in [this documentation](http://pytorch.org/docs/master/_modules/torch/nn/modules/loss.html). Defining the loss and optimizationWe must choose a loss that works well with binary classificationWe have a little more freedom in selecting the optimizer<jupyter_code>import torch.optim as optim

learning_rate = 0.001
criterion = nn.BCELoss()

optimizer = optim.SGD(net.parameters(), lr =learning_rate  , momentum= 0.9)<jupyter_output><empty_output><jupyter_text>Training and Initial ObservationNow, you'll train on your batched training data from `train_loader` for a number of epochs. To quickly observe how your model is training and decide on whether or not you should modify it's structure or hyperparameters, you're encouraged to start off with just one or two epochs at first. As you train, note how your the model's loss behaves over time: does it decrease quickly at first and then slow down? Does it take a while to decrease in the first place? What happens if you change the batch size of your training data or modify your loss function? etc. Use these initial observations to make changes to your model and decide on the best architecture before you train for many epochs and create a final model.<jupyter_code># if you have a gpu, use it!
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

# a little helper function that will help us to visualize accuracy
def asList(arr):    
    return [a[0] for a in arr]<jupyter_output><empty_output><jupyter_text>Define the trainner<jupyter_code>def train_net(n_epochs):

    # prepare the net for training
    net.train()
 
    # this helps us debug
    err_debug = False  # set it to true to see each step of error calc

    # we will use this at the end
    _acc = []
    _loss = []

    for epoch in range(n_epochs):  # loop over the dataset multiple times
        
        running_loss = 0.0
        
        # Calculating accuracy
        running_acc = 0.0

        # train on batches of data, assumes you already have train_loader
        for batch_i, data in enumerate(train_loader):
            # get the input images and their corresponding labels
            images = data['image']
            img_class = data['class']

            # convert variables to floats for regression loss
            img_class = img_class.type(torch.FloatTensor)
            images = images.type(torch.FloatTensor)
            
            # CUDA ... 
            img_class = img_class.to(device)
            images   = images.to(device)
            
            # forward pass to get outputs
            output_class = net(images)

            # calculate the loss between predicted and target keypoints
            loss = criterion(output_class, img_class)
            
            running_acc += torch.eq(img_class, output_class.round()).sum()

            
            ##################################################################
            # Calculating error  ... START
            ##################################################################     
            if err_debug:
                # Values
                print( '\n\n\===> New batch: \nPoint: {} '.format(img_class.tolist())  )            
                print( 'Output: {} '.format(output_class.round()))             
                print( 'Equal: {}'.format(torch.eq(img_class, output_class.round()).tolist()))
                print( 'No of correct: {}'.format(torch.eq(img_class, output_class.round()).sum()))

    
            ##################################################################
            # Calculating error  ... END
            ##################################################################
            
            # zero the parameter (weight) gradients
            optimizer.zero_grad()
            
            # backward pass to calculate the weight gradients
            loss.backward()

            # update the weights
            optimizer.step()

            # print loss statistics
            running_loss += loss.item()
            
            if batch_i % 10 == 9:    # print every 10 batches
                print('Epoch: {}, Batch: {}, Avg. Loss: {} Avg. Accuracy: {}%'
                      .format(epoch + 1, batch_i+1, 
                              running_loss / (batch_size * 10.0) , 
                              running_acc  * 100.0 / (batch_size  * 10.0) ))
                
                # add to a list to return it at then end
                
                _acc.append(running_acc.item() / (batch_size  * 10.0) )
                _loss.append(running_loss  / (batch_size * 10) )

                running_loss = 0.0
                running_acc = 0.0
                
                
    print('Finished Training')
    
    # return this to graph if you need
    history = {'acc' : _acc, 'loss': _loss }
    return history<jupyter_output><empty_output><jupyter_text>Train it<jupyter_code># train your network
n_epochs = 1 # start small, and increase when you've decided on your model structure and hyperparams

history = train_net(n_epochs)
print(history['loss'])
plt.plot( history['acc'], color='skyblue')
plt.plot( history['loss'], color='green')
plt.ylim(0, 100)
plt.legend()<jupyter_output>No handles with labels found to put in legend.<jupyter_text>Test dataSee how your model performs on previously unseen, test data. We've already loaded and transformed this data, similar to the training data. Next, run your trained model on these images to see what kind of keypoints are produced. You should be able to see if your model is fitting each new face it sees, if the points are distributed randomly, or if the points have actually overfitted the training data and do not generalize.<jupyter_code># get a sample of test data again
test_images, test_outputs, gt_pts = net_sample_output(1,1)

print(test_images.data.size())
print(test_outputs.data.size())
print(gt_pts.size())
# you can use the same function as before, by un-commenting the line below:

visualize_output(test_images, test_outputs, gt_pts)<jupyter_output><empty_output><jupyter_text>Saving the modelOnce you've found a good model (or two), save your model so you can load it and use it later!<jupyter_code>model_dir = 'saved_models/'
model_name = 'threshold.pt'

# after training, save your model parameters in the dir 'saved_models'
torch.save(net.state_dict(), model_dir+model_name)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10038944180008655, 'fraction_numerical': 0.016325085559183352, 'mean_word_length': 2.9934024505183787, 'pattern_counts': {'":': 0, '<': 74, '<?xml version=': 0, '>': 79, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '296252', 'license': 'MIT', 'n_tokens_mistral': 7751, 'n_tokens_neox': 7247, 'n_words': 2613, 'path': 'Fruit_Classifier-Colour.ipynb', 'repo_name': 'RenSilvaAU/fruit-tl', 'size': '25421'}