<jupyter_start><jupyter_text>Counting Objects (hard mode) In this notebook we'll use [fastai](http://docs.fast.ai) to count the number of identical objects in pictures. I anticipate this will be harder than our previous work because the features of the objects are going to be identical in each image. Also, since we'll be creating our dataset with `matplotlib` our data is going to be coming from a completely different distribution than the one on which ResNet-34 was trained (ImageNet). We begin my setting up our notebook and importing the fastai libraries:<jupyter_code>%reload_ext autoreload
%autoreload 2
%matplotlib inline

from fastai.vision import *
from fastai.metrics import error_rate

import random

import matplotlib.pyplot as plt
import numpy as np<jupyter_output><empty_output><jupyter_text>Creating a datasetUnlike our previous examples, I was unable to find a suitable dataset for this task. We would like a dataset that contains 1-5 identical elements. Fortunately it should be possible to create such a dataset ourselves. We'll start by generating non-overlapping points and plotting them.<jupyter_code>import matplotlib.pyplot as plt

def createNonOverlappingPoints(numElements):
    x = np.zeros((numElements)) + 2  #Place the cirlces offscreen
    y = np.zeros((numElements)) + 2  #Place the circles offscreen
    
    for i in range(0, numElements):
        
        foundNewCircle = False
        
        #Loop until we find a non-overlapping position for our new circle
        while not foundNewCircle:
            randX = random.uniform(-1, 1)
            randY = random.uniform(-1, 1)

            distanceFromOtherCircles = np.sqrt(np.square(x - randX) + np.square(y - randY))

            # Ensure that this circle is far enough away from all others
            if np.all(distanceFromOtherCircles > 0.2):
                break

        x[i] = randX
        y[i] = randY

    return x, y

x, y = createNonOverlappingPoints(10)
plt.scatter(x,y, s=200)
plt.axes().set_aspect('equal', 'datalim')  # before `plt.show()`        
plt.show()<jupyter_output>/home/josh/.local/lib/python3.6/site-packages/matplotlib/figure.py:98: MatplotlibDeprecationWarning: 
Adding an axes using the same arguments as a previous axes currently reuses the earlier instance.  In a future version, a new instance will always be created and returned.  Meanwhile, this warning can be suppressed, and the future behavior ensured, by passing a unique label to each axes instance.
  "Adding an axes using the same arguments as a previous axes "<jupyter_text>Now that we can create points, let's build our dataset. We'll generate 1,000 images for each class and save each class in a folder with that classes name. This will take a couple minutes to run.<jupyter_code>import os
from tqdm import tnrange

#Iterate over our five classes
for i in tnrange(1, 6):
    #Create 1,000 images of this class
    for j in tnrange(1000):
        
        directory = 'data/counting/' + str(i) + '/'
        path = directory + str(j) + '.png'
        os.makedirs(directory, exist_ok=True)

        #Get points
        x, y = createNonOverlappingPoints(i)
        #Create plot
        plt.cla()
        axes = plt.gca()
        axes.set_xlim([-2,2])
        axes.set_ylim([-2, 2])
        plt.scatter(x,y, s=200)
        plt.axes().set_aspect('equal', 'datalim')  # before `plt.show()`       
        #Save to disk
        plt.savefig(path)<jupyter_output><empty_output><jupyter_text>With our dataset ready, we can load everything up into an ImageDataBunch and verify it all looks correct.<jupyter_code>path = 'data/counting'

np.random.seed(42)
data = ImageDataBunch.from_folder(path, train=".", valid_pct=0.2,
        ds_tfms=get_transforms(), size=224, num_workers=4).normalize(imagenet_stats)

data.classes
data.show_batch(rows=3, figsize=(7,8))<jupyter_output><empty_output><jupyter_text>With this dataset it's a lot easier to see the image transformations (rotations, zooms etc.) that have been made to our dataset.   Training a model<jupyter_code>learner = create_cnn(data, models.resnet34, metrics=error_rate)
learner.fit_one_cycle(3)<jupyter_output><empty_output><jupyter_text>Without any changes we're sitting at over 85% accuracy! This surprised me as I thought this task would be harder for our neural network. There are certain techniques that would make this task almost trivially solvable (simply count the number of blue pixels) but I didn't think our convolutional neural network would be able to pick up on these.Let's see how much further we can improve things.<jupyter_code>learner.save('stage-1')
learner.unfreeze()
learner.lr_find()
learner.recorder.plot()<jupyter_output><empty_output><jupyter_text>It looks like learning rates between `1e-6` and `1e-3` would work for this problem. We'll go somewhere in the middle with `1e-4` and run for a few more epochs.<jupyter_code>learner.fit_one_cycle(15, max_lr=slice(1e-4))<jupyter_output><empty_output><jupyter_text>Nice, we've pushed the accuracy up to ~99%! This surprised me to see as I thought this would be a more challenging task for a convolutional neural network. I had preconceptions about these networks that turned out to be incorrect. Originally I believed that convolutional networks translational invariance property meant that they could detect the presence of a feature, but not necessarily where it was located (due to the convolutional filters being dragged across the image). Let's take a look at the the top losses for this experiment.<jupyter_code>preds,y,losses = learner.get_preds(with_loss=True)
interp = ClassificationInterpretation(data, preds, y, losses)
interp.plot_top_losses(9, figsize=(7,7))<jupyter_output><empty_output><jupyter_text>Since we're only getting two incorrect, there doesn't seem to be much to learn from our top losses. We'll take a quick look at the confusion matrix as well.<jupyter_code>interp.plot_confusion_matrix(figsize=(12,12), dpi=60)<jupyter_output><empty_output><jupyter_text>So it looks like I was incorrect. This wasn't a particularly hard task for neural networks. It's seems likely that with even more samples and training time we might be able to reach 100% accuracy.  One last experiment There's still one burning question I have: Is the network actually counting? (I personally doubt it) Or is it simply learning to recognize things like whitespace between the circles that can only occur when a certain number of elements are present?I'm not sure exactly how best to test this but one thought I had was that as we add more circles, it seems like it would be harder to find patterns that uniquely identify a given count. Maybe if we put so many circles on the screen that the unique patterns become very hard to find. For example, instead of trying 1-5 circles, let's try counting images that only have 45-50 circles.<jupyter_code>import os
from tqdm import tnrange

#Iterate over our 50 classes
for i in tnrange(45, 51):
    #Create 1,000 images of this class
    for j in tnrange(1000):
        
        directory = 'data/counting/' + str(i) + '/'
        path = directory + str(j) + '.png'
        os.makedirs(directory, exist_ok=True)

        #Get points
        x, y = createNonOverlappingPoints(i)
        #Create plot
        plt.cla()
        axes = plt.gca()
        axes.set_xlim([-2,2])
        axes.set_ylim([-2, 2])
        plt.scatter(x,y, s=200)
        plt.axes().set_aspect('equal', 'datalim')  # before `plt.show()`       
        #Save to disk
        plt.savefig(path)
path = 'data/counting'
np.random.seed(42)
data = ImageDataBunch.from_folder(path, train=".", valid_pct=0.2,
        ds_tfms=get_transforms(), size=224, num_workers=4).normalize(imagenet_stats)
data.show_batch(rows=3, figsize=(7,8))<jupyter_output><empty_output><jupyter_text>Good luck finding a pattern in that!Let's see how well it does now.<jupyter_code>learner = create_cnn(data, models.resnet34, metrics=error_rate)
learner.fit_one_cycle(3)<jupyter_output><empty_output><jupyter_text>Finally, a 25% accuracy rate! Something a vanilla network can't actually do! Let's see how well we can co-erce it into learning on this dataset, though.<jupyter_code>learner.save('stage-1')
learner.unfreeze()
learner.lr_find()
learner.recorder.plot()<jupyter_output><empty_output><jupyter_text>Now there's a graph for you. No matter which learning rate we choose it doesn't look like any of them will be able to make progress on this problem. Let's double check that with a small learning rate like `1e-6`.<jupyter_code>learner.fit_one_cycle(15, max_lr=slice(1e-6))<jupyter_output><empty_output><jupyter_text>This makes more sense to me. There are no "features" in this image that would allow a network to look at it and instantly know how many circles are present. I suspect most humans can also not glance at one of these images and know whether or not there are 45 or 46 elements present. I suspect we would have to fall back to a different approach and manually count them out.  One last experiment Part II One last experiment I want to run is to answer the question of: Where does this "count-by-visual-features" approach begin to fail? For this we'll generate a larger dataset containing the number 1-20 and plot the confusion matrix.<jupyter_code>import os
from tqdm import tnrange

#Iterate over our 50 classes
for i in tnrange(1, 21):
    #Create 1,000 images of this class
    for j in tnrange(1000):
        
        directory = 'data/counting/' + str(i) + '/'
        path = directory + str(j) + '.png'
        os.makedirs(directory, exist_ok=True)

        #Get points
        x, y = createNonOverlappingPoints(i)
        #Create plot
        plt.cla()
        axes = plt.gca()
        axes.set_xlim([-2,2])
        axes.set_ylim([-2, 2])
        plt.scatter(x,y, s=200)
        plt.axes().set_aspect('equal', 'datalim')  # before `plt.show()`       
        #Save to disk
        plt.savefig(path)
path = 'data/counting'
np.random.seed(42)
data = ImageDataBunch.from_folder(path, train=".", valid_pct=0.2,
        ds_tfms=get_transforms(), size=224, num_workers=4).normalize(imagenet_stats)
learner = create_cnn(data, models.resnet34, metrics=error_rate)
learner.fit_one_cycle(15, max_lr=slice(1e-5))
preds,y,losses = learner.get_preds(with_loss=True)
interp = ClassificationInterpretation(data, preds, y, losses)
print(interp)
interp.
interp.plot_confusion_matrix(figsize=(12,12), dpi=60)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08659378596087457, 'fraction_numerical': 0.0174530111238972, 'mean_word_length': 4.079883097905504, 'pattern_counts': {'":': 0, '<': 56, '<?xml version=': 0, '>': 57, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '436256', 'license': 'Apache-2.0', 'n_tokens_mistral': 3318, 'n_tokens_neox': 3133, 'n_words': 1168, 'path': '2_Counting.ipynb', 'repo_name': 'JoshVarty/ImageClassification', 'size': '10428'}