<jupyter_start><jupyter_text>Dogs Breeds Dataset DescriptionThis notebook provides a simple data characterization of the training data provided in Dog Breeds Dataset.This is an important step in the ML process to undestand the data used to train our model. It's in here where we can observe some intrinsic data characteristics that guide our deicision for the rest of the training procedure.<jupyter_code># Imports come here
import os
from collections import Counter, OrderedDict

import numpy as np
import torch
from matplotlib import pyplot as plt
from skimage import io
from torch.utils.data import DataLoader
from torchvision.datasets import ImageFolder
from torchvision.transforms import (Compose, Normalize, RandomHorizontalFlip,
                                    RandomResizedCrop, RandomRotation,
                                    ToTensor, functional as F)
from torchvision.utils import make_grid
from tqdm import tqdm

%matplotlib inline
transforms = Compose([
    RandomResizedCrop(224),
    RandomHorizontalFlip(),
    RandomRotation(15),
    ToTensor(),        
    Normalize((0.485, 0.456, 0.406), (0.229, 0.224, 0.225)),
])
def sanitize(classname):
    return classname[10:].lower().replace('_', ' ').capitalize()
data = ImageFolder('./data/dogs/train', transform=transforms)
data.classes = [sanitize(classname) for classname in data.classes]
data.classes
class_to_idx_dict = {k: v for v, k in enumerate(data.classes)}
idx_to_class_dict = {v: k for k, v in class_to_idx_dict.items()}
def class_to_idx(classname):
    return class_to_idx_dict[classname]

def idx_to_class(idx):
    return idx_to_class_dict[idx]
data.target_transforms = Compose([
    idx_to_class,
    sanitize,
    class_to_idx
])<jupyter_output><empty_output><jupyter_text>Cheking class distributionsAn unbalanced dataset can become extremlly challenging to properly train and identify unbalacement early can be very helpful to save time trying to tune a ML model when the problem is in the data.<jupyter_code>breed_counts = OrderedDict(Counter(data.targets).most_common())

_, ax = plt.subplots(figsize=(30, 10))
plt.bar([idx_to_class_dict[breed] for breed in breed_counts.keys()], breed_counts.values())

for tick in ax.get_xticklabels():
    tick.set_rotation(90)

plt.show()
counts = np.array(list(breed_counts.values()))

print(f'Min:\t{np.min(counts)}')
print(f'Mean:\t{np.mean(counts)}')
print(f'Std:\t{np.std(counts)}')
print(f'Max:\t{np.max(counts)}')<jupyter_output>Min:	151
Mean:	173.45
Std:	22.307565981074674
Max:	255<jupyter_text>As we can observe, the **Maltese dog** and some other breeds have more instances than 1 standard deviation from the data's mean. At first this might not be a problem. However, it's good to keep an eyes during the training procedure to check if these classes are dominating the others. Verifying image dataNow that we know some structural characteristics from our dataset, let's analyse the image data itself.<jupyter_code>loader = DataLoader(data, shuffle=True, batch_size=16)
imgs, _ = iter(loader).next()
imgs.size()
def show(imgs):
    if not isinstance(imgs, list):
        imgs = [imgs]
    fix, axs = plt.subplots(ncols=len(imgs), squeeze=False)
    for i, img in enumerate(imgs):
        img = img.detach()
        img = F.to_pil_image(img)
        axs[0, i].imshow(np.asarray(img))
        axs[0, i].set(xticklabels=[], yticklabels=[], xticks=[], yticks=[])
grid = make_grid(imgs)
show(grid)<jupyter_output><empty_output><jupyter_text>As we can observe, some images are just zeroed tensors. Thus, in order to improve our dataset quality, let's remove these images from the folder to facilitate future image loadings.<jupyter_code>removed = 0
for root, dirs, files in os.walk('./data/dogs/'):
    if len(dirs) == 0:
        for f in tqdm(files):
            imgpath = os.path.join(root, f)
            img = io.imread(imgpath)
            if np.count_nonzero(img) == 0:
                os.remove(imgpath)
                removed += 1

print(f'\nRemoved {removed} invalid images.')
data = ImageFolder('./data/dogs/train', transform=transforms)
loader = DataLoader(data, shuffle=True, batch_size=16)
samples, _ = iter(loader).next()
grid = make_grid(samples)
show(grid)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10531286894923259, 'fraction_numerical': 0.0179456906729634, 'mean_word_length': 4.375634517766497, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 16, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '459660', 'license': 'MIT', 'n_tokens_mistral': 1413, 'n_tokens_neox': 1340, 'n_words': 409, 'path': 'dogs-breeds.ipynb', 'repo_name': 'gosiqueira/dog-breed-recognition', 'size': '4235'}