<jupyter_start><jupyter_text>Kaggle Simpsons Dataset Cleansing We'll take a deep look into the [Kaggle Simpsons Dataset](https://www.kaggle.com/alexattia/the-simpsons-characters-dataset) and analize the changes that can be made to improve it. Requirements:- [pip requirements](../requirements.txt)- download the [dataset](https://www.kaggle.com/alexattia/the-simpsons-characters-dataset) (required kaggle account)  Imports<jupyter_code>import os
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>Paths<jupyter_code>trainset_path = '../the-simpsons-characters-dataset/simpsons_dataset/'<jupyter_output><empty_output><jupyter_text>Load labels<jupyter_code>labels = os.listdir(trainset_path)
labels.remove('simpsons_dataset')
len(labels)<jupyter_output><empty_output><jupyter_text>Dataset stats<jupyter_code># get stats from current labels
count = [len(os.listdir(trainset_path + l)) for l in labels]
dfcount = pd.DataFrame(count)
stats = dfcount.describe()

plt.scatter(dfcount.index.tolist(), dfcount)
plt.show()

stats<jupyter_output><empty_output><jupyter_text>Stats analisisAs we can see, we have a very high standard deviation in terms of *image count per label*, this will produce bad results with our data (i.e overfitting on a class).In order to clean our dataset and make it more fit for our model, we need to reduce the labels that have too much images, and the ones that don't have enough.  Remove extra labels First we need to look at the IQR (interquartile range): Q3 - Q1.With this information we can detect the anomalous labels, those that have below `Q1 - 1.5 * IQR` or above `Q3 + 1.5 * IQR`.Those that have below the lower bound, will be removed. And we will not use more images than the upper bound.<jupyter_code>q1, q3 = stats[0]['25%'], stats[0]['75%']
iqr = q3 - q1
lower_bound = q1 - (1.5 * iqr)
upper_bound = q3 + (1.5 * iqr)

lower_bound, upper_bound<jupyter_output><empty_output><jupyter_text>As we can see, our lower bound is negative. In order to mantain a simpler data, we will use as bounds our quantiles directly (Q1 and Q3).<jupyter_code>lower_bound = q1
upper_bound = q3<jupyter_output><empty_output><jupyter_text>Remove below & above<jupyter_code>ncount, nlabels = [], []
for c, l in zip(count, labels):
    if c < lower_bound:
        continue
    nlabels.append(l)
    ncount.append(upper_bound if c > upper_bound else c)
print(len(ncount), len(nlabels))<jupyter_output>31 31<jupyter_text>Get stats from the new labels<jupyter_code># get stats from current labels
ndfcount = pd.DataFrame(ncount)
nstats = ndfcount.describe()

f, a = plt.subplots(1, 2, figsize=(15, 5))
a[0].scatter(ndfcount.index.tolist(), ndfcount)
a[1].scatter(dfcount.index.tolist(), dfcount)
plt.show()

pd.concat([stats, nstats], axis=1)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10962284102925626, 'fraction_numerical': 0.014099400775467043, 'mean_word_length': 5.7894736842105265, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '403591', 'license': 'MIT', 'n_tokens_mistral': 1007, 'n_tokens_neox': 973, 'n_words': 287, 'path': 'notebooks/data_cleaning.ipynb', 'repo_name': 'rusito-23/simpsons-characters-class', 'size': '2837'}