<jupyter_start><jupyter_text>Car damage dataset preprocessorThis notebook will prepare the car damage dataset for the Peltarion platform.Note: This notebook requires installation of Sidekick. To install the package within the notebook, run the following code:`import sys!{sys.executable} -m pip install git+https://github.com/Peltarion/sidekickegg=sidekick`For more information about Sidekick, see:https://github.com/Peltarion/sidekickThe raw dataset is available at:  https://storage.cloud.google.com/bucket-8732/car_damage/raw.zip---<jupyter_code>import functools
from glob import glob
import resource
import os

import pandas as pd
from PIL import Image
import sidekick
from sklearn.model_selection import train_test_split<jupyter_output><empty_output><jupyter_text>Set paths<jupyter_code># Path to the raw dataset (unzipped)
input_path = './raw'
# Path to the zip output
output_path = './preprocessed.zip'<jupyter_output><empty_output><jupyter_text>Get list of paths to all files<jupyter_code>images_rel_path = glob(input_path + '/*/*.jpg') + glob(input_path + '/*/*.png')
print("Images found: ", len(images_rel_path))<jupyter_output>Images found:  1538<jupyter_text>Create DataframeThe class column values are derived from the names of the subfolders in the `input_path`.The image column contains the relative path to the images in the subfolders.  Create image and class columns<jupyter_code>df = pd.DataFrame({'image': images_rel_path})
df['class'] = df['image'].apply(lambda path: os.path.basename(os.path.dirname(path)))
df.head()<jupyter_output><empty_output><jupyter_text>Filter imagesFilter out non-RGB images Create temporary ``image_mode`` column<jupyter_code>def get_mode(path):
    im = Image.open(path)
    im.close()
    return im.mode

df['image_mode'] = df['image'].apply(lambda path: get_mode(path))
df['image_mode'].value_counts()
df = df[df.image_mode =='RGB']
df['image_mode'].value_counts()<jupyter_output><empty_output><jupyter_text>Remove the temporary column<jupyter_code>df = df.drop(['image_mode'], axis=1)
df['class'].value_counts()<jupyter_output><empty_output><jupyter_text>Create subsets for training and validation<jupyter_code>def create_subsets(df, col='class', validation_size=0.20):
    train_data, validate_data = train_test_split(df, test_size=validation_size, random_state=42, stratify=df[[col]])
    train_data.insert(loc=2, column='subset', value='T')
    validate_data.insert(loc=2, column='subset', value='V')
    return train_data.append(validate_data, ignore_index=True)

df = create_subsets(df)
df['subset'].value_counts()<jupyter_output><empty_output><jupyter_text>Upsampling Upsampling (duplicating samples) can be used to prevent bias in an ubalanced dataset<jupyter_code>max_size = df[df['subset']=='T']['class'].value_counts().max()
lst = [df]
for class_index, group in df[df['subset']=='T'].groupby('class'):
    lst.append(group.sample(max_size-len(group), replace=True))
df = pd.concat(lst)

#df['class'].value_counts()
print('\nTraining:')
print(df[df['subset']=='T']['class'].value_counts())
print('\nValidation:')
print(df[df['subset']=='V']['class'].value_counts())<jupyter_output>Training:
glass_shatter     439
door_scratch      439
bumper_dent       439
door_dent         439
tail_lamp         439
unknown           439
head_lamp         439
bumper_scratch    439
Name: class, dtype: int64

Validation:
unknown           110
door_dent          39
door_scratch       31
tail_lamp          27
glass_shatter      27
head_lamp          27
bumper_dent        26
bumper_scratch     16
Name: class, dtype: int64<jupyter_text>Create dataset bundle<jupyter_code>df.head()
image_processor = functools.partial(sidekick.process_image, mode='crop_and_resize', size=(224, 224), file_format='jpeg')
sidekick.create_dataset(
    output_path,
    df,
    path_columns=['image'],
    preprocess={
        'image': image_processor
    }
)
# The duplicated images in the upsampled class will cause warnings<jupyter_output>/usr/local/Cellar/python/3.7.1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/zipfile.py:1470: UserWarning: Duplicate name: 'image/1068.jpeg'
  return self._open_to_write(zinfo, force_zip64=force_zip64)
/usr/local/Cellar/python/3.7.1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/zipfile.py:1470: UserWarning: Duplicate name: 'image/586.jpeg'
  return self._open_to_write(zinfo, force_zip64=force_zip64)
/usr/local/Cellar/python/3.7.1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/zipfile.py:1470: UserWarning: Duplicate name: 'image/531.jpeg'
  return self._open_to_write(zinfo, force_zip64=force_zip64)
/usr/local/Cellar/python/3.7.1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/zipfile.py:1470: UserWarning: Duplicate name: 'image/413.jpeg'
  return self._open_to_write(zinfo, force_zip64=force_zip64)
/usr/local/Cellar/python/3.7.1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/zipfile.py:1470: UserWarning: Duplicate name: 'image/1058.jpeg'
  return [...]
{'fraction_non_alphanumeric': 0.1299537316435325, 'fraction_numerical': 0.031180848923757797, 'mean_word_length': 5.829670329670329, 'pattern_counts': {'":': 0, '<': 34, '<?xml version=': 0, '>': 34, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '179224', 'license': 'MIT', 'n_tokens_mistral': 1798, 'n_tokens_neox': 1706, 'n_words': 329, 'path': 'car_damage_classification/car_damage_preprocess.ipynb', 'repo_name': 'johannapeltarion/community-code', 'size': '4971'}