<jupyter_start><jupyter_text>Dataset Generator (Object Detection)<jupyter_code>import conf
import helper<jupyter_output><empty_output><jupyter_text>Definitions  bounding box map (from GDT-HWD to AWS format) ```json{    "left": xmin    "top": ymin    "width": xmax - xmin    "height": yman - ymin}```  Classes ```json{    "none": 0    "red": 1    "yellow": 2    "white": 3    "blue": 4}```  Step by step transformations (concept) - Create necessary folders<jupyter_code>!rm -rf train train_annotation validation validation_annotation
!mkdir train train_annotation validation validation_annotation<jupyter_output><empty_output><jupyter_text>1. read all annotations ordered (**GDUT-HWD/Annotations**)<jupyter_code>import os

raw_annotation_path = conf.raw_annotation_path
annotations = os.listdir(raw_annotation_path)

annotations.sort()
jpgs = [x.replace('xml', 'jpg') for x in annotations]
jsons = [x.replace('jpg', 'json') for x in annotations]<jupyter_output><empty_output><jupyter_text>2. Split 80% to train e 20% to test- Where X are the jpgs and Y are the annotations<jupyter_code>from sklearn.model_selection import train_test_split
train_annotation, validation_annotation, train_jpgs, validation_jpgs = train_test_split(annotations,
                                                                                        jpgs,
                                                                                        test_size=0.2,
                                                                                        random_state=42)<jupyter_output><empty_output><jupyter_text>Disclaimer!You should update in conf.py the following variables:- num_training_samples<jupyter_code>num_training_samples = len(train_annotation)
print('HEY! num_training_samples to {} in conf.py'.format(num_training_samples))<jupyter_output>HEY! num_training_samples to 2540 in conf.py<jupyter_text>3. Annotation from XML to Json in the template.json format<jupyter_code>import xml.etree.ElementTree as ET<jupyter_output><empty_output><jupyter_text>3.1 Generate train_annotation<jupyter_code>def generate_annotation(path, annotation_list):
    for filename in annotation_list:
        if filename != '.ipynb_checkpoints':
            image_id = filename.split('.')[0]
            root = ET.parse(conf.raw_annotation_path + '/{}'.format(filename)).getroot()

            annotations = []
            categories = []

            for size in root.findall('size'):
                W = size.find('width').text
                H = size.find('height').text
                depth = size.find('depth').text



            for ob in root.findall('object'):
                name = ob.find('name').text
                bndbox = ob.find('bndbox')
                xmin = int(bndbox.find('xmin').text)
                ymin = int(bndbox.find('ymin').text)
                xmax = int(bndbox.find('xmax').text)
                ymax = int(bndbox.find('ymax').text)

                left = xmin
                top = ymin
                width = xmax - xmin
                height = ymax - ymin

                annotation = helper.generate_annotation(name, left, top, width, height)
                categorie =  helper.generate_categorie(name, conf.object_categories)

                annotations.append(annotation)
                categories.append(categorie)

            categories = list({c['class_id']:c for c in categories}.values())

            data = helper.generate_annotation_file_dict(W, H, depth, image_id, annotations, categories)
            helper.dict_to_json(path+'_annotation', image_id, data)
%time
generate_annotation('train', train_annotation)
%time
generate_annotation('validation', validation_annotation)<jupyter_output>CPU times: user 4 µs, sys: 0 ns, total: 4 µs
Wall time: 7.87 µs<jupyter_text>4. copy imagem from Its folder (train or validation)        - Template must be: "/.jpg"        - folder: train or validation        - id: image name<jupyter_code>for filename in train_annotation:
    im = filename.replace('xml', 'jpg')
    resp = ! cp GDUT-HWD/JPEGImages/$im train/
    

for filename in validation_annotation:
    im = filename.replace('xml', 'jpg')
    resp = ! cp GDUT-HWD/JPEGImages/$im validation/<jupyter_output>CPU times: user 2 µs, sys: 1 µs, total: 3 µs
Wall time: 5.01 µs<jupyter_text>5. Move All folders to S3<jupyter_code>resp = ! aws s3 cp train s3://hardhat-dataset-sagemaker-object-detector-solvimm/train --recursive
resp = ! aws s3 cp validation s3://hardhat-dataset-sagemaker-object-detector-solvimm/validation --recursive
resp = ! aws s3 cp train_annotation s3://hardhat-dataset-sagemaker-object-detector-solvimm/train_annotation --recursive
resp = ! aws s3 cp validation_annotation s3://hardhat-dataset-sagemaker-object-detector-solvimm/validation_annotation --recursive<jupyter_output>^C
{'fraction_non_alphanumeric': 0.1064227811265849, 'fraction_numerical': 0.009561421741841613, 'mean_word_length': 2.858861267040898, 'pattern_counts': {'":': 9, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 4}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 12, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '428911', 'license': 'Apache-2.0', 'n_tokens_mistral': 1484, 'n_tokens_neox': 1403, 'n_words': 328, 'path': '2.0-amazon-sagemaker/2.2-hardhat-object-detection/01-DatasetGenerator.ipynb', 'repo_name': 'obernardocosta/aws-ml-demos', 'size': '4811'}