<jupyter_start><jupyter_text>Multiple Objects Detection and ClassificationThe notebook illustrates how the SquiDS package can help to create and test multiple objects detection and classification model.**Please note that the goal of this notebook is not to create the perfect ML model, but to demonstrate how to use the functionality of the SquiDS package.**<jupyter_code>import sys
sys.path.insert(0, '..')

from PIL import ImageDraw, ImageFont
import tensorflow as tf
import numpy as np
from hungarian_loss import hungarian_loss

from squids import (
    create_dataset,
    create_tfrecords,
    explore_tfrecords,
    load_tfrecords,
    preprocess_image,
    Palette,
    Background
)

from helper import plot_images<jupyter_output><empty_output><jupyter_text>Preparing DatasetGenerate synthetic dataset and convert it to TFRecords.<jupyter_code>create_dataset(
    dataset_size=10000,
    image_palette=Palette.BINARY,
    image_background=Background.BLACK,
    random_state=365,
    image_capacity=2,
    verbose=True)
create_tfrecords(verbose=True)<jupyter_output>100%|██████████| 10000/10000 [00:02<00:00, 4215.51it/s]
100%|██████████| 2057/2057 [00:01<00:00, 1139.18it/s]
100%|██████████| 6947/6947 [00:10<00:00, 684.72it/s]
100%|██████████| 996/996 [00:01<00:00, 714.92it/s]<jupyter_text>Explore few samples from the generated dataset.<jupyter_code>image_ids, _ = explore_tfrecords(
    'dataset/synthetic-tfrecords/instances_train',
    return_artifacts=True
)

images  = [
    explore_tfrecords(
        'dataset/synthetic-tfrecords/instances_train',
        image_id,
        with_categories=True,
        with_bboxes=True,
        return_artifacts=True
    )[0]
    for image_id in image_ids[0:25]
]

plot_images(images, "Generated Images")<jupyter_output><empty_output><jupyter_text>Train Object Detection ModelLet's create generators for train and validation datasets and compute steps per epoch for each of them. Note:1. For this model we need to combine bounding boxes and categories one-hot encodings. We can do it by using the `BC` output schema.2. The number of detecting objects is 2.<jupyter_code>ds_train, train_steps_per_epoch = load_tfrecords(
    "dataset/synthetic-tfrecords/instances_train", 
    output_schema = "BC",
    num_detecting_objects=2,
)
print(
    "Training steps per epoch (for the default batch size 32):",
    train_steps_per_epoch
)

ds_val, val_steps_per_epoch = load_tfrecords(
    "dataset/synthetic-tfrecords/instances_val", 
    output_schema = "BC",
    num_detecting_objects=2,
)
print(
    "Validation steps per epoch (for the default batch size 32):",
    val_steps_per_epoch
)<jupyter_output>Training steps per epoch (for the default batch size 32): 55
Validation steps per epoch (for the default batch size 32): 17<jupyter_text>Let's define a simple convolution model.<jupyter_code>localizer = tf.keras.models.Sequential([
	tf.keras.layers.Dense(
		1024, 
		kernel_initializer='random_normal',
		bias_initializer='zeros',
		activation='relu',
        input_shape=(1024,)),
	tf.keras.layers.Dense(2 * 4),
	tf.keras.layers.Reshape((2, 4))
])

classifier = tf.keras.models.Sequential([
	tf.keras.layers.Dense(
		1024, 
		kernel_initializer='random_normal',
		bias_initializer='zeros',
		activation='relu',
        input_shape=(1024,)),
	tf.keras.layers.Dense(2 * 4),
	tf.keras.layers.Reshape((2, 4)),
	tf.keras.layers.Softmax(),
	tf.keras.layers.Lambda(lambda x: x * 10)
])

common = tf.keras.models.Sequential([
	tf.keras.layers.Conv2D(
			filters=32,
            kernel_size=(3, 3),
            padding='same',
            input_shape=(64, 64, 3)),
    tf.keras.layers.MaxPooling2D(
            pool_size=(2, 2)),
	tf.keras.layers.Conv2D(
			filters=64,
            kernel_size=(3, 3),
            padding='same',
            input_shape=(64, 64, 3)),
    tf.keras.layers.MaxPooling2D(
            pool_size=(2, 2)),
	tf.keras.layers.Conv2D(
			filters=128,
            kernel_size=(3, 3),
            padding='same',
            input_shape=(64, 64, 3)),
    tf.keras.layers.MaxPooling2D(
            pool_size=(2, 2)),
	tf.keras.layers.Conv2D(
			filters=256,
            kernel_size=(3, 3),
            padding='same',
            input_shape=(64, 64, 3)),
    tf.keras.layers.MaxPooling2D(
            pool_size=(2, 2)),
	tf.keras.layers.Flatten(),
	tf.keras.layers.Dense(
		1024, 
		kernel_initializer='random_normal',
		bias_initializer='zeros',
		activation='relu'),
])

model_input = common.input
x = common(model_input)
model_output = tf.keras.layers.Concatenate()(
	[localizer(x), classifier(x)])
model = tf.keras.models.Model(model_input, model_output)
model.compile(
    optimizer=tf.keras.optimizers.Adam(),
    loss=hungarian_loss,
)<jupyter_output><empty_output><jupyter_text>Now we are ready to perform the model training. Note that the best model weights are stored in the `best_weights.h5` file.<jupyter_code>checkpoint_callback = tf.keras.callbacks.ModelCheckpoint(
    "best_weights.h5",
    monitor='val_loss',
    mode="min",
    save_best_only=True,
    verbose=1)
model.fit(
    ds_train,
    steps_per_epoch = train_steps_per_epoch,
    epochs=50,
    validation_data=ds_val,
    validation_steps = val_steps_per_epoch,
    callbacks=checkpoint_callback,
    verbose=1
)<jupyter_output>Epoch 1/50<jupyter_text>Test Object Detection ModelLet's test the model on the unseen data.<jupyter_code>image_ids, _ = explore_tfrecords(
    'dataset/synthetic-tfrecords/instances_test',
    return_artifacts=True
)

images  = [
    explore_tfrecords(
        'dataset/synthetic-tfrecords/instances_test',
        image_id,
        return_artifacts=True
    )[0]
    for image_id in image_ids[0:25]
]

plot_images(images, "Test Images")<jupyter_output><empty_output><jupyter_text>Make model prediction and overlay results on the test images.Categories ID:- 0 - Ellipse;- 1 - Triangle;- 2 - Rectangle;<jupyter_code>preprocessed_images = preprocess_image(
    np.array(
        [np.array(image) for image in images]
    ) / 255.0
)

model.load_weights("best_weights.h5")
predictions = model.predict(preprocessed_images)

sample = 0
for image, prediction in zip(images, predictions):
    draw = ImageDraw.Draw(image)
    for detection in prediction:
        draw = ImageDraw.Draw(image)
        bbox = detection[:4].astype(int)
        category_id = np.argmax(detection[4:])
        if category_id > 0:
            x, y, w, h = bbox
            draw.rectangle(
                (x, y, x + w, y + h),
                outline="#00FF00"
            )
            if bbox[1] < 10:
                draw.text(
                    (x + w - 5, y + h),
                    str(category_id),
                    "#00FF00",
                    font=ImageFont.load_default(),
                )
            else:
                draw.text(
                    (x + w - 5, y - 10),
                    str(category_id),
                    "#00FF00",
                    font=ImageFont.load_default(),
                )
    draw.text(
        (5,5),
        str(sample),
        "#FF8000",
        font=ImageFont.load_default(),
    )
    sample += 1

plot_images(images, "Test Images with Predicted BBoxes and Categories")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11660188784008883, 'fraction_numerical': 0.036646307606885066, 'mean_word_length': 3.2357436801881247, 'pattern_counts': {'":': 0, '<': 35, '<?xml version=': 0, '>': 31, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '185130', 'license': 'MIT', 'n_tokens_mistral': 2651, 'n_tokens_neox': 2402, 'n_words': 510, 'path': 'notebooks/multi-object-detector.ipynb', 'repo_name': 'mmgalushka/squids', 'size': '7204'}