<jupyter_start><jupyter_text>Object detection model To find coordinates of a circle center on the image with several geometric shapes.  * [Dataset](Dataset)* [Model architecture](Model-architecture)* [Selection of hyperparameters](Selection-of-hyperparameters)* [Training](Training)* [Inference](Inference)* [Model evaluation](Model-evaluation)* [Operating time](Operating-time)* [Criticism](criticism)* [Summary](Summary)* [Proposals for the improvement and further development](Proposals-for-the-improvement-and-further-development)<jupyter_code>import os
import sys
import numpy as np
import matplotlib.pyplot as plt
from sklearn.metrics import mean_squared_error

sys.path.insert(0, os.path.join("..", ".."))
from batchflow import Dataset, B, V, C, Pipeline
from batchflow.models.tf import ResNet18
from batchflow.research import Results

from src import ShapesBatch, show_samples, show_loss, show_research<jupyter_output><empty_output><jupyter_text>DatasetFor training and model performance assessment we generate images of shape 128x128 on which a circle and a square are randomly located.A size and a position of a figure are generated independently:- A Circle's diameter 2$r$ and a side of a square 2$r$ are randomly sampled from [10, 48].- A position of figure's center is randomly sampled from [$r$, 128 - $r$].A label data for every image is an array with circles center coordinates.Data is generated on each iteration of training and test pipelines.With respect to a number of iterations, there were generated 6000 images for training, 1000 images for testing. Create a dataset as a template for a future generation of 1000 images:<jupyter_code>imset = Dataset(1000, ShapesBatch)<jupyter_output><empty_output><jupyter_text>Examples of dataset items.Green dots on the images correspond to circles' centers from the labels data.<jupyter_code>NUM_IMAGES = 5
batch = imset.next_batch(NUM_IMAGES).sample_images(128, max_r=24)
show_samples(batch.images[:, :, :, 0], p_true=batch.labels)<jupyter_output><empty_output><jupyter_text>Model architectureWe use model ResNet18 from ```batchflow.models.tf```An architecture specification can be found at[https://analysiscenter.github.io/batchflow/api/batchflow.models.tf.resnet.html](https://analysiscenter.github.io/batchflow/api/batchflow.models.tf.resnet.html)Model config:<jupyter_code>model_config={'inputs/images/shape': B("image_shape"),
              'inputs/labels/shape': 2,
              'initial_block/inputs': 'images',
              'loss': 'mse',
              'optimizer': 'Adam',
             }<jupyter_output><empty_output><jupyter_text>Selection of hyperparametersWithin the framework of the chosen model and the standard set of parameters for the optimizer, it is of interest to study the influence of the size of the batch on the rate of convergence of the solution.As shown by [research](./parameters_estimation.ipynb) among the set [20, 50, 100], the size of the batch 100 is optimal. The graphs of the loss function and the metric are shown below, depending on the number of iterations for different batch sizes.<jupyter_code>df = Results(path="parameters_research").load(use_alias=True)

show_research(df, layout=['train/current_loss', 'test/metrics'],
              average_repetitions=True, log_scale=[True, False],
              rolling_window=[10, None])<jupyter_output><empty_output><jupyter_text>We can see that the size of the batch 100 shows a more stable convergence with better accuracy than the size of the batch of 20 or 50.  TrainingWe train our model using stochastic gradient descent. On each iteration we* generate mini-batch data* update the weights Algorith of weights update (optimizer) is Adam with standard hyperparameters:* ```learning_rate``` = 0.01* ```beta1``` = 0.9* ```beta2``` = 0.999* ```epsilon``` = 1e-08.Loss function - MSE (mean squared error).Dataset bypass:* ```batch_size``` = 100* ```n_epochs``` = 6.Training pipeline:<jupyter_code>train_template = (Pipeline()
                  .init_variable('loss_history', init_on_each_run=list)
                  .init_model('dynamic', ResNet18, 'resnet', config=model_config)
                  .sample_images(128, max_r=24)
                  .train_model('resnet', fetches='loss', images=B('images'), labels=B('labels'), 
                               save_to=V('loss_history', mode='a'))
                  .run_later(batch_size=100, n_epochs=6, drop_last=True, bar=True, workers=2, gpu=[1, 2]))<jupyter_output><empty_output><jupyter_text>Execute training with batch size 100<jupyter_code>train_pipeline = (train_template << imset).run()<jupyter_output>100%|██████████| 60/60 [00:39<00:00,  2.51it/s]<jupyter_text>The loss function vs a number of iterations:<jupyter_code>show_loss(train_pipeline.get_variable('loss_history'))<jupyter_output><empty_output><jupyter_text>InferencePredict pipeline:<jupyter_code>predict_template = (Pipeline()
                    .import_model('resnet', train_pipeline)
                    .init_variable('predictions', init_on_each_run=list)
                    .predict_model('resnet', fetches='predictions', images=B('images'),
                                   save_to=V('predictions', mode='e')))<jupyter_output><empty_output><jupyter_text>Precreate data for test:<jupyter_code>src_batch = (Dataset(NUM_IMAGES, ShapesBatch)
             .next_batch(NUM_IMAGES)
             .sample_images(128, circles=1, squares=1, max_r=24))

show_samples(src_batch.images[:, :, :, 0], src_batch.labels)<jupyter_output><empty_output><jupyter_text>Execute predict pipeline:<jupyter_code>predict_pipeline = ((predict_template << Dataset(NUM_IMAGES, ShapesBatch, preloaded=src_batch._data))
                    .run(NUM_IMAGES, shuffle=False, n_epochs=1, drop_last=False, bar=True))<jupyter_output>100%|██████████| 1/1 [00:00<00:00,  3.00it/s]<jupyter_text>Let's see the initial images with the true centers (green dots) and the predicted centers (red dots).<jupyter_code>y_pred = np.stack(predict_pipeline.get_variable('predictions'))

show_samples(src_batch.images[:, :, :, 0], src_batch.labels, y_pred)<jupyter_output><empty_output><jupyter_text>Model evaluationQuality metric, in this case, is Mean Squarred Error on the test set containing of 1000 images.Metric evaluation pipeline:<jupyter_code>test_template = (Pipeline()
                 .import_model('resnet', train_pipeline)
                 .init_variable('images', init_on_each_run=list)
                 .init_variable('predictions', init_on_each_run=list)
                 .init_variable('targets', init_on_each_run=list)
                 .sample_images(128, circles=C("circles"), squares=C("squares"), max_r=24)
                 .update(V('images',  mode='e'), B('images'))
                 .update(V('targets', mode='e'), B('labels'))
                 .predict_model('resnet', fetches='predictions', images=B('images'),
                                save_to=V('predictions', mode='e')))<jupyter_output><empty_output><jupyter_text>Execute test pipeline:<jupyter_code>test_pipeline = ((test_template << imset).set_config(dict(circles=1, squares=1))
                 .run(100, shuffle=False, n_epochs=1, drop_last=False, bar=True))<jupyter_output>100%|██████████| 10/10 [00:03<00:00,  3.09it/s]<jupyter_text>Compute metric:<jupyter_code>y_pred = np.stack(test_pipeline.get_variable('predictions'))
y_true = np.stack(test_pipeline.get_variable('targets'))
print("MSE: %.3f" % mean_squared_error(y_true, y_pred))<jupyter_output>MSE: 24.819<jupyter_text>*The obtained mse may vary for independent launches of model training. It is useful, along with a point estimate, to obtain the distribution of the metric over N independent runs of the training procedure.* Examples of test images with the true centers (green dots) and the predicted centers (red dots).<jupyter_code>img = np.array(test_pipeline.get_variable('images'))

show_samples(img[:NUM_IMAGES, :, :, 0], y_true[:NUM_IMAGES], y_pred[:NUM_IMAGES])<jupyter_output><empty_output><jupyter_text>Operating timeDevice: CPU i5.One train step on batch size 100 takes 10 seconds. Training time is 10 minutes (60 iterations).One image inference takes 26 ms, for batch size 1000 - 22 sec.  CriticismThe model's mean squared error is 25. Error decreases to 14 if there are no squares on the images.<jupyter_code>test_pipeline = ((test_template << imset).set_config(dict(circles=1, squares=0))
                 .run(100, shuffle=False, n_epochs=1, drop_last=False, bar=True))

y_pred = np.stack(test_pipeline.get_variable('predictions'))
y_true = np.stack(test_pipeline.get_variable('targets'))
print("MSE: %.3f" % mean_squared_error(y_true, y_pred))<jupyter_output>100%|██████████| 10/10 [00:02<00:00,  3.75it/s]<jupyter_text>Examples of test images with the true centers (green dots) and the predicted centers (red dots).<jupyter_code>img = np.array(test_pipeline.get_variable('images'))

show_samples(img[:NUM_IMAGES, :, :, 0], y_true[:NUM_IMAGES], y_pred[:NUM_IMAGES])<jupyter_output><empty_output><jupyter_text>Create an array of distances between the true and predicted center of the figures:<jupyter_code>errs = ((y_true - y_pred) ** 2).sum(axis=1)<jupyter_output><empty_output><jupyter_text>Construct a histogram of the distribution of errors. It behaves as expected:<jupyter_code>plt.hist(errs, bins=100)
plt.xlabel("Error"), plt.ylabel("Frequency")
plt.show()<jupyter_output><empty_output><jupyter_text>However, if the exact centers of the circles are plotted and the magnitude of the error is shown in color, the distribution is different from the expected uniform distribution. There is an area in which large errors are more common (red dots of darker color):<jupyter_code>plt.scatter(y_true[:, 0], y_true[:, 1], c=np.clip(errs, 0, 150), cmap="Reds", s=20)
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12167960768287699, 'fraction_numerical': 0.024826317940335105, 'mean_word_length': 5.172131147540983, 'pattern_counts': {'":': 0, '<': 92, '<?xml version=': 0, '>': 80, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '21', 'dir': 'jupyter-structured-clean-dedup', 'id': '320260', 'license': 'Apache-2.0', 'n_tokens_mistral': 3256, 'n_tokens_neox': 3008, 'n_words': 787, 'path': 'model_description/regression_model/model_description.ipynb', 'repo_name': 'gazprom-neft/standards', 'size': '9788'}