<jupyter_start><jupyter_text>Train UNet ModelIn this Notebook you will learn how to train your UNet architecture with Dataloop and PytorchUNet is an Encoder - Decoder architecture for creating segmentation maps<jupyter_code>import json
import torch
import datetime
import matplotlib as mpl
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import dtlpy as dl
from dtlpy.ml import train_utils
from dtlpy.ml.dataset_generators.torch_dataset_generator import DataGenerator
import tempfile
# clean up the GPU
import gc
gc.collect()
torch.cuda.empty_cache()<jupyter_output><empty_output><jupyter_text>Get the DataLoop entitieslets get the model and dataset entities from our dataloop platform<jupyter_code>model = dl.models.get('unet')  # This is the global model
# Data entities
project = dl.projects.get('shefi-contests', '50f0fc03-4d70-455d-b485-c78cca53f2be')
dataset = dl.datasets.get('carvana', '61b9bbc1e8ad454a9aa7d285')<jupyter_output><empty_output><jupyter_text>SnapshotNow we can create a new snapshot - we will add your name and data to the suffix to make the snapshot has a unique name<jupyter_code>whoami = dl.client_api.info()['user_email']
now = datetime.datetime.now()

# Create a new snapshot - personally and with currect datetime
snapshot_name = f"carvana-train-example-{whoami.split('@')[0]}-{now.isoformat(timespec='minutes')}"
snapshot = model.snapshots.create(
    snapshot_name=snapshot_name,
    dataset_id=dataset.id,
    description='train unet example',
    bucket=project.buckets.create(bucket_type=dl.BucketType.ITEM, model_name=model.name, snapshot_name=snapshot_name),
    tags=['example', 'notebook'],
    configuration={'id_to_label_map': {'1': 'car'},
                   'image_normalize_mu': 0, 'image_normalize_std': 1,
                   'input_shape': [640, 960], 'batch_size': 2,
                   'num_epochs': 2},
    project_id=project.id,
    labels=['car']
)<jupyter_output>2022-01-03 13:02:33.649 [ERROR]-[MainThread]-[v1.47.1]dtlpy.repositories.snapshots: Snapshot does not support 'unlocked dataset'. Please change 'carvana' to readonly<jupyter_text>Lets View the Model and Snapshot entitiesWe use the to_df in order to convert to a DataFrame and view it<jupyter_code>model.to_df()
snapshot.to_df()<jupyter_output><empty_output><jupyter_text>One last thing to make sure before we trainOur `adapter` train method expects the data to be organized as: train-validation-test  this can be created manually on small datasets using `train_utils.create_dataset_partition()`Our dataset is already prepared, we will just verify it<jupyter_code>train_items = dataset.get_partitions(partitions=dl.SnapshotPartitionType.TRAIN)
val_items = dataset.get_partitions(partitions=dl.SnapshotPartitionType.VALIDATION)
test_items = dataset.get_partitions(partitions=dl.SnapshotPartitionType.TEST)

print(f"Dataset {dataset.name} Data partition, TRAIN: {train_items.items_count}, VALIDATION {val_items.items_count}, TEST {test_items.items_count} ")<jupyter_output>Dataset carvana Data partition, TRAIN: 4070, VALIDATION 1018, TEST 0<jupyter_text>Finally we can start to trainWe initialize the adapter using the `build` method.The `Adapter` is the base class to connect between dataloop platform and our specific model  some method are inheritance from the base adapter and some are written specifically per modeleach architecture has it's own adapter which you can view it's raw code<jupyter_code>adapter = model.build()
adapter.load_from_snapshot(snapshot=snapshot)
# adapter._set_adapter_handler('DEBUG')
root_path, data_path, output_path = adapter.prepare_training()
adapter.train(data_path=data_path, output_path=output_path,)<jupyter_output>Loading Data Generator: 100%|██████████| 4070/4070 [01:01<00:00, 66.30it/s]
Loading Data Generator: 100%|██████████| 1018/1018 [00:19<00:00, 52.19it/s]
I: unet-adapter         13:06:15 [unet_adapter.py:233](train):: Network:
	3 input channels
	2 output channels (classes)
	Bilinear up-scaling
I: unet-adapter         13:06:15 [unet_adapter.py:237](train):: Model loaded from model.pth
I: unet-adapter         13:06:23 [unet_adapter.py:266](train):: Starting training:
            Epochs:          2
            Batch size:      2
            Learning rate:   1e-05
            Training size:   4070
            Validation size: 1018
            Device:          cuda:0
            Images shape:    [640, 960]
            Mixed Precision: False
            Output:          /home/shefi/.dataloop/training/61d2f458c1aaaa764d669897_20220103-130239/output
            TensorBoard:     /home/shefi/.dataloop/training/61d2f458c1aaaa764d669897_20220103-130239/output/tb
        
Epoch 1/2:   0%|          | [...]<jupyter_text>SAVINGThe current adapter now holds the best model fit for our data.In order to upload the weights and other configurations we need to save our snapshot.  We will use a temp dir - so we save all content to that dir and upload it (other option is to upload all the *`output_path`* which has more runtime files)<jupyter_code>temp_dir = tempfile.mkdtemp(prefix=snapshot.name, suffix=now.strftime('%F-%H%M%S'))
adapter.save_to_snapshot(local_path=temp_dir)<jupyter_output><empty_output><jupyter_text>USING THE MODEL - PREDICTIONWe will use the DataGenerator to view the image (this utility already connects with our dataloop item and annotations)<jupyter_code>datagen = DataGenerator(data_path=os.path.join(data_path, 'validation'),
                        dataset_entity=snapshot.dataset,
                        annotation_type=dl.AnnotationType.SEGMENTATION,
)

# example - get 1 entry and visualize it
datagen.visualize(20)<jupyter_output><empty_output><jupyter_text>Data ItemOur data generator returns Data Item dictionary  we can parse it to get the item and annotations<jupyter_code>data_item = datagen[20]
print(f"To get the item_id from the dataItem ({data_item.keys()}) object we can use the ann json")
ann_json = json.load(open(data_item['annotation_filepath'], 'r'))
item_id = ann_json['id']
item = dl.items.get(item_id=item_id)
predictions = adapter.predict_items(items=[item], with_upload=False)
item_predictions = predictions[0]
# we can ignore the label 0 which usually uses for background
predictions = adapter.predict_items(items=[item], with_upload=False, with_bg=False)
item_sematic_preds = predictions[0]
item_sematic_preds.print()
# we can create polygons instead of sematic segmetations
predictions = adapter.predict_items(items=[item], with_upload=False, to_poly=True)
item_polygons_preds = predictions[0]
item_polygons_preds.print()
annotated = item_polygons_preds.show(data_item['image'], thickness=5 )
plt.imshow(annotated)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09664871334530221, 'fraction_numerical': 0.03620586475164572, 'mean_word_length': 5.066243194192378, 'pattern_counts': {'":': 0, '<': 36, '<?xml version=': 0, '>': 34, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '433381', 'license': 'MIT', 'n_tokens_mistral': 2276, 'n_tokens_neox': 2110, 'n_words': 567, 'path': 'model_management/segmentation/training_instance_segmentation_unet.ipynb', 'repo_name': 'shiragit/dtlpy-documentation', 'size': '6684'}