<jupyter_start><jupyter_text>2D Nuclear Segmentation with Mask-RCNNAdapted from https://github.com/fizyr/keras-maskrcnn.<jupyter_code>import os
import errno

import numpy as np
import tensorflow as tf

import deepcell
import deepcell_retinamask<jupyter_output><empty_output><jupyter_text>Load the data Download the data from `deepcell.datasets``deepcell.datasets` provides access to a set of annotated live-cell imaging datasets which can be used for training cell segmentation and tracking models.All dataset objects share the `load_data()` method, which allows the user to specify the name of the file (`path`), the fraction of data reserved for testing (`test_size`) and a `seed` which is used to generate the random train-test split.Metadata associated with the dataset can be accessed through the `metadata` attribute.<jupyter_code># Download the data (saves to ~/.keras/datasets)
filename = 'HeLa_S3.npz'
test_size = 0.1 # % of data saved as test
seed = 0 # seed for random train-test split

(X_train, y_train), (X_test, y_test) = deepcell.datasets.hela_s3.load_data(filename, test_size=test_size, seed=seed)

print('X.shape: {}\ny.shape: {}'.format(X_train.shape, y_train.shape))<jupyter_output>Downloading data from https://deepcell-data.s3.amazonaws.com/nuclei/HeLa_S3.npz
6370107392/6370099814 [==============================] - 218s 0us/step
X.shape: (6480, 216, 256, 1)
y.shape: (6480, 216, 256, 1)<jupyter_text>Set up filepath constants<jupyter_code># the path to the data file is currently required for `train_model_()` functions

# NOTE: Change DATA_DIR if you are not using `deepcell.datasets`
DATA_DIR = os.path.expanduser(os.path.join('~', '.keras', 'datasets'))

DATA_FILE = os.path.join(DATA_DIR, filename)

# confirm the data file is available
assert os.path.isfile(DATA_FILE)
# Set up other required filepaths

# If the data file is in a subdirectory, mirror it in MODEL_DIR and LOG_DIR
PREFIX = os.path.relpath(os.path.dirname(DATA_FILE), DATA_DIR)

ROOT_DIR = '/data'  # TODO: Change this! Usually a mounted volume
MODEL_DIR = os.path.abspath(os.path.join(ROOT_DIR, 'models', PREFIX))
LOG_DIR = os.path.abspath(os.path.join(ROOT_DIR, 'logs', PREFIX))

# create directories if they do not exist
for d in (MODEL_DIR, LOG_DIR):
    try:
        os.makedirs(d)
    except OSError as exc:  # Guard against race condition
        if exc.errno != errno.EEXIST:
            raise<jupyter_output><empty_output><jupyter_text>Create the RetinaMask ModelHere we instantiate a `RetinaMask` model from `deepcell.model_zoo` for object detection and masking.<jupyter_code>from deepcell_retinamask.utils.anchor_utils import get_anchor_parameters

# Generate backbone information from the data
backbone_levels, pyramid_levels, anchor_params = get_anchor_parameters(y_train.astype('int'))

num_classes = 1  # "object" is the only class
from deepcell_retinamask import model_zoo

backbone = 'resnet50'

# training model is `retinanet` while prediction model is `retinanet_bbox`
model = model_zoo.RetinaMask(
    backbone=backbone,
    input_shape=X_train.shape[1:],
    class_specific_filter=False,
    num_classes=num_classes,
    backbone_levels=backbone_levels,
    pyramid_levels=pyramid_levels,
    anchor_params=anchor_params)<jupyter_output>WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/util/deprecation.py:574: calling map_fn_v2 (from tensorflow.python.ops.map_fn) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Use fn_output_signature instead<jupyter_text>Prepare for training Set up training parameters.There are a number of tunable hyper parameters necessary for training deep learning models:**model_name**: Incorporated into any files generated during the training process.**n_epoch**: The number of complete passes through the training dataset.**lr**: The learning rate determines the speed at which the model learns. Specifically it controls the relative size of the updates to model values after each batch.**optimizer**: The TensorFlow module [tf.keras.optimizers](https://www.tensorflow.org/api_docs/python/tf/keras/optimizers) offers optimizers with a variety of algorithm implementations. DeepCell typically uses the Adam or the SGD optimizers.**lr_sched**: A learning rate scheduler allows the learning rate to adapt over the course of model training. Typically a larger learning rate is preferred during the start of the training process, while a small learning rate allows for fine-tuning during the end of training.**batch_size**: The batch size determines the number of samples that are processed before the model is updated. The value must be greater than one and less than or equal to the number of samples in the training dataset.**min_objects**: The minimum number of objects in every training image. Images with fewer than `min_objects` objects will be discarded.<jupyter_code>from tensorflow.keras.optimizers import SGD, Adam
from deepcell.utils.train_utils import rate_scheduler

model_name = '{}_retinamask'.format(backbone)

n_epoch = 3  # Number of training epochs
lr = 1e-5

optimizer = Adam(lr=lr, clipnorm=0.001)

lr_sched = rate_scheduler(lr=lr, decay=0.99)

batch_size = 8

min_objects = 3<jupyter_output><empty_output><jupyter_text>Create the DataGeneratorsThe `RetinaNetDataGenerator` outputs a raw image (`X`) with bounding boxes and classifications for every object in the labeled data (`y`).<jupyter_code>from deepcell_retinamask.image_generators import RetinaNetGenerator

# this will do preprocessing and realtime data augmentation
datagen = RetinaNetGenerator(
    rotation_range=180,
    zoom_range=(.8, 1.2),
    horizontal_flip=True,
    vertical_flip=True)

datagen_val = RetinaNetGenerator()
from deepcell_retinamask.utils.anchor_utils import guess_shapes

train_data = datagen.flow(
    {'X': X_train, 'y': y_train},
    seed=seed,
    include_bbox=True,
    include_masks=True,
    pyramid_levels=pyramid_levels,
    min_objects=min_objects,
    anchor_params=anchor_params,
    compute_shapes=guess_shapes,
    batch_size=batch_size)

val_data = datagen_val.flow(
    {'X': X_test, 'y': y_test},
    seed=seed,
    include_bbox=True,
    include_masks=True,
    pyramid_levels=pyramid_levels,
    min_objects=min_objects,
    anchor_params=anchor_params,
    compute_shapes=guess_shapes,
    batch_size=batch_size)<jupyter_output>WARNING:tensorflow:Removing 671 of 6480 images with fewer than 3 objects.
WARNING:tensorflow:Removing 77 of 720 images with fewer than 3 objects.<jupyter_text>Visualize the annotated data.<jupyter_code>import matplotlib.pyplot as plt
import numpy as np

from deepcell_retinamask.utils.plot_utils import draw_annotations

index = np.random.randint(0, len(X_train) + 1)

image = X_train[index]
mask = y_train[index]

image = 0.01 * np.tile(np.expand_dims(image[..., 0], axis=-1), (1, 1, 3))
draw = image.copy()

annotations = train_data.load_annotations(mask)

# draw the detections
draw_annotations(draw, annotations,
                 label_to_name=lambda x: 'cell')

fig, axes = plt.subplots(1, 2, figsize=(15, 15))
ax = axes.ravel()

ax[0].imshow(image, cmap='jet')
ax[0].set_title('Source Image')

ax[1].imshow(draw, cmap='jet')
ax[1].set_title('Annotated Image')

fig.tight_layout()
plt.show()<jupyter_output>Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers).
Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers).<jupyter_text>Compile the model with a loss functionEach output is trained with it's own loss function, regression loss is used for the bounding box task while a different loss is used for the categorical task. Both of these are encapsulated by the `RetinaNetLosses` class.<jupyter_code>from deepcell_retinamask.losses import RetinaNetLosses

retinanet_losses = RetinaNetLosses(
    sigma=3.0, alpha=0.25, gamma=2.0,
    mask_size=(28, 28), iou_threshold=0.5)

loss = {
    'regression': retinanet_losses.regress_loss,
    'classification': retinanet_losses.classification_loss,
    'masks': retinanet_losses.mask_loss
}
model.compile(loss=loss, optimizer=optimizer)<jupyter_output><empty_output><jupyter_text>Train the RetinaMask modelCreate a prediction model, convert the data generators to `Datasets`, and call `fit()` on the compiled model, along with a default set of callbacks.<jupyter_code>from tensorflow.python.data import Dataset

# Some versions of TensorFlow 2.x do not handle multiple
# outputs in X or y, but is compatible with a Dataset instead.

input_type_dict = {'input': tf.float32, 'boxes_input': tf.float32}
input_shape_dict = {
    'input': tuple([None] + list(train_data.x.shape[1:])),
    'boxes_input': (None, None, 4)
}

output_type_dict = {
    'regression': tf.float32,
    'classification': tf.float32,
    'masks': tf.float32,
}
output_shape_dict = {
    'regression': (None, None, None),
    'classification': (None, None, None),
    'masks': (None, None, None),
}

train_dataset = Dataset.from_generator(
    lambda: train_data,
    (input_type_dict, output_type_dict),
    output_shapes=(input_shape_dict, output_shape_dict))

val_dataset = Dataset.from_generator(
    lambda: val_data,
    (input_type_dict, output_type_dict),
    output_shapes=(input_shape_dict, output_shape_dict))
from deepcell_retinamask.model_zoo.retinamask import retinamask_bbox

prediction_model = retinamask_bbox(
    model,
    nms=True,
    anchor_params=anchor_params,
    class_specific_filter=False)
from deepcell.utils.train_utils import get_callbacks
from deepcell.utils.train_utils import count_gpus
from deepcell_retinamask.callbacks import RedirectModel, Evaluate

model_path = os.path.join(MODEL_DIR, '{}.h5'.format(model_name))

num_gpus = count_gpus()

print('Training on', num_gpus, 'GPUs.')

train_callbacks = get_callbacks(
    model_path,
    lr_sched=lr_sched,
    save_weights_only=num_gpus >= 2,
    monitor='val_loss',
    verbose=1)

eval_callback = RedirectModel(
    Evaluate(val_data,
             iou_threshold=0.5,
             score_threshold=.01,
             max_detections=100,
             weighted_average=True),
    prediction_model)

loss_history = model.fit(
    train_dataset,
    steps_per_epoch=train_data.y.shape[0] // batch_size,
    epochs=n_epoch,
    validation_data=val_dataset,
    validation_steps=val_data.y.shape[0] // batch_size,
    callbacks=train_callbacks)<jupyter_output>Training on 1 GPUs.
Epoch 1/3<jupyter_text>Predict on test dataUse the trained model to predict on new data and post-process the results into a label mask.<jupyter_code># Set up the prediction model
from deepcell_retinamask.model_zoo.retinamask import retinamask_bbox

prediction_model = retinamask_bbox(model, anchor_params=anchor_params)
prediction_model.load_weights(os.path.join(MODEL_DIR, model_name + '.h5'), by_name=True)
import matplotlib.pyplot as plt
import os
import time

import numpy as np

from deepcell_retinamask.utils.plot_utils import draw_detections, draw_masks


index = np.random.randint(low=0, high=X_test.shape[0])
print('Image Number:', index)

image, mask = X_test[index:index + 1], y_test[index:index + 1]

boxes, scores, labels, masks = prediction_model.predict(image)[-4:]

image = 0.01 * np.tile(np.expand_dims(image[0, ..., 0], axis=-1), (1, 1, 3))
mask = np.squeeze(mask)

# copies to draw on
draw = image.copy()

# draw the masks
draw_masks(draw, boxes[0], scores[0], masks[0],
           score_threshold=0.5)

# draw the detections
draw_detections(draw, boxes[0], scores[0], labels[0],
                label_to_name=lambda x: 'cell',
                score_threshold=0.5,)

fig, axes = plt.subplots(ncols=3, nrows=1, figsize=(15, 15), sharex=True, sharey=True)
ax = axes.ravel()

ax[0].imshow(image, cmap='jet')
ax[0].set_title('Source Image')

ax[1].imshow(mask, cmap='jet')
ax[1].set_title('Labeled Data')

ax[2].imshow(draw, cmap='jet')
ax[2].set_title('Detections')

fig.tight_layout()
plt.show()<jupyter_output>Image Number: 267
{'fraction_non_alphanumeric': 0.10179940019993336, 'fraction_numerical': 0.01741086304565145, 'mean_word_length': 5.262389149713093, 'pattern_counts': {'":': 0, '<': 35, '<?xml version=': 0, '>': 36, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '228734', 'license': 'Apache-2.0', 'n_tokens_mistral': 4126, 'n_tokens_neox': 3901, 'n_words': 1086, 'path': 'notebooks/RetinaMask.ipynb', 'repo_name': 'vanvalenlab/deepcell-retinamask', 'size': '12004'}