<jupyter_start><jupyter_text>3D U-Net InferenceThis notebook shows how to perform inference on the trained model using both the Keras/TensorFlow model and the OpenVINO model. It also compares the outputs to show that OpenVINO is faster on the same hardware than Keras/TensorFlow and produces the same predictions.  Always run the OpenVINO setup variable script before using it for the first time.<jupyter_code>!source /opt/intel/openvino/bin/setupvars.sh<jupyter_output><empty_output><jupyter_text>If you haven't converted the model already, then use the OpenVINO model optimizer to convert from TensorFlow to OpenVINO.The file `tf_protobuf/3d_unet_decathlon.pb` should be automatically created at the end of `train.py` by using the best trained Keras HDF5 model file. (From TensorFlow 2.0 and onward, the saved files should all be TensorFlow protobuf (.pb))<jupyter_code>!python $INTEL_OPENVINO_DIR/deployment_tools/model_optimizer/mo_tf.py \
        --input_model tf_protobuf/3d_unet_decathlon.pb  \
        --input_shape [1,1,144,144,144]  \
        --output_dir openvino_models/FP32/ \
        --data_type FP32<jupyter_output><empty_output><jupyter_text>Import the libraries<jupyter_code>import sys
import os
import csv

import numpy as np
import logging as log
from time import time
from openvino.inference_engine import IENetwork, IECore

import tensorflow as tf
import keras as K

import nibabel as nib

from tqdm import tqdm

os.environ["TF_CPP_MIN_LOG_LEVEL"] = "2"  # Get rid of the AVX, SSE warnings<jupyter_output><empty_output><jupyter_text>Runtime arguments<jupyter_code>class args:
    number_iter = 5
    device = "CPU"
    stats = False
    plot=False
    csv_file = "test.csv"
    openvino_model = "./openvino_models/FP32/3d_unet_decathlon.xml"
    keras_model = "./saved_model/3d_unet_decathlon.hdf5"
    cpu_extension=""
    plugin_dir=""<jupyter_output><empty_output><jupyter_text>Dice score calculation<jupyter_code>def dice_score(pred, truth):
    """
    Sorensen Dice score
    Measure of the overlap between the prediction and ground truth masks
    """
    numerator = np.sum(np.round(pred) * truth) * 2.0
    denominator = np.sum(np.round(pred)) + np.sum(truth)

    return numerator / denominator<jupyter_output><empty_output><jupyter_text>Crop image at center<jupyter_code>def crop_img(img, msk, crop_dim, n_channels, n_out_channels):
    """
    Crop the image and mask
    """

    number_of_dimensions = len(crop_dim)

    slices = []

    for idx in range(number_of_dimensions):  # Go through each dimension

        cropLen = crop_dim[idx]
        imgLen = img.shape[idx]

        start = (imgLen-cropLen)//2

        slices.append(slice(start, start+cropLen))

    # No slicing along channels
    slices_img = slices.copy()
    slices_msk = slices.copy()

    slices_img.append(slice(0, n_channels))
    slices_msk.append(slice(0, n_out_channels))

    return img[tuple(slices_img)], msk[tuple(slices_msk)]<jupyter_output><empty_output><jupyter_text>Z normalize image<jupyter_code>def z_normalize_img(img):
    """
    Normalize the image so that the mean value for each image
    is 0 and the standard deviation is 1.
    """
    for channel in range(img.shape[-1]):

        img_temp = img[..., channel]
        img_temp = (img_temp - np.mean(img_temp)) / np.std(img_temp)

        img[..., channel] = img_temp

    return img<jupyter_output><empty_output><jupyter_text>Load data from Nifti filesThe critical point is that OpenVINO expects the tensor to be in a different order than TensorFlow (channels first versus channels last).  Notice that we simply need to transpose the dimensions after loading to get it in the right order.<jupyter_code>def load_data(imgFile, mskFile, crop_dim, n_channels, n_out_channels, openVINO_order=True):
    """
    Modify this to load your data and labels
    """

    imgs = np.empty((len(imgFile),*crop_dim,n_channels))
    msks = np.empty((len(mskFile),*crop_dim,n_out_channels))
    fileIDs = []

    for idx in range(len(imgFile)):

        img_temp = np.array(nib.load(imgFile[idx]).dataobj)
        msk = np.array(nib.load(mskFile[idx]).dataobj)

        if n_channels == 1:
            img = img_temp[:, :, :, [0]]  # FLAIR channel
        else:
            img = img_temp

        # Add channels to mask
        msk[msk > 0] = 1.0
        msk = np.expand_dims(msk, -1)


        # Crop the image to the input size
        img, msk = crop_img(img, msk, crop_dim, n_channels, n_out_channels)

        # z-normalize the pixel values
        img = z_normalize_img(img)

        fileIDs.append(os.path.basename(imgFile[idx]))

        imgs[idx] = img
        msks[idx] = msk

    if openVINO_order:
        imgs = imgs.transpose((0, 4, 1, 2, 3))
        msks = msks.transpose((0, 4, 1, 2, 3))

    return imgs, msks, fileIDs<jupyter_output><empty_output><jupyter_text>Load the OpenVINO model<jupyter_code>def load_model(model_xml, fp16=False):
    """
    Load the OpenVINO model.
    """
    log.info("Loading U-Net model to the plugin")

    model_bin = os.path.splitext(model_xml)[0] + ".bin"

    return model_xml, model_bin<jupyter_output><empty_output><jupyter_text>Print the OpenVINO statistics<jupyter_code>def print_stats(exec_net, input_data, n_channels, batch_size, input_blob, out_blob, args):
    """
    Prints layer by layer inference times.
    Good for profiling which ops are most costly in your model.
    """

    # Start sync inference
    log.info("Starting inference ({} iterations)".format(args.number_iter))
    log.info("Number of input channels = {}".format(n_channels))
    log.info("Input data shape = {}".format(input_data.shape))
    infer_time = []

    for i in range(args.number_iter):
        t0 = time()
        res = exec_net.infer(
            inputs={input_blob: input_data[0:batch_size, :n_channels]})
        infer_time.append((time() - t0) * 1000)

    average_inference = np.average(np.asarray(infer_time))
    log.info("Average running time of one batch: {:.5f} ms".format(
        average_inference))
    log.info("Images per second = {:.3f}".format(
        batch_size * 1000.0 / average_inference))

    perf_counts = exec_net.requests[0].get_perf_counts()
    log.info("Performance counters:")
    log.info("{:<70} {:<15} {:<15} {:<15} {:<10}".format("name",
                                                         "layer_type",
                                                         "exec_type",
                                                         "status",
                                                         "real_time, us"))
    for layer, stats in perf_counts.items():
        log.info("{:<70} {:<15} {:<15} {:<15} {:<10}".format(layer,
                                                             stats["layer_type"],
                                                             stats["exec_type"],
                                                             stats["status"],
                                                             stats["real_time"]))<jupyter_output><empty_output><jupyter_text>Read CSV file<jupyter_code>def read_csv_file(filename):
    """
    Read the CSV file with the image and mask filenames
    """
    imgFiles = []
    mskFiles = []
    with open(filename, "rt") as f:
        data = csv.reader(f)
        for row in data:
            if len(row) > 0:
                imgFiles.append(row[0])
                mskFiles.append(row[1])

    return imgFiles, mskFiles, len(imgFiles)
log.basicConfig(format="[ %(levelname)s ] %(message)s",
                level=log.INFO, stream=sys.stdout)

log.info(args)

log.info("Loading test data from file: {}".format(args.csv_file))<jupyter_output><empty_output><jupyter_text>OpenVINO inference<jupyter_code>ie = IECore()
if args.cpu_extension and "CPU" in args.device:
    ie.add_extension(args.cpu_extension, "CPU")

# Read IR
# If using MYRIAD then we need to load FP16 model version
model_xml, model_bin = load_model(args.openvino_model, args.device == "MYRIAD")
log.info("Loading network files:\n\t{}\n\t{}".format(model_xml, model_bin))
net = IENetwork(model=model_xml, weights=model_bin)

if "CPU" in args.device:
    supported_layers = ie.query_network(net, "CPU")
    not_supported_layers = [l for l in net.layers.keys() if l not in supported_layers]
    if len(not_supported_layers) != 0:
        log.error("Following layers are not supported by the plugin for specified device {}:\n {}".
                  format(args.device, ', '.join(not_supported_layers)))
        log.error("Please try to specify cpu extensions library path in sample's command line parameters using -l "
                  "or --cpu_extension command line argument")
        sys.exit(1)

"""
Ask OpenVINO for input and output tensor names and sizes
"""
input_blob = next(iter(net.inputs))  # Name of the input layer
out_blob = next(iter(net.outputs))   # Name of the output layer

# Load data
batch_size, n_channels, height, width, depth = net.inputs[input_blob].shape
batch_size, n_out_channels, height_out, width_out, depth_out = net.outputs[out_blob].shape
crop_dim = [height, width, depth]
"""
Read the CSV file with the filenames of the images and masks
"""
imgFiles, mskFiles, num_imgs = read_csv_file(args.csv_file)
"""
Load the data for OpenVINO
"""
input_data, label_data_ov, img_indicies = load_data(imgFiles, mskFiles,
            crop_dim, n_channels, n_out_channels, openVINO_order=True)<jupyter_output><empty_output><jupyter_text>Variable input and batch sizeFor certain networks, the input shape and batch size can be changed during model load. This can be useful for fully convlutional networks.<jupyter_code># Reshape the OpenVINO network to accept the different image input shape
# NOTE: This only works for some models (e.g. fully convolutional)
batch_size = 1
n_channels = input_data.shape[1]
height = input_data.shape[2]
width = input_data.shape[3]
depth = input_data.shape[4]

net.reshape({input_blob:(batch_size,n_channels,height,width,depth)})
batch_size, n_channels, height, width, depth = net.inputs[input_blob].shape
batch_size, n_out_channels, height_out, width_out, depth_out = net.outputs[out_blob].shape

log.info("The network inputs are:")
for idx, input_layer in enumerate(net.inputs.keys()):
    log.info("{}: {}, shape = {} [N,C,H,W,D]".format(idx,input_layer,net.inputs[input_layer].shape))

log.info("The network outputs are:")
for idx, output_layer in enumerate(net.outputs.keys()):
    log.info("{}: {}, shape = {} [N,C,H,W,D]".format(idx,output_layer,net.outputs[output_layer].shape))
# Loading model to the plugin
log.info("Loading model to the plugin")
exec_net = ie.load_network(network=net, device_name=args.device)
del net
if args.stats:
    # Print the latency and throughput for inference
    print_stats(exec_net, input_data, n_channels,
                batch_size, input_blob, out_blob, args)
"""
OpenVINO inference code
input_blob is the name (string) of the input tensor in the graph
out_blob is the name (string) of the output tensor in the graph
Essentially, this looks exactly like a feed_dict for TensorFlow inference
"""
# Go through the sample validation dataset to plot predictions
predictions_ov = np.zeros((num_imgs, n_out_channels,
                        depth_out, height_out, width_out))

log.info("Starting OpenVINO inference")
ov_times = []
for idx in tqdm(range(0, num_imgs)):

    start_time = time()

    res = exec_net.infer(inputs={input_blob: input_data[[idx],:n_channels]})

    ov_times.append(time() - start_time)

    predictions_ov[idx, ] = res[out_blob]

    #print("{}, {}".format(imgFiles[idx], dice_score(res[out_blob],label_data_ov[idx])))


log.info("Finished OpenVINO inference")

del exec_net<jupyter_output><empty_output><jupyter_text>Keras inference<jupyter_code>"""
Load the data for Keras
"""
input_data, label_data_keras, img_indicies = load_data(imgFiles, mskFiles,
                    crop_dim, n_channels, n_out_channels,
                    openVINO_order=False)

# Load OpenVINO model for inference
model = K.models.load_model(args.keras_model, compile=False)

# Inference only Keras
K.backend._LEARNING_PHASE = tf.constant(0)
K.backend.set_learning_phase(False)
K.backend.set_learning_phase(0)
K.backend.set_image_data_format("channels_last")

predictions_keras = np.zeros((num_imgs,
                        height_out, width_out, depth_out, n_out_channels))

log.info("Starting Keras inference")
keras_times = []
for idx in tqdm(range(num_imgs)):

    start_time = time()
    res = model.predict(input_data[[idx],...,:n_channels])

    keras_times.append(time() - start_time)

    #print("{}, {}".format(imgFiles[idx], dice_score(res,label_data_keras[idx])))

    predictions_keras[idx] = res

log.info("Finished Keras inference")
import matplotlib.pyplot as plt

%matplotlib inline<jupyter_output><empty_output><jupyter_text>Plot predictions<jupyter_code>save_directory = "predictions_openvino"
try:
    os.stat(save_directory)
except:
    os.mkdir(save_directory)
def plot_predictions(idx):
    """
    Evaluate model with Dice metric
    """
    out_channel = 0
    slice_no = 102
    
    img = input_data[[idx],...,:n_channels]
    ground_truth = label_data_keras[idx, :, :, :, out_channel]

    # Transpose the OpenVINO prediction back to NCHWD (to be consistent with Keras)
    pred_ov = np.transpose(predictions_ov, [0,2,3,4,1])[idx, :, :, :, out_channel]
    pred_keras = predictions_keras[idx, :, :, :, out_channel]

    dice_ov = dice_score(pred_ov, ground_truth)
    dice_keras = dice_score(pred_keras, ground_truth)

    plt.figure(figsize=(15,15))
    plt.subplot(1,4,1)
    plt.imshow(img[0,:,:,slice_no,0], cmap="bone")
    plt.title("MRI")
    
    plt.subplot(1,4,2)
    plt.imshow(ground_truth[:,:,slice_no])
    plt.title("Ground truth")
    
    plt.subplot(1,4,3)
    plt.imshow(pred_keras[:,:,slice_no])
    plt.title("Keras\nDice {:.6f}".format(dice_keras))
    
    plt.subplot(1,4,4)
    plt.imshow(pred_ov[:,:,slice_no])
    plt.title("OpenVINO\nDice {:.6f}".format(dice_ov))
    
    log.info("Maximum Absolute Pixel Difference in Predictions is {:.6f}".format(np.max(np.abs(pred_ov[:,:,slice_no] - pred_keras[:,:,slice_no]))))<jupyter_output><empty_output><jupyter_text>Plot some sample predictionsHere we show the MRI input, the ground truth mask, the Keras/TF model prediction, and the OpenVINO model prediction. We also calculate the maximum absolute pixel difference which is usually < 1e-5 (on the order of floating point precision).<jupyter_code>plot_predictions(0)
plot_predictions(2)
plot_predictions(7)
plot_predictions(17)
plot_predictions(25)
plot_predictions(31)
plot_predictions(35)<jupyter_output><empty_output><jupyter_text>Inference times<jupyter_code>log.info("Average inference time: \n"
         "OpenVINO = {} seconds (s.d. {})\n"
         "Keras/TF = {} seconds (s.d. {})\n".format(np.mean(ov_times),
         np.std(ov_times),
         np.mean(keras_times),
         np.std(keras_times)))
log.info("Raw OpenVINO inference times = {} seconds".format(ov_times))
log.info("Raw Keras inference times = {} seconds".format(keras_times))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11673950470136406, 'fraction_numerical': 0.009800026486558071, 'mean_word_length': 3.554583835946924, 'pattern_counts': {'":': 0, '<': 80, '<?xml version=': 0, '>': 71, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 7}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 16, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '307875', 'license': 'Apache-2.0', 'n_tokens_mistral': 5186, 'n_tokens_neox': 4968, 'n_words': 1179, 'path': '3D/Compare_Keras_Model_w_OpenVINO.ipynb', 'repo_name': 'ravi9/unet', 'size': '15102'}