<jupyter_start><jupyter_text>AWS Inferentia inference on Amazon EC2 Inf1 instanceThis example demonstrates AWS Inferentia inference with TensorFlow and AWS Neuron SDK compiler and runtimeThis example was tested on Amazon EC2 `inf1.xlarge` the following AWS Deep Learning AMI: `Deep Learning AMI (Ubuntu 18.04) Version 35.0`Run this notebook using the following conda environment:`aws_neuron_tensorflow_p36`Prepare your imagenet validation TFRecord files using the following helper script:https://github.com/tensorflow/models/blob/archive/research/inception/inception/data/download_and_preprocess_imagenet.shSave it to `/home/ubuntu/datasets/` or update the dataset location in the `get_dataset()` function<jupyter_code># !pip install matplotlib pandas
!/opt/aws/neuron/bin/neuron-cli reset
import os
import time
import shutil
import json
import requests
import numpy as np
import pandas as pd
import tensorflow as tf
import tensorflow.neuron as tfn
import tensorflow.compat.v1.keras as keras
from tensorflow.keras.applications.resnet50 import ResNet50, preprocess_input
from tensorflow.keras.preprocessing import image
from concurrent import futures
from itertools import compress
# https://github.com/tensorflow/tensorflow/issues/29931
temp = tf.zeros([8, 224, 224, 3])
_ = tf.keras.applications.resnet50.preprocess_input(temp)<jupyter_output><empty_output><jupyter_text>Resnet50 FP32 saved model<jupyter_code># Export SavedModel
saved_model_dir = 'resnet50_saved_model'
shutil.rmtree(saved_model_dir, ignore_errors=True)

keras.backend.set_learning_phase(0)
model = ResNet50(weights='imagenet')
tf.saved_model.simple_save(session = keras.backend.get_session(),
                           export_dir = saved_model_dir,
                           inputs = {'input_1:0': model.inputs[0]},
                           outputs = {'probs/Softmax:0': model.outputs[0]})<jupyter_output>WARNING:tensorflow:From /home/ubuntu/anaconda3/envs/aws_neuron_tensorflow_p36/lib/python3.6/site-packages/tensorflow_core/python/ops/resource_variable_ops.py:1630: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.
Downloading data from https://github.com/keras-team/keras-applications/releases/download/resnet/resnet50_weights_tf_dim_ordering_tf_kernels.h5
102973440/102967424 [==============================] - 1s 0us/step
WARNING:tensorflow:From <ipython-input-4-cc6f128953d9>:10: simple_save (from tensorflow.python.saved_model.simple_save) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.simple_save.
WARNING:tensorflow:From /home/ubuntu/anaconda3/envs/aws_neuron_tenso[...]<jupyter_text>Compile models with different batch sizes and cores<jupyter_code>def compile_inf1_model(saved_model_dir, inf1_model_dir, batch_size=1, num_cores=1, use_static_weights=False):
    print(f'-----------batch size: {batch_size}, num cores: {num_cores}----------')
    print('Compiling...')
    
    compiled_model_dir = f'resnet50_batch_{batch_size}_inf1_cores_{num_cores}'
    inf1_compiled_model_dir = os.path.join(inf1_model_dir, compiled_model_dir)
    shutil.rmtree(inf1_compiled_model_dir, ignore_errors=True)

    example_input = np.zeros([batch_size,224,224,3], dtype='float32')

    compiler_args = ['--verbose','1', '--num-neuroncores', str(num_cores)]
    if use_static_weights:
        compiler_args.append('--static-weights')
    
    start_time = time.time()
    compiled_res = tfn.saved_model.compile(model_dir = saved_model_dir,
                            model_feed_dict={'input_1:0': example_input},
                            new_model_dir = inf1_compiled_model_dir,
                            dynamic_batch_size=True,
                            compiler_workdir=f'./compiler-workdir/{inf1_compiled_model_dir}',
                            compiler_args = compiler_args)
    print(f'Compile time: {time.time() - start_time}')
    
    compile_success = False
    perc_on_inf = compiled_res['OnNeuronRatio'] * 100
    if perc_on_inf > 50:
        compile_success = True
            
    print(inf1_compiled_model_dir)
    print(compiled_res)
    print('----------- Done! ----------- \n')
    
    return compile_success<jupyter_output><empty_output><jupyter_text>Use `tf.data` to read ImageNet validation dataset<jupyter_code>def deserialize_image_record(record):
    feature_map = {'image/encoded': tf.io.FixedLenFeature([], tf.string, ''),
                  'image/class/label': tf.io.FixedLenFeature([1], tf.int64, -1),
                  'image/class/text': tf.io.FixedLenFeature([], tf.string, '')}
    obj = tf.io.parse_single_example(serialized=record, features=feature_map)
    imgdata = obj['image/encoded']
    label = tf.cast(obj['image/class/label'], tf.int32)   
    label_text = tf.cast(obj['image/class/text'], tf.string)   
    return imgdata, label, label_text

def val_preprocessing(record):
    imgdata, label, label_text = deserialize_image_record(record)
    label -= 1
    image = tf.io.decode_jpeg(imgdata, channels=3, 
                              fancy_upscaling=False, 
                              dct_method='INTEGER_FAST')

    shape = tf.shape(image)
    height = tf.cast(shape[0], tf.float32)
    width = tf.cast(shape[1], tf.float32)
    side = tf.cast(tf.convert_to_tensor(256, dtype=tf.int32), tf.float32)

    scale = tf.cond(tf.greater(height, width),
                  lambda: side / width,
                  lambda: side / height)
    
    new_height = tf.cast(tf.math.rint(height * scale), tf.int32)
    new_width = tf.cast(tf.math.rint(width * scale), tf.int32)
    
    image = tf.image.resize(image, [new_height, new_width], method='bicubic')
    image = tf.image.resize_with_crop_or_pad(image, 224, 224)
    
    image = tf.keras.applications.resnet50.preprocess_input(image)
    
    return image, label, label_text

def get_dataset(batch_size, use_cache=False):
    data_dir = '/home/ubuntu/datasets/*'
    files = tf.io.gfile.glob(os.path.join(data_dir))
    dataset = tf.data.TFRecordDataset(files)
    
    dataset = dataset.map(map_func=val_preprocessing, num_parallel_calls=tf.data.experimental.AUTOTUNE)
    dataset = dataset.batch(batch_size=batch_size)
    dataset = dataset.prefetch(buffer_size=tf.data.experimental.AUTOTUNE)
    dataset = dataset.repeat(count=1)
    
    if use_cache:
        shutil.rmtree('tfdatacache', ignore_errors=True)
        os.mkdir('tfdatacache')
        dataset = dataset.cache(f'./tfdatacache/imagenet_val')
    
    return dataset<jupyter_output><empty_output><jupyter_text>Single AWS Inferentia chip execution* Single core compiled models with automatic data parallel model upto 4 cores* Multi-core compiled models for pipeline execution<jupyter_code>def inf1_predict_benchmark_single_threaded(neuron_saved_model_name, batch_size, user_batch_size, num_cores, use_cache=False, warm_up=10):
    print(f'Running model {neuron_saved_model_name}, user_batch_size: {user_batch_size}\n')

    model_inf1 = tf.contrib.predictor.from_saved_model(neuron_saved_model_name)

    iter_times = []
    pred_labels = []
    actual_labels = []
    display_threshold = 0
    warm_up = 10

    ds = get_dataset(user_batch_size, use_cache)

    ds_iter = ds.make_initializable_iterator()
    ds_next = ds_iter.get_next()
    ds_init_op = ds_iter.initializer

    with tf.Session() as sess:
        if use_cache:
            sess.run(ds_init_op)
            print('\nCaching dataset ...')
            start_time = time.time()
            try:
                while True:
                    (validation_ds,label,_) = sess.run(ds_next)
            except tf.errors.OutOfRangeError:
                pass
            print(f'Caching finished: {time.time()-start_time} sec')  

        try:
            sess.run(ds_init_op)
            counter = 0
            
            total_datas = 1000
            display_every = 100
            display_threshold = display_every
            
            ipname = list(model_inf1.feed_tensors.keys())[0]
            resname = list(model_inf1.fetch_tensors.keys())[0]
            
            walltime_start = time.time()

            while True:
                (validation_ds,batch_labels,_) = sess.run(ds_next)

                model_feed_dict={ipname: validation_ds}

                if counter == 0:
                    for i in range(warm_up):
                        _ = model_inf1(model_feed_dict);                    

                start_time = time.time()
                inf1_results = model_inf1(model_feed_dict);
                iter_times.append(time.time() - start_time)
                
                actual_labels.extend(label for label_list in batch_labels for label in label_list)
                pred_labels.extend(list(np.argmax(inf1_results[resname], axis=1)))

                if counter*user_batch_size >= display_threshold:
                    print(f'Images {counter*user_batch_size}/{total_datas}. Average i/s {np.mean(user_batch_size/np.array(iter_times[-display_every:]))}')
                    display_threshold+=display_every

                counter+=1

        except tf.errors.OutOfRangeError:
            pass
        
    acc_inf1 = np.sum(np.array(actual_labels) == np.array(pred_labels))/len(actual_labels)
    iter_times = np.array(iter_times)
    
    instacne_ip = '172.16.17.32'
    
    results = pd.DataFrame(columns = [f'inf1_compiled_batch_size_{batch_size}_compiled_cores_{num_cores}'])
    results.loc['instance_type']           = [requests.get(f'http://{instance_ip}/latest/meta-data/instance-type').text]
    results.loc['compiled_batch_size']     = [batch_size]
    results.loc['user_batch_size']         = [user_batch_size]
    results.loc['accuracy']                = [acc_inf1]
    results.loc['prediction_time']         = [np.sum(iter_times)]
    results.loc['wall_time']               = [time.time() - walltime_start]
    results.loc['images_per_sec_mean']     = [np.mean(user_batch_size / iter_times)]
    results.loc['images_per_sec_std']      = [np.std(user_batch_size / iter_times, ddof=1)]
    results.loc['latency_mean']            = [np.mean(iter_times) * 1000]
    results.loc['latency_99th_percentile'] = [np.percentile(iter_times, q=99, interpolation="lower") * 1000]
    results.loc['latency_median']          = [np.median(iter_times) * 1000]
    results.loc['latency_min']             = [np.min(iter_times) * 1000]
    display(results.T)

    return results, iter_times
inf1_model_dir = 'resnet50_inf1_saved_models'
saved_model_dir = 'resnet50_saved_model'


# testing batch size
batch_list = [1,2,4,8,16,32,64]
for batch in batch_list:
    print('batch size:', batch, 'compile start')
    compile_inf1_model(saved_model_dir, inf1_model_dir, batch_size=batch, num_cores=1)

inf1_model_dir = 'resnet50_inf1_saved_models'

compile_options = [{'batch_size': 1, 'num_cores': 1},
                  {'batch_size': 2, 'num_cores': 1},
                  {'batch_size': 4, 'num_cores': 1},
                  {'batch_size': 8, 'num_cores': 1},
                  {'batch_size': 16, 'num_cores': 1},
                  {'batch_size': 32, 'num_cores': 1},
                  {'batch_size': 64, 'num_cores': 1}]

iter_ds = pd.DataFrame()
results = pd.DataFrame()

for opt in compile_options:
    batch_size = opt["batch_size"]
    num_cores = opt["num_cores"]
    compiled_model_dir = f'resnet50_batch_{batch_size}_inf1_cores_{num_cores}'
    inf1_compiled_model_dir = os.path.join(inf1_model_dir, compiled_model_dir)
   
    print(f'inf1_compiled_model_dir: {inf1_compiled_model_dir}')
    col_name = lambda opt: f'inf1_{batch_size}_multicores_{num_cores}'
    
    res, iter_times = inf1_predict_benchmark_single_threaded(inf1_compiled_model_dir,
                                                                     batch_size = batch_size,
                                                                     user_batch_size = batch_size*10,
                                                                     num_cores = num_cores,
                                                                     use_cache=False, 
                                                                     warm_up=10)
    
    iter_ds = pd.concat([iter_ds, pd.DataFrame(iter_times, columns=[col_name(opt)])], axis=1)
    results = pd.concat([results, res], axis=1)
    
display(results)
inf1_model_dir = 'resnet50_inf1_saved_models'
saved_model_dir = 'resnet50_saved_model'


# testing batch size
core_list = [1,2,3,4,5,6]
for core in core_list:
    print('core size:', core, 'compile start')
    compile_inf1_model(saved_model_dir, inf1_model_dir, batch_size=1, num_cores=core)

inf1_model_dir = 'resnet50_inf1_saved_models'

compile_options = [{'batch_size': 1, 'num_cores': 1},
                  {'batch_size': 1, 'num_cores': 2},
                  {'batch_size': 1, 'num_cores': 3},
                  {'batch_size': 1, 'num_cores': 4},
                  {'batch_size': 1, 'num_cores': 5},
                  {'batch_size': 1, 'num_cores': 6}]

iter_ds = pd.DataFrame()
results = pd.DataFrame()

for opt in compile_options:
    batch_size = opt["batch_size"]
    num_cores = opt["num_cores"]
    compiled_model_dir = f'resnet50_batch_{batch_size}_inf1_cores_{num_cores}'
    inf1_compiled_model_dir = os.path.join(inf1_model_dir, compiled_model_dir)
   
    print(f'inf1_compiled_model_dir: {inf1_compiled_model_dir}')
    col_name = lambda opt: f'inf1_{batch_size}_multicores_{num_cores}'
    
    res, iter_times = inf1_predict_benchmark_single_threaded(inf1_compiled_model_dir,
                                                                     batch_size = batch_size,
                                                                     user_batch_size = batch_size*10,
                                                                     num_cores = num_cores,
                                                                     use_cache=False, 
                                                                     warm_up=10)
    
    iter_ds = pd.concat([iter_ds, pd.DataFrame(iter_times, columns=[col_name(opt)])], axis=1)
    results = pd.concat([results, res], axis=1)
    
display(results)<jupyter_output>inf1_compiled_model_dir: resnet50_inf1_saved_models/resnet50_batch_1_inf1_cores_1
Running model resnet50_inf1_saved_models/resnet50_batch_1_inf1_cores_1, user_batch_size: 10

INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:The specified SavedModel has no variables; no checkpoints were restored.
Images 100/1000. Average i/s 492.7609983197934
Images 200/1000. Average i/s 498.76356187020656
Images 300/1000. Average i/s 487.3440884835897
Images 400/1000. Average i/s 496.515840915395
Images 500/1000. Average i/s 494.38088381082173
Images 600/1000. Average i/s 492.91520717919605
Images 700/1000. Average i/s 492.3006276132294
Images 800/1000. Average i/s 489.16447506571853
Images 900/1000. Average i/s 486.91232490944327
{'fraction_non_alphanumeric': 0.1067071160060674, 'fraction_numerical': 0.03627250544087582, 'mean_word_length': 2.838987341772152, 'pattern_counts': {'":': 0, '<': 20, '<?xml version=': 0, '>': 22, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 33, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '276720', 'license': 'Apache-2.0', 'n_tokens_mistral': 5187, 'n_tokens_neox': 4699, 'n_words': 854, 'path': 'inf1-neuron-sdk-resnet50.ipynb', 'repo_name': 'manchann/ai-accelerators-examples', 'size': '15162'}