<jupyter_start><jupyter_text>TH Nürnberg - Neural Network Translator - <NAME>, <NAME> Evaluate the Performance of the sample Keras Neural Networks for the Neural-Network-TranslatorThis jupyter notebooks is an addition to the performance analysis in the project report. It provides all information about the models as well as the used input values and the function calls. Therefore, this notebook is suitable for repeating the performance tests. For a detailed description of the neural networks, please see the notebook "keras example networks".<jupyter_code>from tensorflow import keras
import tensorflow as tf
import numpy as np
from sklearn.preprocessing import scale
import time
import os
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>Create an out directory to save the models:<jupyter_code>if not os.path.exists('out'):
    os.makedirs('out')<jupyter_output><empty_output><jupyter_text>Average Pooling 1D In this section, we will build a neural network only consisting of one single average pooling 1D layer. The created model can then be flashed on the Arduino microcontroller to evaluate the performance.<jupyter_code>data_x = np.array([[1,2,3,4,5,6]])
data_y = np.array([[1,2,3]])

data_x = scale(data_x,axis=0)
data_x = tf.reshape(data_x,[-1,6,1])
data_y = tf.reshape(data_y,[-1,3,1])

model = keras.Sequential()
model.add(keras.layers.AveragePooling1D(pool_size=2, strides=2 ,padding='valid', input_shape = (6,1)))

model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

model.fit(data_x, data_y, epochs=1, batch_size=10)

model.save('out/avg_pool_1d.h5')<jupyter_output><empty_output><jupyter_text>In the next step, we will prepare ten sample inputs to verify the results as well as the prediction performance.<jupyter_code>input_array = np.array([[9,119,80,35,0,29],
                        [94,33,146,36.6,0.254,51],
                        [10,125,70,26,115,31.1],
                        [76,0,0,39.4,0.257,43],
                        [1,97,66,15,140,23.2],
                        [82,19,110,22.2,0.245,57],
                        [5,117,92,0,0,34.1],
                        [75,26,0,36,0.546,60],
                        [3,158,76,36,245,31.6],
                        [58,11,54,24.8,0.267,22]
                       ])<jupyter_output><empty_output><jupyter_text>To make it more convenient, we will prepare the input data so that we can simply copy and paste the values in the serial dialog of our Arduino IDE. We can now simply copy and paste the values inbetween of the square brackets [ ].<jupyter_code>for i in range(0,10):
    print(input_array[i].flatten().tolist())<jupyter_output><empty_output><jupyter_text>In the next step, we will perform the predictionswith the trained network.<jupyter_code>total_duration = 0
for i in range(0,10):
    input = tf.reshape(input_array[i],[1,6,1])
    time_before = int(round(time.time_ns() / 1000))
    predictions = model.predict(input)
    time_after = int(round(time.time_ns() / 1000))
    print(predictions)
    total_duration += time_after - time_before
    print("process time in microseconds: " + str(time_after - time_before))
average_duration = float(total_duration)/10
print("total_duration: " + str(total_duration))
print("average_duration: " + str(average_duration))<jupyter_output><empty_output><jupyter_text>Now we can compare the output and the duration of the prediction of our trained model with the output of the neural network translator.  Average Pooling 2D In this section, we repeat the building process but instead of a 1D average pooling layer, we will build a neural network only consisting of one single average pooling 2D layer. The created model can then be again used to validate the performance of the average pooling layer.<jupyter_code>data_x = np.array([[1,1,1], [1,1,1], [1,1,1], [1,1,1], [1,1,1]])
data_y = np.array([[[1],[1]],[[1],[1]],[[1],[1]],[[1],[1]]])

data_x = scale(data_x,axis=0)
data_x = tf.reshape(data_x,[-1,5,3,1])
data_y = tf.reshape(data_y,[-1,4,2,1])

model = keras.Sequential()
model.add(keras.layers.AveragePooling2D(pool_size=(2,2), strides=(1,1),padding='valid', input_shape = (5,3,1)))

model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

model.fit(data_x, data_y, epochs=1, batch_size=10)

model.save('out/avg_pool_2d.h5')<jupyter_output><empty_output><jupyter_text>In the next step, we will again prepare ten input samples to check if the result of our build model is equal to the result of our translated model and how it performs.<jupyter_code>input_array = np.array([[[26,1,37], [115,189,31.3], [31.3,29.6,103], [0.205,0,83], [41,36,2.2]],
                        [[9,119,80], [35,0,29], [94,33,146], [36.6,0.254,51], [10,125,70]],
                        [[26,115,31.1], [76,0,0], [39.4,0.257,43], [1,97,66], [15,140,23.2]],
                        [[82,19,110], [22.2,0.245,57], [5,117,92], [0,0,34.1], [75,26,0]],
                        [[36,0.546,60], [3,158,76], [36,245,31.6], [58,11,54], [24.8,0.267,22]],
                        [[1,79,60], [42,48,43.5], [0.678,23,2], [75,64,24], [55,29.7,0.37]],
                        [[33,8,179], [72,42,130], [32.7,0.719,36], [6,85,78], [0,0,31.2]],
                        [[0.382,42,0], [129,110,46], [130,67.1,0.319], [26,5,143], [78,0,0]],
                        [[45,0.19,47], [5,130,82], [0,0,39.1], [0.956,37,6], [87,80,0]],
                        [[0,23.2,0.084], [0,119,64], [18,92,34.9], [0.725,23,1], [0,74,20]]
                       ])<jupyter_output><empty_output><jupyter_text>To make it more convenient, we will prepare the input data so that we can simply copy and paste the values in the serial dialog of our Arduino IDE. We can now simply copy and paste the values inbetween of the square brackets [ ].<jupyter_code>for i in range(0,10):
    print(input_array[i].flatten().tolist())<jupyter_output><empty_output><jupyter_text>In the next step we will perform the predictions with the trained network.<jupyter_code>total_duration = 0
for i in range(0,10):
    input = tf.reshape(input_array[i],[1,5,3,1])
    time_before = int(round(time.time_ns() / 1000))
    predictions = model.predict(input)
    time_after = int(round(time.time_ns() / 1000))
    print(predictions)
    total_duration += time_after - time_before
    print("process time in microseconds: " + str(time_after - time_before))    
average_duration = float(total_duration)/10
print("total_duration: " + str(total_duration))
print("average_duration: " + str(average_duration))<jupyter_output><empty_output><jupyter_text>We can now compare the output and the runtime of our trained model with the output and the runtime of the neural network translator.  Max Pooling 1D In this section, we will build a neural network only consisting of one single max pooling 1D layer. The created model can then be used to validate the performance and the runtime of the max pooling layer.<jupyter_code>data_x = np.array([[1,2,3,4,5,6]])
data_y = np.array([[1,2,3]])

data_x = scale(data_x,axis=0)
data_x = tf.reshape(data_x,[-1,6,1])
data_y = tf.reshape(data_y,[-1,3,1])

model = keras.Sequential()
model.add(keras.layers.MaxPooling1D(pool_size=2, strides=2 ,padding='valid', input_shape = (6,1)))

model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

model.fit(data_x, data_y, epochs=1, batch_size=10)

model.save('out/max_pool_1d.h5')<jupyter_output><empty_output><jupyter_text>In the next step, we will prepare ten sample inputs to check if the result of our build model is equal to the result of our translated model and how it performs.<jupyter_code>input_array = np.array([[9,119,80,35,0,29],
                        [94,33,146,36.6,0.254,51],
                        [10,125,70,26,115,31.1],
                        [76,0,0,39.4,0.257,43],
                        [1,97,66,15,140,23.2],
                        [82,19,110,22.2,0.245,57],
                        [5,117,92,0,0,34.1],
                        [75,26,0,36,0.546,60],
                        [3,158,76,36,245,31.6],
                        [58,11,54,24.8,0.267,22]
                       ])<jupyter_output><empty_output><jupyter_text>To make it more convenient, we will prepare the input data so that we can simply copy and paste the values in the serial dialog of our Arduino IDE. We can now simply copy and paste the values inbetween of the square brackets [ ].<jupyter_code>for i in range(0,10):
    print(input_array[i].flatten().tolist())<jupyter_output><empty_output><jupyter_text>In the next step we will perform the predictions with the trained network.<jupyter_code>total_duration = 0
for i in range(0,10):
    input = tf.reshape(input_array[i],[1,6,1])
    time_before = int(round(time.time_ns() / 1000))
    predictions = model.predict(input)
    time_after = int(round(time.time_ns() / 1000))
    print(predictions)
    total_duration += time_after - time_before
    print("process time in microseconds: " + str(time_after - time_before))
average_duration = float(total_duration)/10
print("total_duration: " + str(total_duration))
print("average_duration: " + str(average_duration))<jupyter_output><empty_output><jupyter_text>We can now compare the output and the runtime of our trained model with the output and the runtime of the neural network translator.  Max Pooling 2D In this section, we repeat the building process but instead of a 1D max pooling layer we will build a neural network only consisting of one single max pooling 2D layer. The created model can then be again used to validate the implementation and the performance of the max pooling layer.<jupyter_code>data_x = np.array([[1,1,1], [1,1,1], [1,1,1], [1,1,1], [1,1,1]])
data_y = np.array([[[1],[1]],[[1],[1]],[[1],[1]],[[1],[1]]])

data_x = scale(data_x,axis=0)
data_x = tf.reshape(data_x,[-1,5,3,1])
data_y = tf.reshape(data_y,[-1,4,2,1])

model = keras.Sequential()
model.add(keras.layers.MaxPooling2D(pool_size=(2,2), strides=(1,1),padding='valid', input_shape = (5,3,1)))

model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

model.fit(data_x, data_y, epochs=1, batch_size=10)

model.save('out/max_pool_2d.h5')<jupyter_output><empty_output><jupyter_text>In the next step; we will prepare a sample of ten inputs to check if the result of our build model is equal to the result of our translated model and how it performs.<jupyter_code>input_array = np.array([[[26,1,37], [115,189,31.3], [31.3,29.6,103], [0.205,0,83], [41,36,2.2]],
                        [[9,119,80], [35,0,29], [94,33,146], [36.6,0.254,51], [10,125,70]],
                        [[26,115,31.1], [76,0,0], [39.4,0.257,43], [1,97,66], [15,140,23.2]],
                        [[82,19,110], [22.2,0.245,57], [5,117,92], [0,0,34.1], [75,26,0]],
                        [[36,0.546,60], [3,158,76], [36,245,31.6], [58,11,54], [24.8,0.267,22]],
                        [[1,79,60], [42,48,43.5], [0.678,23,2], [75,64,24], [55,29.7,0.37]],
                        [[33,8,179], [72,42,130], [32.7,0.719,36], [6,85,78], [0,0,31.2]],
                        [[0.382,42,0], [129,110,46], [130,67.1,0.319], [26,5,143], [78,0,0]],
                        [[45,0.19,47], [5,130,82], [0,0,39.1], [0.956,37,6], [87,80,0]],
                        [[0,23.2,0.084], [0,119,64], [18,92,34.9], [0.725,23,1], [0,74,20]]
                       ])<jupyter_output><empty_output><jupyter_text>To make it more convenient, we will prepare the input data so that we can simply copy and paste the values in the serial dialog of our Arduino IDE. We can now simply copy and paste the values inbetween of the square brackets [ ].<jupyter_code>for i in range(0,10):
    print(input_array[i].flatten().tolist())<jupyter_output><empty_output><jupyter_text>In the next step, we will perform the predictions with the trained network.<jupyter_code>total_duration = 0
for i in range(0,10):
    input = tf.reshape(input_array[i],[1,5,3,1])
    time_before = int(round(time.time_ns() / 1000))
    predictions = model.predict(input)
    time_after = int(round(time.time_ns() / 1000))
    print(predictions)
    total_duration += time_after - time_before
    print("process time in microseconds: " + str(time_after - time_before))    
average_duration = float(total_duration)/10
print("total_duration: " + str(total_duration))
print("average_duration: " + str(average_duration))<jupyter_output><empty_output><jupyter_text>We can now compare the output and the performance of our trained model with the output and the performance of the neural network translator.  2 Layer Diabates In the following, we will create a sample neural network which consists of two dense layers. Thereby, the provided diabetes.csv file is used as training and test data.<jupyter_code>dataset = np.loadtxt("diabetes.csv", delimiter=",", skiprows=1 )
diabetes_X = dataset[:,0:8]
diabetes_Y = dataset[:,8]
diabetes_X = scale(diabetes_X,axis=0)

model = keras.Sequential()
model.add(keras.layers.Dense(8, input_dim=8, activation='sigmoid'))
model.add(keras.layers.Dense(1, activation='sigmoid'))

model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

model.fit(diabetes_X, diabetes_Y, epochs=300, batch_size=10)

model.save('out/2_layer_diabetes.h5')<jupyter_output><empty_output><jupyter_text>In the following, the predictions are performed as well as the processing time will be measured to compare the framework with the translated model created by the neural network translator for the arduino.<jupyter_code>total_duration = 0
for i in range(1,11):
    input_array = dataset[i][0:8]
    input = tf.reshape(input_array,[1,8])
    print("Input: " + str(np.array(input[0]).flatten().tolist()))
    time_before = int(round(time.time_ns() / 1000))
    predictions = model.predict(input)
    time_after = int(round(time.time_ns() / 1000))
    print("Preditction: " + str(predictions))
    total_duration += time_after - time_before
    print("Processing time in microseconds: " + str(time_after - time_before))
average_duration = float(total_duration)/10
print("total_duration: " + str(total_duration))
print("average_duration: " + str(average_duration))<jupyter_output><empty_output><jupyter_text>3 Layer Diabates In addition to the 2-layer-diabetes neural network, a 3-layer-diabetes neural network can be created to test the neural network translator. Not only does this neural network have an additional dense layer, but it also uses different activation functions. Build and save the neural network model:<jupyter_code>dataset = np.loadtxt("diabetes.csv", delimiter=",", skiprows=1 )
dataset[1:11]

model = keras.Sequential()
model.add(keras.layers.Dense(16, input_dim=8, activation='relu'))
model.add(keras.layers.Dense(8, activation="relu"))
model.add(keras.layers.Dense(1, activation='sigmoid'))

model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

model.fit(diabetes_X, diabetes_Y, epochs=300, batch_size=10)
model.save('out/3_layer_diabetes.h5')<jupyter_output><empty_output><jupyter_text>Perform predictions and measure runtime:<jupyter_code>total_duration = 0
for i in range(1,11):
    input_array = dataset[i][0:8]
    input = tf.reshape(input_array,[1,8])
    print("Input: " + str(np.array(input[0]).flatten().tolist()))
    time_before = int(round(time.time_ns() / 1000))
    predictions = model.predict(input)
    time_after = int(round(time.time_ns() / 1000))
    print("Preditction: " + str(predictions))
    total_duration += time_after - time_before
    print("Processing time in microseconds: " + str(time_after - time_before))
average_duration = float(total_duration)/10
print("total_duration: " + str(total_duration))
print("average_duration: " + str(average_duration))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14817170111287759, 'fraction_numerical': 0.07968203497615263, 'mean_word_length': 4.278952668680765, 'pattern_counts': {'":': 0, '<': 91, '<?xml version=': 0, '>': 91, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '22', 'dir': 'jupyter-structured-clean-dedup', 'id': '461138', 'license': 'MIT', 'n_tokens_mistral': 6044, 'n_tokens_neox': 5391, 'n_words': 1311, 'path': 'samples/keras/keras example networks - performance evaluation.ipynb', 'repo_name': 'TerrestrialEclipse/Neural-Network-Translator', 'size': '15745'}