<jupyter_start><jupyter_text>Reference https://machinelearningmastery.com/time-series-prediction-lstm-recurrent-neural-networks-python-keras/  Section 1: Let's import the packages we will use in our lab.<jupyter_code>import math, time
import matplotlib.pyplot as plt
import pandas as pd
import numpy as np
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras.layers import *
from sklearn.preprocessing import MinMaxScaler
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import train_test_split
import pydot
import os
import json
import requests
import joblib
print("Using TensorFlow version", tf.__version__)
from numpy.random import seed
seed(1234)
tf.random.set_seed(5678)
save_dir = "./saved_model"<jupyter_output><empty_output><jupyter_text>Section 2: Let's load the order quantity dataset that we will use to train and test our LSTM network.<jupyter_code>data=pd.read_csv("./data_550k.csv")
print('Number of rows:', data.shape[0])
print('Number of columns:', data.shape[1])
data.head(15)<jupyter_output><empty_output><jupyter_text>Section 3: Let's take a look at the distribution of the dataset with some descriptive statistics and plot some of the values.<jupyter_code>data.describe()
plt.figure(figsize=(20,6))
plt.plot(data.values)
plt.title('Actual order quantity vs unit time')
plt.xlim(0, 1000)
plt.xlabel('Unit Time')
plt.ylabel('Order quantity')
plt.show()<jupyter_output><empty_output><jupyter_text>Section 4: Let's define the time step, which is the number of historical order quantities we will use to predict the next order quantity.  Since our time step is 20, we will use the past 20 orders to predict the next order.  Since we are predicting order quantity, we will have 1 feature and 1 label.<jupyter_code>time_steps = 20
feature_size = 1
label_size = 1<jupyter_output><empty_output><jupyter_text>Section 5: Let's split the input dataset into training and testing datasets with a 70:30 ratio.<jupyter_code>data = np.array(data).flatten()
train_size = int(len(data) * 0.70)
test_size = int(len(data) * 0.30)
train  = data[0:train_size]
test = data[train_size:len(data)]<jupyter_output><empty_output><jupyter_text>Section 6: Let's create our features (X) and labels (y) from our training and testing datasets.<jupyter_code>X_train = []
for i in range(len(train)-time_steps):
    for j in range(time_steps):
        X_train.append(train[i+j])
y_train = []
for i in range(len(train)-time_steps):
    y_train.append(train[i + time_steps])
X_test = []
for i in range(len(test)-time_steps):
    for j in range(time_steps):
        X_test.append(test[i+j])
y_test = []
for i in range(len(test)-time_steps):
    y_test.append(test[i + time_steps])<jupyter_output><empty_output><jupyter_text>Section 7: Let's print out a few values of our training dataset and see how the training features (X_train) and training labels (y_train) are grouped.<jupyter_code>print(train[0:30])
print(X_train[0:20])
print(y_train[0:5])<jupyter_output><empty_output><jupyter_text>Section 8: Let's also print out a few values of our testing dataset and see how the testing features (X_test) and testing labels (y_test) are grouped.<jupyter_code>print(test[0:30])
print(X_test[0:20])
print(y_test[0:5])<jupyter_output><empty_output><jupyter_text>Section 9: Let's normalize the training dataset within a range of 0 to 1.  We will use the slope and intercept of the fitted MinMaxScaler to scale and unscale the data.  We will do this with custom Lambda layers in our model.<jupyter_code>scaler = MinMaxScaler(feature_range=(0, 1))
Xscaled = scaler.fit_transform(pd.DataFrame(train))
print(Xscaled)
scalerSlope = scaler.scale_[0]
scalerIntercept = scaler.min_[0]
print("slope = ", scalerSlope)
print("intercept = ", scalerIntercept)
def scale(X):    
    X = (X * scalerSlope) + scalerIntercept
    return X

def scaleInv(X):
    X = (X - scalerIntercept) / scalerSlope
    return X<jupyter_output><empty_output><jupyter_text>Section 10: Let's reshape our data to fit the format of an LSTM network.  This will be (number of samples, time_steps, feature_size).<jupyter_code>X_train = np.reshape(X_train, ((len(train)-time_steps), time_steps, feature_size))
y_train = np.reshape(y_train, ((len(train)-time_steps), label_size))
X_test = np.reshape(X_test, ((len(test)-time_steps), time_steps, feature_size))
y_test = np.reshape(y_test, ((len(test)-time_steps), label_size))
print(X_train.shape)
print(y_train.shape)
print(X_test.shape)
print(y_test.shape)<jupyter_output><empty_output><jupyter_text>Section 11: Let's define the parameters for our LSTM network.  Our network will be built with 50 cells.  We will use a batch size of 64 to train our network.<jupyter_code>#LSTM parameters
cells = 50
batch_size = 64<jupyter_output><empty_output><jupyter_text>Section 12: Let's build our LSTM network.  We will use the Keras framework to define a 2-layer LSTM.  Since we are using the LSTM for a regression analysis, we will use a linear activation function and mean squared error for our loss function.<jupyter_code>tf_input = (time_steps, feature_size)

model = tf.keras.models.Sequential([    
    tf.keras.layers.Lambda(scale),
    tf.keras.layers.LSTM(cells, input_shape=tf_input, return_sequences=True),
    tf.keras.layers.LSTM(cells),
    tf.keras.layers.Dense(label_size, activation='linear'),
    tf.keras.layers.Lambda(scaleInv)])
    
optimizer = keras.optimizers.Adam(lr=0.001)
model.compile(optimizer = optimizer, loss = 'mean_squared_error', metrics = ['mean_squared_error'])<jupyter_output><empty_output><jupyter_text>Section 13: Let's train our network.  We will train for 1 epoch, or 1 iteration through the training dataset, to save time in the lab.  Try experimenting with more epochs and notice how the loss and mean squared error decrease after each epoch.<jupyter_code>print ("Training...")
model.fit(X_train.astype("float64"), y_train, epochs = 1, batch_size = batch_size, verbose=1, shuffle=False)
model.summary()
tf.keras.utils.plot_model(model, save_dir+"/model.png", show_shapes=True)<jupyter_output><empty_output><jupyter_text>Section 14: Let's plot the first 250 predicted order quantities and compare our results to the actual order quantities.  The values in the left column are the actual 21st order quantities for the first 15 groups of sequences.  The values in the right column represent the model's prediction of the 21st order quantities for the first 15 groups of sequences.<jupyter_code>predicted_order_quantity = model.predict(X_test[0:250]).astype(int)
print("y_test predicted_order_quantity")
for i in range(15):
    print(y_test[i], predicted_order_quantity[i])
# Visualising the results
plt.figure(figsize=(15,6))
plt.plot(y_test, color = 'red', label = 'Actual order quantity')
plt.plot(predicted_order_quantity, color = 'blue', label = 'Predicted order quantity')
plt.xticks(np.arange(0,len(X_test),50))
plt.title('Actual order quantity vs prediction')
plt.xlim(0, 250)
plt.xlabel('Sequence')
plt.ylabel('Order quantity')
plt.legend()
plt.show()<jupyter_output><empty_output><jupyter_text>Section 15: Let's plot the difference between our predicted order quantities and the actual order quantities.  As you can see, the difference is mostly within +/-5 units.<jupyter_code>plt.figure(figsize=(12,8))
plt.plot(y_test[0:250] - predicted_order_quantity)
plt.xlim(0, 250)
plt.ylim(-50,51)
plt.yticks(np.arange(-50, 51, 5))
plt.axhline(y=-5, color='r')
plt.axhline(y=5, color='r')
plt.title('Difference between predicted and actual order quantities')
plt.xlabel('Sequence')
plt.ylabel('Range')
plt.show()<jupyter_output><empty_output><jupyter_text>Section 16: Let's compute the percentage of occurences with a difference between predicted order quantities and actual order quantities of less than 5.<jupyter_code>diff = np.absolute(y_test[0:250] - predicted_order_quantity)
print('{:5.0f}%'.format((diff < 5).sum() / diff.shape[0] * 100))<jupyter_output><empty_output><jupyter_text>Section 17: Let's save our model, restore it, and re-run the predictions to verify our results are the same.  Again, the values in the left column are the actual 21st order quantities for the first 15 groups of sequences.  The values in the right column represent the model's prediction of the 21st order quantities for the first 15 groups of sequences.<jupyter_code>model.save_weights(os.path.join(save_dir+"/1","wts"))
model.save(save_dir+"/1")
restored_model = tf.keras.models.load_model(save_dir+"/1")
restored_model.compile(optimizer = 'adam', loss = 'mean_squared_error', metrics = ['mean_squared_error'])
restored_predicted_order_quantity = restored_model.predict(X_test[0:250]).astype(int)
print("y_test restored_predicted_order_quantity")
for i in range(15):
    print(y_test[i], restored_predicted_order_quantity[i])<jupyter_output><empty_output><jupyter_text>Section 18: Let's work with our model in TensorFlow Serving and re-run the predictions to verify our results are the same.  Returning back to the instructions in github, start the tensorflow-serving container. Once it is up, return to this notebook to issue a request.<jupyter_code>data = json.dumps({"instances": X_test[0:15].tolist()})
json_response = requests.post('http://tf-lab-serving:8501/v1/models/saved_model:predict', data=data)
result = np.array(json.loads(json_response.text)["predictions"]).astype(int)
result<jupyter_output><empty_output><jupyter_text>Section 19: Let's try to manually enter a sequence of 20 order quantities and let the model predict the next quantity.  Copy the following line of code and paste in the next cell.  Replace the s with integers:X_input = np.array([[], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], []])<jupyter_code>X_input = np.array([[71],
       [71],
       [71],
       [41],
       [71],
       [71],
       [32],
       [71],
       [70],
       [70],
       [25],
       [50],
       [45],
       [71],
       [13],
       [71],
       [20],
       [71],
       [71],
       [69]])
X_input = np.reshape(X_input, (-1, X_input.shape[0], feature_size))
print(X_input.shape)
data_input = json.dumps({"instances": X_input.tolist()})
json_response = requests.post('http://tf-lab-serving:8501/v1/models/saved_model:predict', data=data_input)
result = np.array(json.loads(json_response.text)["predictions"]).astype(int)
result<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11910741885625965, 'fraction_numerical': 0.024149922720247297, 'mean_word_length': 5.744625407166124, 'pattern_counts': {'":': 2, '<': 78, '<?xml version=': 0, '>': 77, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '223002', 'license': 'Apache-2.0', 'n_tokens_mistral': 3624, 'n_tokens_neox': 3364, 'n_words': 965, 'path': 'tensorflow-lab/zAI_TF_Lab.ipynb', 'repo_name': 'IBM/ai-on-z-containers', 'size': '10352'}