<jupyter_start><jupyter_text>Human Activity Recognition using 2D-PoseIn this practical, we will be developing a model to recognise activity such as jumping, boxing, waving 1 hand, etc. The activity is defined as a sequence of human poses (given by keypoints of skeletal joints) and these poses are estimated by a pretrained model (Google's PoseNet).  Section 1 - Import Libraries and Setup FoldersLet's import all the necessary libraries<jupyter_code>import pandas as pd
import os
import numpy as np
import math
from tqdm import tqdm
from datetime import datetime
import matplotlib.pyplot as plt
import matplotlib.animation as animation
from matplotlib import rc

import tensorflow as tf
import tensorflow.keras as keras
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Activation, Input, Bidirectional, Dropout, LSTM, TimeDistributed, Flatten<jupyter_output><empty_output><jupyter_text>Section 2 - DatasetWe will be using the following dataset from https://github.com/stuarteiffert/RNN-for-Human-Activity-Recognition-using-2D-Pose-InputThe data is 2D positions (x,y coordinates) of 18 joints across a timeseries of 32 frames (window-width), with an associated class label for the frame series.The dataset consist of the following files:- X_test.txt : testing dataset x inputs (36 keypoints per line, 32 lines per datapoint)- X_train.txt : training dataset x inputs (36 keypoints per line, 32 lines per datapoint)- X_val.txt : validation dataset x inputs (36 keypoints per line, 32 lines per datapoint)- Y_test.txt : testing class labels- Y_train.txt : training class labels- Y_val.txt : validation class labels<jupyter_code>!wget https://nyp-aicourse.s3.ap-southeast-1.amazonaws.com/it3103/2D-Pose-Data.zip
!unzip 2D-Pose-Data.zip
train_df = pd.read_csv('2D-Pose-Data/X_train.txt', header=None)
train_label_df = pd.read_csv('2D-Pose-Data/Y_train.txt', header=None)
test_df = pd.read_csv('2D-Pose-Data/X_test.txt', header=None)
test_label_df = pd.read_csv('2D-Pose-Data/Y_test.txt', header=None)
train_df.shape
def fun(x): 
    print(x)
train_df[:5].apply(fun, axis=1)
train_df.head()
train_label_df.head()
train_label_df.value_counts()<jupyter_output><empty_output><jupyter_text>Section 3- Create the input dataWe cannot use the dataframe directly for use by our LSTM network. We need to create a dataset that consists sequence of 32 timesteps (frames) of 36 keypoints. In other words, we need our data to be of the shape (batch_size, 32, 36).<jupyter_code>X_train = train_df.to_numpy().reshape(-1, 32, 36)
y_train = train_label_df.to_numpy() - 1
print(X_train.shape)
print(y_train.shape)
X_test = test_df.to_numpy().reshape(-1, 32, 36)
y_test = test_label_df.to_numpy() - 1
print(X_test.shape)
print(y_test.shape)
# Process OpenPose's Joints
#
def process_joints(x):

    # No processing yet.
    return x<jupyter_output><empty_output><jupyter_text>Section 5 - Visualize Our Dataset We can view each frame as a timestep on the x-axis, and each of the 36 numbers (the x and y coordinates of 18 joints) as individual line plots.  It provides some visual clue as to how the different joints move over time, but they are still difficult to imagine and visualize.<jupyter_code>%matplotlib inline 
sample = 0
plt.plot(X_train[sample])
plt.show()<jupyter_output><empty_output><jupyter_text>A better way to visualize is do a scatter plot of the X and Y coordianates of the various joints and animating them so that we can see their movements over time.NOTE: These are the various types of actions captured in the dataset:JUMPING, JUMPING_JACKS, BOXING, WAVING_2HANDS, WAVING_1HAND", "CLAPPING_HANDS"<jupyter_code>sample = 0

# This function returns a set of data for every frame that is
# called from the animation.FuncAnimation below.
#
def animate_pose(frame):
    # Retrieve the even number values as X-coordinates
    # and the odd number values as Y-coordinates
    #
    # Once you have these 2 sets of values, you can
    # pass them into the line.set_data to get matplotlib
    # to draw a scatter plot 
    #
    graph_x = X_train[sample][frame][0::2]
    graph_y = X_train[sample][frame][1::2]
    line.set_data(graph_x, graph_y)
    return line,

fig, ax = plt.subplots()
plt.close()

ax.set_xlim(0, 800)
ax.set_ylim(600, 0)

line, = ax.plot([], [], 'o', color='black');

anim = animation.FuncAnimation(fig, animate_pose, 32,  interval=50, blit=True)
rc('animation', html='jshtml')
anim<jupyter_output><empty_output><jupyter_text>Section 6 - Define and Train Your ModelTry to define your Keras sequential model using any Keras layers. You should have minimally:1. At least 1 RNN, or GRU, or LSTM Layer2. At least 1 Dense Layer (for the softmax classification)3. The input shape should be (32, 36)Try to experiment and find a suitable model. You can decide if you want to use Bidirectional models, and add any Dropout layers if it helps to improve your model performance. Share your model with the class together with the validation accuracy!A typical model can be designed this way:```    model.add(LSTM(128, input_shape=(32, 36)))    model.add(Dropout(0.2))    model.add(Dense(6, activation='softmax'))    model.compile(loss='sparse_categorical_crossentropy', optimizer='adam', metrics=['accuracy'])    model.summary()```This is part of model selection and evaluation in any machine learning project. You may find that a good validation accuracy for you model may hover near about 85-90%.<jupyter_code>np.unique(y_train)
# Create our LSTM model here
#
def create_model():

    # Use Keras to create a Sequential model here with any layers that 
    # you'd like.
    #
    model = Sequential()

    model.add(LSTM(128, input_shape=(32, 36)))
    model.add(Dropout(0.2))

    model.add(Dense(6, activation='softmax'))

    model.compile(loss='sparse_categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
    model.summary()
    return model


# Create our model
#
model = create_model()


# Create the training folder
#
training_session_id = datetime.now().strftime("%Y-%m-%d %H-%M-%S")
training_session_folder = '/train_%s' % (training_session_id)
os.makedirs(training_session_folder, exist_ok=True)

callbacks_list = [
    keras.callbacks.ModelCheckpoint(
        filepath=training_session_folder + '/model.{epoch:04d}-val_acc-{val_accuracy:4.2f}-loss-{val_loss:4.2f}.h5',
        monitor='val_loss', save_best_only=True),
    keras.callbacks.EarlyStopping(monitor='val_loss', patience=10, restore_best_weights=True)
]


# Train our model
#
history = model.fit(x=X_train, y=y_train, batch_size=256, epochs=40, verbose=1, callbacks=callbacks_list, validation_data=(X_test, y_test))
X_train.shape
model.evaluate(X_test, y_test)<jupyter_output>180/180 [==============================] - 1s 5ms/step - loss: 0.3688 - accuracy: 0.8612<jupyter_text>Section 7 - Scale / Translation NormalizationSo far, we have not talked about how we can normalize our skeletal keypoints so that the pose data is scale / translation invariant. This means that regardless of how far the person is from the camera, or when the person moves left or right or up or down, the coordinates of all joint positions should always be relative to a fixed frame of reference.To take care of translation (left / right / up / down) invariance, we are shift all points together so that neck point is always placed at (0, 0). To take care of scale invariance, we estimate the torso height (which is either the length of the neck point to either hip, or the width of the shoulders). We then divide all joint coordinates by the torso height.To do so, we will create a `process_joints()` function to include code to normalize the skeleton key points as described above:1. ref = P[1] or the midpoint of P[2], P[5]2. reflength = length(ref to P[8]) or length(ref to P[11]) 3. Compute    - P[i].x = (P[i].x - ref.x) / reflength   - P[i].y = (P[i].y - ref.y) / reflength``` Declare a function that can compute length between two points   (x1,y1) - (x2,y2)def compute_length(x1, y1, x2, y2):    return math.sqrt((x1-x2)*(x1-x2) + (y1-y2)*(y1-y2)) Process OpenPose's Joints NOTE: The "x" parameter consists of an array of consecutive x and y values  within the same array. x = [p0.x, p0.y, p1.x, p1.y, p2.x, p2.y, ..., p17.x, p17.y]     (a total of 36 values) def process_joints(x):    r = [0] * 36     Initialize some values for the reference length and the reference point.        refx = 0    refy = 0    reflength = 1     Step 1: Let's find the reference point (neck)        if x[2] != 0 or x[3] != 0:                 refx = x[2]                 use the neck X, Y        refy = x[3]    elif (x[4] != 0 or x[5] != 0) and (x[10] != 0 or x[11] != 0):        refx = (x[4] + x[10]) / 2   estimate the neck X, Y from the mid point        refy = (x[5] + x[11]) / 2   of the left/right shoulder         Step 2: Let's first estimate the torso length.        if x[16] != 0 and x[17] != 0:                     reflength = compute_length(x[16], x[17], refx, refy)    neck to right hip    elif x[22] != 0 and x[23] != 0:        reflength = compute_length(x[22], x[23], refx, refy)    neck to left hip     Step 3:     Perform the translation and the scaling.        for i in range(0, 18):        r[i*2] = (x[i*2] - refx) / reflength        r[i*2 + 1] = (x[i*2 + 1] - refy) / reflength         Return the re-mapped and normalized result        return r```<jupyter_code># Declare a function that can compute length (euclidean distance) between two points
#   (x1,y1) - (x2,y2)
def compute_length(x1, y1, x2, y2):
    return math.sqrt((x1-x2)*(x1-x2) + (y1-y2)*(y1-y2))

# Process OpenPose's Joints

# NOTE: The "keypoints" parameter consists of an array of consecutive x and y values 
# within the same array.
#
# keypoints = [p0.x, p0.y, p1.x, p1.y, p2.x, p2.y, ..., p17.x, p17.y]
#     (a total of 36 values) 
def process_joints(keypoints):

    normalized_keypoints = [0] * 36

    refx = 0
    refy = 0
    reflength = 1

    # Step 1: Let's find the reference point (neck)
    #
    if keypoints[2] != 0 or keypoints[3] != 0:         
        refx = keypoints[2]                # use the neck X, Y
        refy = keypoints[3]
    elif (keypoints[4] != 0 or keypoints[5] != 0) and (keypoints[10] != 0 or keypoints[11] != 0):
        refx = (keypoints[4] + keypoints[10]) / 2  # estimate the neck X, Y from the mid point
        refy = (keypoints[5] + keypoints[11]) / 2  # of the left/right shoulder
    
    # Step 2: Let's first estimate the torso length.
    #
    if keypoints[16] != 0 and keypoints[17] != 0:             
        reflength = compute_length(keypoints[16], keypoints[17], refx, refy)   # neck to right hip
    elif keypoints[22] != 0 and keypoints[23] != 0:
        reflength = compute_length(keypoints[22], keypoints[23], refx, refy)   # neck to left hip

    # Step 3:
    # Perform the translation and the scaling.
    #
    for i in range(0, 18):
        normalized_keypoints[i*2] = (keypoints[i*2] - refx) / reflength
        normalized_keypoints[i*2 + 1] = (keypoints[i*2 + 1] - refy) / reflength
    
    # Return the re-mapped and normalized result
    #
    return normalized_keypoints


train_df_normalized = train_df.apply(process_joints, axis=1, result_type='expand')
test_df_normalized = test_df.apply(process_joints, axis=1, result_type='expand')
train_df_normalized.head()
train_df.shape
train_df_normalized.shape
X_train_normalized = train_df_normalized.to_numpy().reshape(-1, 32, 36)
X_test_normalized = test_df_normalized.to_numpy().reshape(-1, 32, 36)
model = create_model()


# Create the training folder
#
training_session_id = datetime.now().strftime("%Y-%m-%d %H-%M-%S")
training_session_folder = '/train_%s' % (training_session_id)
os.makedirs(training_session_folder, exist_ok=True)

callbacks_list = [
    keras.callbacks.ModelCheckpoint(
        filepath=training_session_folder + '/model.{epoch:04d}-val_acc-{val_accuracy:4.2f}-loss-{val_loss:4.2f}.h5',
        monitor='val_loss', save_best_only=True),
    keras.callbacks.EarlyStopping(monitor='val_loss', patience=10, restore_best_weights=True)
]


# Train our model
#
history = model.fit(x=X_train_normalized, y=y_train, 
                    batch_size=256, 
                    epochs=40, 
                    verbose=1, 
                    callbacks=callbacks_list, 
                    validation_data=(X_test_normalized, y_test))<jupyter_output>Model: "sequential_5"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
lstm_5 (LSTM)                (None, 128)               84480     
_________________________________________________________________
dropout_5 (Dropout)          (None, 128)               0         
_________________________________________________________________
dense_5 (Dense)              (None, 6)                 774       
=================================================================
Total params: 85,254
Trainable params: 85,254
Non-trainable params: 0
_________________________________________________________________
Epoch 1/40
89/89 [==============================] - 4s 21ms/step - loss: 0.7585 - accuracy: 0.7066 - val_loss: 0.3559 - val_accuracy: 0.8557
Epoch 2/40
89/89 [==============================] - 1s 14ms/step - loss: 0.2466 - accuracy: 0.9146 - val_lo[...]<jupyter_text>Section 8 - Re-Load the Data and Re-Train Your Model AgainAt this juncture, re-run the cells under the following sections again without changing the model or training parameters:- Section 4 - Processing the Data- Section 6 - Define and Train Your ModelSee how the model's training and validation accuracy has changed after normalizating the data.**Discuss:**1. Why did the normalization cause a change in the training / validation performance?2. What do you think you can do to improve performance instead of normalizating the data?3. Other than scaling / translating the points, what other kinds of normalization can you do?4. What are the downsides to normalizing the data?  Section 9 - Evaluate the Training History and Performance<jupyter_code>#------------------------------------------------------------------------------------------

import matplotlib.pyplot as plt

# This function is used to display the history the train/test accuracy/loss
# of the Keras training.
#
#   history - Pass in the history returned from the model.fit(...) method.
#
def display_training_loss_and_accuracy(history):
    
    plt.figure(figsize=(20,4))
    
    # summarize history for accuracy
    plt.subplot(1, 2, 1)
    plt.plot(history.history['accuracy'])
    plt.plot(history.history['val_accuracy'])
    plt.title('model accuracy')
    plt.ylabel('accuracy')
    plt.xlabel('epoch')
    plt.legend(['train', 'test'], loc='upper left')
    
    # summarize history for loss
    plt.subplot(1, 2, 2)
    plt.plot(history.history['loss'])
    plt.plot(history.history['val_loss'])
    plt.title('model loss')
    plt.ylabel('loss')
    plt.xlabel('epoch')
    plt.legend(['train', 'test'], loc='upper right')
    plt.show()    

display_training_loss_and_accuracy(history)



labels = ["JUMPING", "JUMPING_JACKS", "BOXING", "WAVING_2HANDS", "WAVING_1HAND", "CLAPPING_HANDS"]
#------------------------------------------------------------------------------------------

import numpy as np
import pandas as pd
from sklearn.metrics import confusion_matrix, classification_report
import seaborn as sns
import matplotlib.pyplot as plt
from functools import reduce
 
def display_classification_confusion_matrix(keras_model, x_train, y_train, x_test, y_test, labels):
    
    '''
    x_train = []
    x_test = []
    y_train = []
    y_test = []
    '''
 
    print(x_train.shape)
    train_preds = keras_model.predict(x_train)
    test_preds = keras_model.predict(x_test)
    train_preds = np.argmax(train_preds, axis=1)
    test_preds = np.argmax(test_preds, axis=1)
    #test_generator.on_epoch_end()

    plt.figure(figsize=(20,6))  

    labels = np.array(labels)
    # Print the first Confusion Matrix for the training data
    #
    cm = confusion_matrix(y_train, train_preds)

    cm_df = pd.DataFrame(cm, labels, labels)          
    plt.subplot(1, 2, 1)
    plt.title('Confusion Matrix (Train Data)')
    sns.heatmap(cm_df, annot=True)
    plt.ylabel('Actual')
    plt.xlabel('Predicted')        
    
    # Print the second Confusion Matrix for the test data
    #    
    cm = confusion_matrix(y_test, test_preds)
    
    cm_df = pd.DataFrame(cm, labels, labels)          
    plt.subplot(1, 2, 2)
    plt.title('Confusion Matrix (Test Data)')
    sns.heatmap(cm_df, annot=True)
    plt.ylabel('Actual')
    plt.xlabel('Predicted')        
    
    plt.show()

    # Finally display the classification reports
    #
    print ("Train Data:")
    print ("--------------------------------------------------------")
    print(classification_report(y_train, train_preds, target_names=labels))
    print ("")
    print ("Test Data:")
    print ("--------------------------------------------------------")
    print(classification_report(y_test, test_preds, target_names=labels))
    

# Exclude the O tags from the confusion matrix.
#
display_classification_confusion_matrix(model, X_train_normalized, y_train, X_test_normalized, y_test, labels)<jupyter_output>(22625, 32, 36)<jupyter_text>Section 9 - Save and Download ModelRun the following cell to save your model.<jupyter_code>model.save("model.savedmodel")<jupyter_output>WARNING:absl:Found untraced functions such as lstm_cell_5_layer_call_and_return_conditional_losses, lstm_cell_5_layer_call_fn, lstm_cell_5_layer_call_fn, lstm_cell_5_layer_call_and_return_conditional_losses, lstm_cell_5_layer_call_and_return_conditional_losses while saving (showing 5 of 5). These functions will not be directly callable after loading.<jupyter_text>Run the following the zip the "model.savedmodel" folder into a single zip file.Download that zip file from Colab once you are done! We will be using this for the next practical exercise.<jupyter_code>!zip model.savedmodel.zip -r model.savedmodel
model = keras.models.load_model('model.savedmodel')
model.summary()
sample_index = 2000
sample = X_test[sample_index]
label = y_test[sample_index]

sample = np.expand_dims(sample, axis=0)
pred = model(sample)
print('actual = {}'.format(labels[np.argmax(label)]))
print('predicted = {}'.format(labels[np.argmax(pred)]))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12978216104909504, 'fraction_numerical': 0.027202774466240382, 'mean_word_length': 3.661530689568073, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 37, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '438993', 'license': 'Apache-2.0', 'n_tokens_mistral': 6324, 'n_tokens_neox': 5806, 'n_words': 1806, 'path': 'week15/human_activity_recognition.ipynb', 'repo_name': 't-kandasami/it3103', 'size': '18454'}