<jupyter_start><jupyter_text>C3D-Siamese Network<jupyter_code>"""
Created on Mon Jan 22 16:20:03 2018

@author: Wail
"""

from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
import matplotlib.pyplot as plt 
from random import randint
import pickle
import os.path
import time
from six.moves import xrange  # pylint: disable=redefined-builtin
import tensorflow as tf
#import input_data
#import c3d_model
import numpy as np
from IPython.display import Image
pickle_path_kth="data/kth_data.p"
pickle_path_Weizmann="data/Weizmann_test.p"<jupyter_output><empty_output><jupyter_text>Presenting C3D-siamese model First, a siamese network is defined as follow:<jupyter_code>Image(filename='../img/siamese.png')<jupyter_output><empty_output><jupyter_text>C3D model is defined as follow:<jupyter_code>Image(filename='../img/c3d.jpg')<jupyter_output><empty_output><jupyter_text>Purpose  The purpose of the C3D-siamese model is to detect similarity between actions (specificaly imitation). In other words given these 4 videos: - swim_person1.avi and swim_person2.avi - boxing_person3.avi and boxing_person4.avi We want our model to give us the following results: - C3D-siamese(swim_person1.avi,swim_person2.avi)=1 ---> similar - C3D-siamese(swim_person1.avi,boxing_person4.avi)=0  ----> not similar  Model Architecture  the C3D-siamese architecture is derived from the siamese one where:- The inputs are videos decomposed into a series of frames (X1: video1, X2: video2)- The Neural Network is a 3-Dimensional Convolutional Networks pre-trained on **sports1m** dataset and used for feature extraction. *for further details visit this link (http://vlg.cs.dartmouth.edu/c3d/)*- For the distance function we took a different path. Knowing that NN(X1)==Y1 and NN(X2)==Y2 we defined our distance function (or vector difference) as Z=abs(Y1-Y2) where Z is still a vector whereas in the classical siamese architecture is z equal to (1 :similar or 0 :not-similar)- Now we use Z as an input to another Neural network for binary classification similar/not-similar  We train only this Network:<jupyter_code>Image(filename='../img/nn.png')<jupyter_output><empty_output><jupyter_text>KTH and WEIZMANN Datasets - **KTH is for training**- **WEIZMANN is for testing**  KTH description 1. Classes - Number of action classes = 6. - Classes/verbs: walking, jogging, running, boxing, hand waving, and hand clapping.2. Videos    - 600 videos.    - Resolution = 160x120.    - Black and white videos.    - Static camera.3. Properties    - Homogeneous indoor/outdoor backgrounds.    - Performed by 25 persons, 4 scenes, and 6 classes/verbs. 4. Website for more infos (http://www.nada.kth.se/cvap/actions/)  WEIZMANN description 1. Classes    - Number of action classes = 10.    - Verbs: walk, run, jump, gallop sideways, bend, one-hand wave, two-hands wave, jump in place, jumping jack, skip.2. Videos    - 90 videos.    - Resolution = 180x144.    - Static camera.3. Properties    - Homogeneous outdoor backgrounds.                  C3D features extraction  **The features extraction procedure is overwhelming therefore it will be described in another notebook**                        - Videos -----> fc6 features (4069 dimension vector)  KTH features<jupyter_code>### features are pickled for efficiency
kth_data = pickle.load( open( pickle_path_kth, "rb" ) )
print("Classes = ",kth_data['classes'])
print("KTH train data shape: ", kth_data['train_data'].shape)
print("KTH test data shape: ", kth_data['test_data'].shape)
print("KTH train labels between 0-6 : ", kth_data['train_labels'][:20])
print("KTH test labels between 0-6 : ", kth_data['test_labels'][:20])<jupyter_output>Classes =  ['walking', 'jogging', 'running', 'boxing', 'handwaving', 'handclapping']
KTH train data shape:  (499, 4096)
KTH test data shape:  (101, 4096)
KTH train labels between 0-6 :  [1 0 1 1 0 5 2 0 0 0 0 1 3 3 0 3 1 3 5 4]
KTH test labels between 0-6 :  [2 4 2 0 3 1 1 1 1 0 3 4 0 3 3 3 4 0 2 3]<jupyter_text>WEIZMANN features<jupyter_code>### features are pickled for efficiency, same as before except weizmann dataset is used for testing
Weizmann_test_data=pickle.load( open(pickle_path_Weizmann, "rb" ) )
print(Weizmann_test_data)<jupyter_output>{'classes': ['run', 'walk', 'skip', 'jack', 'bend', 'wave', 'jump', 'side', 'pj'], 'test_labels': array([7, 4, 1, 7, 3, 7, 6, 3, 5, 8, 4, 0, 4, 2, 8, 7, 1, 4, 8, 2, 1, 5,
       1, 5, 5, 4, 5, 4, 5, 8, 8, 4, 3, 5, 5, 1, 6, 3, 6, 8, 8, 3, 7, 5,
       5, 7, 5, 3, 6, 2, 0, 0, 1, 5, 2, 2, 5, 1, 5, 2, 5, 5, 3, 4, 1, 3,
       1, 5, 6, 4, 0, 3, 8, 0, 0, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
       2, 2, 2, 2, 2]), 'test_data': array([[0.        , 0.        , 0.        , ..., 0.        , 0.6928104 ,
        0.        ],
       [0.        , 0.        , 0.17376561, ..., 0.        , 0.        ,
        0.        ],
       [0.        , 0.75278556, 0.        , ..., 1.09073   , 0.        ,
        0.        ],
       ...,
       [0.        , 0.        , 0.        , ..., 1.1108038 , 0.        ,
        0.        ],
       [0.        , 0.9212859 , 0.        , ..., 0.        , 0.        ,
        0.        ],
       [0.1273484 , 0.        , 0.        , ..., 0.        , 0.        ,
        0.        [...]<jupyter_text>Implementation with Tensorflow<jupyter_code>#### trainable variables
def _variable_on_cpu(name, shape, initializer,trainable):
  #with tf.device('/cpu:%d' % cpu_id):
  with tf.device('/gpu:0'):
    var = tf.get_variable(name, shape, initializer=initializer,trainable=trainable)
  return var

def _variable_with_weight_decay(name, shape, stddev, wd,trainable=False):
  var = _variable_on_cpu(name, shape, tf.truncated_normal_initializer(stddev=stddev),trainable=trainable)
  if wd is not None:
    weight_decay = tf.nn.l2_loss(var) * wd
    tf.add_to_collection('losses', weight_decay)
  return var
########## getting  KTH training  batchs ################################
def get_batch_train(batch_size,kth_data):
    train=kth_data["train_data"]
    train_label=kth_data["train_labels"]
    rand_indexes=[randint(0, train.shape[0]-1) for i in range(batch_size)]
    return train[rand_indexes],train_label[rand_indexes] 
########## getting WEIZMANN test data  #################
def get_all_test(kth_data):
    test=kth_data["test_data"]
    test_label=kth_data["test_labels"]
    rand_indexes=[randint(0, test.shape[0]-1) for i in range(test.shape[0])]
    return test[rand_indexes],test_label[rand_indexes]<jupyter_output><empty_output><jupyter_text>Training/Testing the Network<jupyter_code>def Siamese():
  _dropout=0.8  
  tf.reset_default_graph()
  ########### DATA KTH and WEIZMANN  
  kth_data = pickle.load( open( pickle_path_kth, "rb" ) )
  Weizmann_test_data=pickle.load( open(pickle_path_Weizmann, "rb" ) )
  ################ placeholders: Input data ######################  
  X_input = tf.placeholder(tf.float32, [None, 4096])
  Z_input = tf.placeholder(tf.float32, [None, 4096])
  Y = tf.placeholder(tf.float32, [None, 2])

  sess = tf.Session(config=tf.ConfigProto(allow_soft_placement=True))
 
#  ANN Variables

  W_1=_variable_with_weight_decay('W_1', [4096, 2000], 0.04, 0.005,trainable=True)
  b_1=_variable_with_weight_decay('b_1', [2000], 0.04, 0.0,trainable=True)
  
  W_2=_variable_with_weight_decay('W_2', [2000, 1000], 0.04, 0.005,trainable=True)
  b_2=_variable_with_weight_decay('b_2', [1000], 0.04, 0.0,trainable=True)
  
  W_3=_variable_with_weight_decay('W_3', [1000, 200], 0.04, 0.005,trainable=True)
  b_3=_variable_with_weight_decay('b_3', [200], 0.04, 0.0,trainable=True)
  
  
  W_4=_variable_with_weight_decay('W_4', [200, 2], 0.04, 0.005,trainable=True)
  b_4=_variable_with_weight_decay('b_4', [2], 0.04, 0.0,trainable=True)
 
  ### ANN operations ###
  
  l1=tf.add(tf.matmul(tf.abs(X_input-Z_input),W_1), b_1)   
  l1=tf.nn.relu(l1, 'relu11')
  l1=tf.nn.dropout(l1, _dropout)
  
  l2=tf.add(tf.matmul(l1,W_2), b_2)
  l2=tf.nn.relu(l2, 'relu22')
  l2=tf.nn.dropout(l2, _dropout)
  
  l3=tf.add(tf.matmul(l2,W_3), b_3)
  l3=tf.nn.relu(l3, 'relu33')
  l3=tf.nn.dropout(l3, _dropout)
  
  Siamese_output=tf.add(tf.matmul(l3,W_4), b_4)
  Siamese_output=tf.nn.relu(Siamese_output, 'relu44')

  loss_opp = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(
             logits=Siamese_output, labels=Y))
  prediction = tf.nn.softmax(Siamese_output)
  
  # Evaluate model
  correct_pred = tf.equal(tf.argmax(prediction, 1), tf.argmax(Y, 1))
  accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))

  optimizer = tf.train.AdamOptimizer(learning_rate=0.0001)
  train_op = optimizer.minimize(loss_opp)
  init = tf.global_variables_initializer()
  
  sess.run(init)

  loss_list=[]
  acc_list=[]
  acc_test_list=[]
  acc_test_list_weizmann=[]
  
  for step in xrange(4000):
    # generate random batches for training  
    X,X_label=get_batch_train(kth_data=kth_data,batch_size=150)
    Z,Z_label=get_batch_train(kth_data=kth_data,batch_size=150)
    
    A=Z_label==X_label
    y=[[1,0] if a else [0,1] for a in A ] 
    
    ######## calculate batch loss #######################
    loss=sess.run([loss_opp,train_op],  feed_dict={X_input: X, Z_input: Z, Y:y})
    ####### calculate batch accuracy #####################
    acc=sess.run([accuracy],  feed_dict={X_input: X, Z_input: Z, Y:y})
    
    loss_list.append(loss[0]) 
    acc_list.append(acc[0])
    
    print("loss  "+str(loss[0])+"  accuracy "+ str(acc[0]))
 
    
  X,X_label=get_all_test(kth_data=kth_data)
  Z,Z_label=get_all_test(kth_data=kth_data)
  A=Z_label==X_label
  y=[[1,0] if a else [0,1] for a in A ] 
  print("####### accuracy score on Kth loading########### ")          
  for i in range(20):
      acc_test=sess.run([accuracy],   feed_dict={X_input: X, Z_input: Z, Y:y})
      acc_test_list.append(acc_test[0])
      
  X,X_label=get_all_test(kth_data=Weizmann_test_data)
  Z,Z_label=get_all_test(kth_data=Weizmann_test_data)
  A=Z_label==X_label
  y=[[1,0] if a else [0,1] for a in A ] 
  print("####### accuracy score on Weizmann loading########### ")          
  for i in range(20):
      acc_test=sess.run([accuracy],   feed_dict={X_input: X, Z_input: Z, Y:y})
      acc_test_list_weizmann.append(acc_test[0])   
      
      
  ##### save the model #######    
    #  saver = tf.train.Saver() 
    #  saver.save(sess, '/home/jan/Desktop/Tensorflow/kth_model/C3D')  
  print("done with Accuracy on Kth test data "+ str(np.mean(acc_test_list)))
  print("done with Accuracy on Weizmann data "+ str(np.mean(acc_test_list_weizmann)))
  
  
  return loss_list,acc_list,acc_test_list,acc_test_list_weizmann
################# Results #######################################
loss_list,acc_list,acc_test_list,acc_test_list_weizmann=Siamese()<jupyter_output>WARNING:tensorflow:From <ipython-input-7-0cd2b5bbcc22>:46: softmax_cross_entropy_with_logits (from tensorflow.python.ops.nn_ops) is deprecated and will be removed in a future version.
Instructions for updating:

Future major versions of TensorFlow will allow gradients to flow
into the labels input on backprop by default.

See tf.nn.softmax_cross_entropy_with_logits_v2.

loss  0.64776886  accuracy 0.79333335
loss  0.5894172  accuracy 0.7266667
loss  0.5082188  accuracy 0.8066667
loss  0.5663003  accuracy 0.76
loss  0.60314614  accuracy 0.76666665
loss  0.5451954  accuracy 0.78
loss  0.6262056  accuracy 0.75333333
loss  0.56454784  accuracy 0.7266667
loss  0.5889228  accuracy 0.73333335
loss  0.6063364  accuracy 0.7133333
loss  0.594445  accuracy 0.72
loss  0.6053417  accuracy 0.7266667
loss  0.60582393  accuracy 0.7133333
loss  0.60562986  accuracy 0.6933333
loss  0.58090943  accuracy 0.74
loss  0.51562023  accuracy 0.8
loss  0.6127381  accuracy 0.7266667
loss  0.5346875  accuracy 0.78
[...]<jupyter_text>Accuracy score for both Datasets - **Done with Accuracy on Kth test data 0.7935643**- **Done with Accuracy on Weizmann data 0.87096775**  Plot accuracy and loss for KTH test during training<jupyter_code>plt.plot(range(len(loss_list)),loss_list)
plt.xlabel("iterations")
plt.ylabel("cross entropy loss")

plt.plot(range(len(acc_list)),acc_list)
plt.xlabel("iterations")
plt.ylabel("Accuracy")<jupyter_output><empty_output><jupyter_text>Test accuracy on WEIZMANN<jupyter_code>print(np.mean(acc_test_list_weizmann))<jupyter_output>0.87096775
{'fraction_non_alphanumeric': 0.14721183547390668, 'fraction_numerical': 0.06787514225329215, 'mean_word_length': 3.982989064398542, 'pattern_counts': {'":': 0, '<': 38, '<?xml version=': 0, '>': 41, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '477793', 'license': 'CECILL-B', 'n_tokens_mistral': 5191, 'n_tokens_neox': 4633, 'n_words': 1090, 'path': 'examples/C3D-Siamese.ipynb', 'repo_name': 'Wail13/SyncPy', 'size': '12302'}