<jupyter_start><jupyter_text>Task IV - Classical Graph Neural Network (GNN) Part Problem StatementFor Task IV, you will use ParticleNet’s data for Quark/Gluon jet classification available here with its corresponding description.● Choose 2 Graph-based architectures of your choice to classify jets as being quarks or gluons. Provide a description on what considerations you have taken to project this point-cloud dataset to a set of interconnected nodes and edges.● Discuss the resulting performance of the 2 chosen architectures. DatasetThe dataset used in this project was actually used in  <NAME>, <NAME>, <NAME>, Energy Flow Networks: Deep Sets for Particle Jets, JHEP 01 (2019) 121, arXiv:1810.05165.It has 20 files consisting of jet data with each file having 100k samples.There are two arrays in each file    X: (100000,M,4), exactly 50k quark and 50k gluon jets, randomly sorted, where M is the max multiplicity of the jets in that file (other jets have been padded with zero-particles), and the features of each particle are its pt, rapidity, azimuthal angle, and pdgid.     y: (100000,), an array of labels for the jets where gluon is 0 and quark is 1. However due to hardware limitations, we use only the first file of the complete dataset i.e. 100k total samples of data in this project. Architecture & ApproachA jet is one of the most ubiquitous objects in proton-proton collision events at the LHC. In essence, a jet isa collimated spray of particles. In this task, the jet data is represented as a particle cloud analogous to the point cloud representation of 3D shapes used in computer vision. The permutation symmetry of the particle cloud makes it a natural and promising representation of jets.I studied the following Graph-based architectures: 1. ABCNet 2. ParticleNet                        3. ParticleNet Lite  4. Message Passing Neural Networks  Important Points    ABCNet[3] is an attention based features update approach which also consists of a GAPLayer(Graph attention pooling layer)    that transforms the feature vectors to a higher dimensional space and passes to the attention layer with attention coefficients.This is helpful for quark gluon tagging because the attention mechanism considers the local features required for better classification performance    ParticleNet and PartcileNet Lite have fundamentally similar architectures that use the power of EdgeConv to parameterize over local dependencies in the point-cloud representation and give better results that most other approaches.    Message Passing Neural Nets[4] uses Adjacency learning which is particularly important in applications where the input set may have some underlying geometric structure and hence can serve the purpose for quark gluon classificationAfter studying all of the above architectures briefly, it is evident that if the necessary computing resources are available, then ABCNet performs best with over 230k features(considerably less than ParticeNet ~360k features) because it uses an extra power of attention mechanisms.Message Passing Nets are a good option but they consider the structure to be permutation equivariant which affects classification performance. ParticleNet is a better option because it handles large unordered data efficiently considering permutation invariance between datapoints and techniques like EdgeConv(a permutationally symmetric operation on pointclouds) However, all the above architectures require heavy computing resources due to the high number of features they use for training. Project Approach ParticleNet, a CNN-like deep neural network architecture is used for jet tagging with particle cloud data.This project will follow mainly the reference [2] but with some changes to explore things that is not yet explored in the paper.However,regular CNNs are suitable for data with some uniform grid like ordering of images, while the particle cloud is an unordered, permutation invariant representation. EdgeConv, another convolution-like operation that exploits the local  neighbourhood information based on the k-nearest neighbouring particles for a particular particle is a good fit for point cloud data.![Screenshot%20%28150%29.png](attachment:Screenshot%20%28150%29.png)where the feature vectors of the neighbors, xij , are substituted by their differences from the central point xi and h¯Θ can be implemented as a multilayer perceptron(MLP) whose parameters are shared among all edges.An important thing to note is that ParticleNet uses DGCNN(Dynamic Graph Convolutional Neural Network), the word 'dynamic' here denotes that the EdgeConv operation maps a point cloud datapoint from original feature space to another latent space and updates the co-ordinates according to latent space which are then used in calculating the k-nearest neighbour distances[1].In other words, the proximity of points can be dynamically learned with EdgeConv operations. Due to this, the graph describing the point clouds are dynamically updated to reflect the changes in the edges, i.e., the neighbors of each point. Single EdgeConv block  This project implements two architectures :<!--  ParticleNet consists of three EdgeConv blocks. The first EdgeConv block uses the spatial coordinates of the particles in the pseudorapidity-azimuth space to compute the distances, while the subsequent blocks use the learned feature vectors as coordinates. --> 1. ParticleNet Lite (Dynamic)ParticleNet Lite consists of three EdgeConv blocks with hyperparameter-values less than that are used in ParticleNet.The first EdgeConv block uses the spatial coordinates of the particles in the pseudorapidity-azimuth space to compute the distances, while the subsequent blocks use the learned feature vectors as coordinates. 2. ParticleNet Lite (Static)This architecture is exactly the same as the dynamic ParticleNet Lite except the fact that there is no dynamic update of the co-ordinates of particles based on the learned feature vectors, i.e the co-ordinates are always in the pseudorapidity-azimuth space to compute the distances between them.Both networks use a global average pooling for aggregating over the values obtained from the EdgeConv blocks along with a fully connected network with 2 units and output layer with a softmax function. HyperparametersThe hyperparameters used are kept the same as implemented in the reference[2]k in k- nearest neighbours' distance measure: -->For both archectures, k=7Also, the channels per layer for both architectures: -->    for each EdgeConv block is (32, 32, 32), (64, 64, 64), respectively.The LR(learning rate) schedule is described later in the form of code.  Loading and prepocessing the dataset<jupyter_code>def data_preprocess(data_1,data_2,label_1,label_2):
    if(data_1.shape[1] != data_2.shape[1]):
        if(data_1.shape[1]>data_2.shape[1]):
            data_2=np.concatenate((data_2,np.zeros((data_2.shape[0],data_1.shape[1]-data_2.shape[1],data_2.shape[2]))),axis=1)
        if(data_2.shape[1]>data_1.shape[1]):
            data_1=np.concatenate((data_1,np.zeros((data_1.shape[0],data_2.shape[1]-data_1.shape[1],data_1.shape[2]))) ,axis=1)
            
    dataset_X = np.concatenate((data_1,data_2), axis=0)
    dataset_y = np.concatenate((label_1,label_2), axis=0)
    
    return dataset_X,dataset_y


data = np.load('./GSOC22_QMLHEP_task_dataset/QG_jets.npz',allow_pickle=True)
X_train = data['X']
y_train = data['y']
# for i in range(1):
#     data_temp = np.load('./GSOC22_QMLHEP_task_dataset/QG_jets_' + str(i+1) + '.npz', allow_pickle=True)
#     print('file '+str(i+1)+' is loaded')
#     X_temp=data_temp['X']
#     y_temp=data_temp['y']
#     X_train,y_train = data_preprocess(X_train,X_temp,y_train,y_temp)
#Transform the labels to one-hot format
y_train = tf.keras.utils.to_categorical(y_train)
#check
print(X_train.shape)
# X_train.shape[0] = Number of training examples
# X_train.shape[1] = Maximum number of particles in a jet
# X_train.shape[2] = Total number of features
print(y_train.shape)
from sklearn.utils import shuffle
from sklearn.model_selection import train_test_split
# Splitting the dataset into three parts- 80% training data, 10% validation data, 10% test data
X_train,y_train = shuffle(X_train, y_train, random_state=1024)
X_train,X_val,y_train,y_val = train_test_split(X_train,y_train,test_size=0.2,random_state=1024)
X_val,X_test,y_val,y_test= train_test_split(X_val,y_val,test_size=0.5,random_state=1024)

num_particles = X_train.shape[1]
#check
print(X_train.shape)
print(X_test.shape)
print(X_val.shape)
print(y_train.shape)
train_dict = {
    #the co-ordinates of particles in (rapidity, azimuthal angle) space
    'points' : X_train[:,:,1:3],
    #the features of each particle are its pt, rapidity, azimuthal angle
    'features' : X_train,
    # a mask array, taking a value of 0 for padded positions (shape=num_samples,num_particles,1)
    'mask': np.sum(X_train, axis=2).reshape(len(X_train), num_particles, 1)
    }

val_dict = {
    #the co-ordinates of particles in (rapidity, azimuthal angle) space
    'points' : X_val[:,:,1:3],
    #the features of each particle are its pt, rapidity, azimuthal angle
    'features' : X_val,
    # a mask array, taking a value of 0 for padded positions (shape=num_samples,num_particles,1)
    'mask': np.sum(X_val, axis=2).reshape(len(X_val), num_particles, 1)
    }

test_dict = {
    #the co-ordinates of particles in (rapidity, azimuthal angle) space
    'points' : X_test[:,:,1:3],
    #the features of each particle are its pt, rapidity, azimuthal angle
    'features' : X_test,
    # a mask array, taking a value of 0 for padded positions (shape=num_samples,num_particles,1)
    'mask': np.sum(X_test, axis=2).reshape(len(X_test), num_particles, 1)
    }<jupyter_output><empty_output><jupyter_text>Initializing the Models<jupyter_code>from tensorflow import keras
from tf_keras_model import get_particle_net_lite
from tf_keras_model_static import get_particle_net_lite_static
num_classes = 2
input_shapes = {k:train_dict[k].shape[1:] for k in train_dict}
print(input_shapes)
model_lite_static = get_particle_net_lite_static(num_classes,input_shapes)
model_lite = get_particle_net_lite(num_classes,input_shapes)<jupyter_output><empty_output><jupyter_text>Hyperparameters<jupyter_code>batch_size = 300
batch_size_lite = 1200

epochs = 20
# learning rate schedule
def lr_schedule(epochs):
    learning_rate = 0.001
    if epochs>9:
        learning_rate*=0.1
    if epochs>18:
        learning_rate*=0.01
    logging.info('Learning rate: %f'%learning_rate)
    return learning_rate<jupyter_output><empty_output><jupyter_text>I. ParticleNet Lite model (Dynamic)<jupyter_code>model_lite.compile(loss = 'categorical_crossentropy',
             optimizer=keras.optimizers.Adam(learning_rate=lr_schedule(0)),
             metrics=['accuracy',tf.keras.metrics.AUC()])
model_lite.summary()
lr_scheduler=keras.callbacks.LearningRateScheduler(lr_schedule)
progress_bar = keras.callbacks.ProgbarLogger()
callbacks=[lr_scheduler,progress_bar]
print(train_dict['points'].shape)
print(train_dict['features'].shape)
print(train_dict['mask'].shape)<jupyter_output>(80000, 139, 2)
(80000, 139, 4)
(80000, 139, 1)<jupyter_text>Training the model<jupyter_code>history_dynamic = model_lite.fit(train_dict,
                   y_train, 
                   epochs=epochs,
                   batch_size=batch_size,
                   callbacks=callbacks,
                   validation_data=(val_dict,y_val),
                   shuffle=True)
#  import csv

# w = csv.writer(open("./train_history_dgcnn.csv", "w"))
# for key, val in history_dynamic.history.items():
#     w.writerow([key, val])<jupyter_output><empty_output><jupyter_text>AUC<jupyter_code>plt.plot(np.linspace(1,epochs,epochs, dtype=int), history_dynamic.history['auc_2'], color='green', label='Training AUC')
plt.plot(np.linspace(1,epochs,epochs, dtype=int), history_dynamic.history['val_auc_2'], color='red', label='Validation AUC')
plt.xlabel("Epochs")
plt.ylabel("AUC")
plt.title("AUC of Static Graph CNN Model")
plt.legend()
plt.show()<jupyter_output><empty_output><jupyter_text>Accuracy<jupyter_code>plt.plot(np.linspace(1,epochs,epochs, dtype=int), history_dynamic.history['accuracy'], color='green', label='Training AUC')
plt.plot(np.linspace(1,epochs,epochs, dtype=int), history_dynamic.history['val_accuracy'], color='red', label='Validation AUC')
plt.xlabel("Epochs")
plt.ylabel("Accuracy")
plt.title("Accuracy of Static Graph CNN Model")
plt.legend()
plt.show()<jupyter_output><empty_output><jupyter_text>Loss<jupyter_code>plt.plot(np.linspace(1,epochs,epochs, dtype=int), history_dynamic.history['loss'], color='green', label='Training AUC')
plt.plot(np.linspace(1,epochs,epochs, dtype=int), history_dynamic.history['val_loss'], color='red', label='Validation AUC')
plt.xlabel("Epochs")
plt.ylabel("Loss")
plt.title("Loss of Static Graph CNN Model")
plt.legend()
plt.show()<jupyter_output><empty_output><jupyter_text>Evaluation<jupyter_code>loss, acc, auc = model_lite.evaluate(test_dict, y_test)
print("Evaluation Metric of Dynamic Graph CNN Model on Test Dataset")
print("Loss:", loss)
print("Accuracy:", acc)
print("AUC:", auc)<jupyter_output>313/313 [==============================] - 16s 51ms/step - loss: 0.4262 - accuracy: 0.8119 - auc_2: 0.8859
Evaluation Metric of Dynamic Graph CNN Model on Test Dataset
Loss: 0.42618003487586975
Accuracy: 0.8119000196456909
AUC: 0.8858910799026489<jupyter_text>II. ParticleNet Lite model (static)<jupyter_code>model_lite_static.compile(loss = 'categorical_crossentropy',
             optimizer=keras.optimizers.Adam(learning_rate=lr_schedule(0)),
             metrics=['accuracy',tf.keras.metrics.AUC()])
model_lite_static.summary()
lr_scheduler=keras.callbacks.LearningRateScheduler(lr_schedule)
progress_bar = keras.callbacks.ProgbarLogger()
callbacks=[lr_scheduler,progress_bar]
print(train_dict['points'].shape)
print(train_dict['features'].shape)
print(train_dict['mask'].shape)<jupyter_output>(80000, 139, 2)
(80000, 139, 4)
(80000, 139, 1)<jupyter_text>Training the model<jupyter_code>history = model_lite_static.fit(train_dict,
                   y_train, 
                   epochs=epochs,
                   batch_size=batch_size,
                   callbacks=callbacks,
                   validation_data=(val_dict,y_val),
                   shuffle=True)
#  import csv

# w = csv.writer(open("./train_history_sgcnn.csv", "w"))
# for key, val in history.history.items():
#     w.writerow([key, val])<jupyter_output><empty_output><jupyter_text>AUC<jupyter_code>plt.plot(np.linspace(1,epochs,epochs, dtype=int), history.history['auc'], color='green', label='Training AUC')
plt.plot(np.linspace(1,epochs,epochs, dtype=int), history.history['val_auc'], color='red', label='Validation AUC')
plt.xlabel("Epochs")
plt.ylabel("AUC")
plt.title("AUC of Dynamic Graph CNN Model")
plt.legend()
plt.show()<jupyter_output><empty_output><jupyter_text>Accuracy<jupyter_code>plt.plot(np.linspace(1,epochs,epochs, dtype=int), history.history['accuracy'], color='green', label='Training AUC')
plt.plot(np.linspace(1,epochs,epochs, dtype=int), history.history['val_accuracy'], color='red', label='Validation AUC')
plt.xlabel("Epochs")
plt.ylabel("Accuracy")
plt.title("Accuracy of Dynamic Graph CNN Model")
plt.legend()
plt.show()<jupyter_output><empty_output><jupyter_text>Loss<jupyter_code>plt.plot(np.linspace(1,epochs,epochs, dtype=int), history.history['loss'], color='green     ', label='Training AUC')
plt.plot(np.linspace(1,epochs,epochs, dtype=int), history.history['val_loss'], color='red', label='Validation AUC')
plt.xlabel("Epochs")
plt.ylabel("Loss")
plt.title("Loss of Dynamic Graph CNN Model")
plt.legend()
plt.show()<jupyter_output><empty_output><jupyter_text>Evaluation<jupyter_code>loss, acc, auc = model_lite_static.evaluate(test_dict, y_test)
print("Evaluation Metric of Dynamic Graph CNN Model on Test Dataset")
print("Loss:", loss)
print("Accuracy:", acc)
print("AUC:", auc)<jupyter_output>313/313 [==============================] - 10s 32ms/step - loss: 0.4192 - accuracy: 0.8115 - auc: 0.8899
Evaluation Metric of Dynamic Graph CNN Model on Test Dataset
Loss: 0.41921645402908325
Accuracy: 0.8115000128746033
AUC: 0.8898574113845825<jupyter_text>Results & Conclusion<jupyter_code># rows = 
columns = ['Train', 'Validation', 'Test']
Metrics = ['Accuracy', 'AUC', 'Loss']
dynamic_results={
                'Train':['0.8153','0.8910','0.4175'],
                'Validation':['0.8095','0.8862','0.4264'],
                'Test':['0.8119','0.8858','0.4261']}
static_results={'Train':['0.8182','0.8942','0.4116'],
               'Validation':['0.8124','0.8096','0.4183'],
               'Test':['0.8115','0.8899','0.4192']}

results_dynamic=pd.DataFrame(dynamic_results,index=Metrics)
results_static=pd.DataFrame(static_results,index=Metrics)
print('ParticleNet Lite (Dynamic)')
print(results_dynamic)
print('ParticleNet Lite (Static)')
print(results_static)<jupyter_output>ParticleNet Lite (Dynamic)
           Train Validation    Test
Accuracy  0.8153     0.8095  0.8119
AUC       0.8910     0.8862  0.8858
Loss      0.4175     0.4264  0.4261
ParticleNet Lite (Static)
           Train Validation    Test
Accuracy  0.8182     0.8124  0.8115
AUC       0.8942     0.8096  0.8899
Loss      0.4116     0.4183  0.4192
{'fraction_non_alphanumeric': 0.10430406108989934, 'fraction_numerical': 0.03586717574916117, 'mean_word_length': 5.297632058287796, 'pattern_counts': {'":': 0, '<': 64, '<?xml version=': 0, '>': 70, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '259994', 'license': 'MIT', 'n_tokens_mistral': 5865, 'n_tokens_neox': 5259, 'n_words': 1604, 'path': 'QMLHEP_testquestion_4.ipynb', 'repo_name': 'Amey-2002/QMLHEP', 'size': '17304'}