<jupyter_start><jupyter_text>------      Higher Order Tutorial on Deep Learning    Graph Convolution Networks    1.0.0 - Graph Classification  ------  Keras DGL - Node Classification:  `tl;dr:  MutliGraphCNN(output_dim, num_filters)([X,Adj])`Importing: ```pythonfrom keras_dgl.layers import MutliGraphCNN```Just like any keras model: ```pythonoutput = MultiGraphCNN(100, num_filters, activation='elu')([X, Adj])output = MultiGraphCNN(100, num_filters, activation='elu')([output, Adj])output = Lambda(lambda x: K.mean(x, axis=1))(output)  ``` ------  Graph Neural Networks Mathematically, the GCN model follows this formula:$H^{(l+1)} = \sigma(\tilde{D}^{-\frac{1}{2}}\tilde{A}\tilde{D}^{-\frac{1}{2}}H^{(l)}W^{(l)})$Here, $H^{(l)}$ denotes the $l^{th}$ layer in the network,$\sigma$ is the non-linearity, and $W$ is the weight matrix forthis layer. $D$ and $A$, as commonly seen, represent degreematrix and adjacency matrix, respectively. The ~ is a renormalization trickin which we add a self-connection to each node of the graph, and build thecorresponding degree and adjacency matrix.  The shape of the input$H^{(0)}$ is $N \times D$, where $N$ is the number of nodesand $D$ is the number of input features. We can chain up multiplelayers as such to produce a node-level representation output with shape$N \times F$, where $F$ is the dimension of the output nodefeature vector.The equation can be efficiently implemented using sparse matrixmultiplication kernels (such as Kipf's`https://github.com/tkipf/pygcn`). The above DGL implementationin fact has already used this trick due to the use of builtin functions. Tounderstand what is under the hood, please read the tutorial on page rank specified in this repository.__References__: [1] Kipf, <NAME>., and <NAME>. "Semi-supervised classification with graph convolutional networks." arXiv preprint arXiv:1609.02907 (2016). [2] Defferrard, Michaël, <NAME>, and <NAME>. "Convolutional neural networks on graphs with fast localized spectral filtering." In Advances in Neural Information Processing Systems, pp. 3844-3852. 2016. [3] Simonovsky, Martin, and <NAME>. "Dynamic edge-conditioned filters in convolutional neural networks on graphs." In Proc. CVPR. 2017.<jupyter_code>%%bash
if [ ! -d "keras-deep-graph-learning" ] ; then git clone https://github.com/ypeleg/keras-deep-graph-learning; fi
from tachles import fix_gcn_paths, load_mutag
fix_gcn_paths()
import keras_dgl
from keras_dgl.layers import MultiGraphCNN, MultiGraphAttentionCNN
from examples.utils import normalize_adj_numpy, evaluate_preds, preprocess_edge_adj_tensor<jupyter_output><empty_output><jupyter_text>The MUTAG DatasetThe MUTAG dataset is distributed baseline dataset for graph learning. It contains information about 340 complex molecules that are potentially carcinogenic, which is given by the isMutagenic property.The molecules can be classified as “mutagenic” or “not mutagenic”.<jupyter_code>A, A_orig, X, Y, num_edge_features, num_graph_nodes, num_graphs, orig_num_graph_nodes, orig_num_graphs = load_mutag()
print X.shape, Y.shape, A.shape
import keras.backend as K
import numpy as np
import networkx as nx
import matplotlib.pyplot as plt

from keras.layers import Dense, Activation, Dropout, Input, Lambda
from keras.models import Model, Sequential
from keras.callbacks import Callback
from keras.regularizers import l2
from keras.optimizers import Adam
def plot_graph(adjacency_matrix):
    rows, cols = np.where(adjacency_matrix == 1)
    edges = zip(rows.tolist(), cols.tolist())
    gr = nx.Graph()
    gr.add_edges_from(edges)
    fig, ax = plt.subplots(1, 1, figsize=(6, 6))
    nx.draw_networkx(gr, ax=ax, with_labels=False, node_size=5, width=.5)
    ax.set_axis_off()
    plt.show()
    plt.close()
print X[0]
# plot_graph(A)<jupyter_output>[[0 0 1 0 0 0 0]
 [0 0 1 0 0 0 0]
 [0 0 1 0 0 0 0]
 [0 0 1 0 0 0 0]
 [0 0 1 0 0 0 0]
 [0 0 1 0 0 0 0]
 [0 0 1 0 0 0 0]
 [0 0 1 0 0 0 0]
 [0 0 1 0 0 0 0]
 [0 0 1 0 0 0 0]
 [0 0 0 0 0 1 0]
 [0 0 0 0 0 0 1]
 [0 0 0 0 0 0 1]
 [0 0 0 0 0 0 0]
 [0 0 0 0 0 0 0]
 [0 0 0 0 0 0 0]
 [0 0 0 0 0 0 0]
 [0 0 0 0 0 0 0]
 [0 0 0 0 0 0 0]
 [0 0 0 0 0 0 0]
 [0 0 0 0 0 0 0]
 [0 0 0 0 0 0 0]
 [0 0 0 0 0 0 0]
 [0 0 0 0 0 0 0]
 [0 0 0 0 0 0 0]
 [0 0 0 0 0 0 0]
 [0 0 0 0 0 0 0]
 [0 0 0 0 0 0 0]]<jupyter_text>----[[source]](https://github.com/vermaMachineLearning/keras-deep-graph-learning/blob/master/keras_dgl/layers/multi_graph_cnn_layer.pyL9) MutliGraphCNN```pythonMutliGraphCNN(output_dim, num_filters, activation=None, use_bias=True, kernel_initializer='glorot_uniform', bias_initializer='zeros', kernel_regularizer=None, bias_regularizer=None, activity_regularizer=None, kernel_constraint=None, bias_constraint=None)```MutliGraphCNN assumes that the number of nodes for each graph in the dataset is same. For graph with arbitrary size, one can simply append appropriate zero rows or columns in adjacency matrix (and node feature matrix) based on max graph size in the dataset to achieve this uniformity.__Arguments__- __output_dim__: Positive integer, dimensionality of each graph node feature output space (or also referred dimension of graph node embedding).- __num_filters__: Positive integer, number of graph filters used for constructing  __graph_conv_filters__ input.- __activation__: Activation function to use.If you don't specify anything, no activation is applied(ie. "linear" activation: `a(x) = x`).- __use_bias__: Boolean, whether the layer uses a bias vector.- __kernel_initializer__: Initializer for the `kernel` weights matrix.- __bias_initializer__: Initializer for the bias vector.- __kernel_regularizer__: Regularizer function applied tothe `kernel` weights matrix.- __bias_regularizer__: Regularizer function applied to the bias vector.- __activity_regularizer__: Regularizer function applied tothe output of the layer (its "activation")..- __kernel_constraint__: Constraint function applied to the kernel matrix.- __bias_constraint__: Constraint function applied to the bias vector.__Input shapes__* __graph node feature matrix__ input as a 3D tensor with shape: `(batch_size, num_graph_nodes, input_dim)` corresponding to graph node input feature matrix for each graph.* __graph_conv_filters__ input as a 3D tensor with shape: `(batch_size, num_filters*num_graph_nodes, num_graph_nodes)` `num_filters` is different number of graph convolution filters to be applied on graph. For instance `num_filters` could be power of graph Laplacian.__Output shape__* 3D tensor with shape: `(batch_size, num_graph_nodes, output_dim)`	representing convoluted output graph node embedding matrix for each graph in batch size.[[source]](https://github.com/vermaMachineLearning/keras-deep-graph-learning/blob/master/examples/multi_gcnn_graph_classification_example.py)  The model itself<jupyter_code>num_filters = num_edge_features
graph_conv_filters = preprocess_edge_adj_tensor(A, symmetric=True)
X_input = Input(shape=(X.shape[1], X.shape[2]))
graph_conv_filters_input = Input(shape=(graph_conv_filters.shape[1], graph_conv_filters.shape[2]))

output = MultiGraphCNN(100, num_filters, activation='elu')([X_input, graph_conv_filters_input])
output = Dropout(0.2)(output)
output = MultiGraphCNN(100, num_filters, activation='elu')([output, graph_conv_filters_input])
output = Dropout(0.2)(output)
output = Lambda(lambda x: K.mean(x, axis=1))(output)  
output = Dense(Y.shape[1])(output)
output = Activation('softmax')(output)

nb_epochs = 200
batch_size = 169

model = Model(inputs=[X_input, graph_conv_filters_input], outputs=output)
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['acc'])
model.summary()
model.fit([X, graph_conv_filters], Y, batch_size=batch_size, validation_split=0.1, epochs=nb_epochs, shuffle=True, verbose=1)<jupyter_output>Train on 169 samples, validate on 19 samples
Epoch 1/200
169/169 [==============================] - 0s 2ms/step - loss: 1.1826 - acc: 0.3254 - val_loss: 0.6614 - val_acc: 0.5789
Epoch 2/200
169/169 [==============================] - 0s 405us/step - loss: 0.5530 - acc: 0.6746 - val_loss: 0.9288 - val_acc: 0.5789
Epoch 3/200
169/169 [==============================] - 0s 401us/step - loss: 0.6636 - acc: 0.6746 - val_loss: 1.0836 - val_acc: 0.5789
Epoch 4/200
169/169 [==============================] - 0s 403us/step - loss: 0.7484 - acc: 0.6746 - val_loss: 1.0878 - val_acc: 0.5789
Epoch 5/200
169/169 [==============================] - 0s 424us/step - loss: 0.7366 - acc: 0.6746 - val_loss: 0.9995 - val_acc: 0.5789
Epoch 6/200
169/169 [==============================] - 0s 448us/step - loss: 0.6828 - acc: 0.6746 - val_loss: 0.8698 - val_acc: 0.5789
Epoch 7/200
169/169 [==============================] - 0s 403us/step - loss: 0.6029 - acc: 0.6746 - val_loss: 0.7431 - val_acc: 0.5789
Epoch 8/200
[...]<jupyter_text>Your Turn!  Run The same but this time with Attention CGNN!  MultiGraphAttentionCNN```pythonMutliGraphCNN(output_dim, num_filters, activation=None, use_bias=True, kernel_initializer='glorot_uniform', bias_initializer='zeros', kernel_regularizer=None, bias_regularizer=None, activity_regularizer=None, kernel_constraint=None, bias_constraint=None)```MutliGraphCNN assumes that the number of nodes for each graph in the dataset is same. For graph with arbitrary size, one can simply append appropriate zero rows or columns in adjacency matrix (and node feature matrix) based on max graph size in the dataset to achieve this uniformity.__Arguments__- __output_dim__: Positive integer, dimensionality of each graph node feature output space (or also referred dimension of graph node embedding).- __num_filters__: Positive integer, number of graph filters used for constructing  __graph_conv_filters__ input.- __activation__: Activation function to use.If you don't specify anything, no activation is applied- __use_bias__: Boolean, whether the layer uses a bias vector.- __kernel_initializer__: Initializer for the `kernel` weights matrix- __bias_initializer__: Initializer for the bias vector- __kernel_regularizer__: Regularizer function applied tothe `kernel` weights matrix- __bias_regularizer__: Regularizer function applied to the bias vector- __activity_regularizer__: Regularizer function applied tothe output of the layer (its "activation").- __kernel_constraint__: Constraint function applied to the kernel matrix- __bias_constraint__: Constraint function applied to the bias vector__Input shapes__* __graph node feature matrix__ input as a 3D tensor with shape: `(batch_size, num_graph_nodes, input_dim)` corresponding to graph node input feature matrix for each graph.* __graph_conv_filters__ input as a 3D tensor with shape: `(batch_size, num_filters*num_graph_nodes, num_graph_nodes)` `num_filters` is different number of graph convolution filters to be applied on graph. For instance `num_filters` could be power of graph Laplacian.__Output shape__* 3D tensor with shape: `(batch_size, num_graph_nodes, output_dim)`	representing convoluted output graph node embedding matrix for each graph in batch size.[[source]](https://github.com/vermaMachineLearning/keras-deep-graph-learning/blob/master/examples/multi_graph_attention_cnn_graph_classification_example.py)<jupyter_code>num_filters = 2
print A.shape
A_eye_tensor = []
for _ in range(orig_num_graphs):
    Identity_matrix = np.eye(orig_num_graph_nodes)
    A_eye_tensor.append(Identity_matrix)

A_eye_tensor = np.array(A_eye_tensor)
A_orig = np.add(A_orig, A_eye_tensor)
graph_conv_filters = preprocess_edge_adj_tensor(A_orig, symmetric=True)
# build model
X_input = Input(shape=(X.shape[1], X.shape[2]))
A_input = Input(shape=(A_orig.shape[1], A_orig.shape[2]))
graph_conv_filters_input = Input(shape=(graph_conv_filters.shape[1], graph_conv_filters.shape[2]))

output = MultiGraphAttentionCNN(100, num_filters=num_filters, num_attention_heads=2, attention_combine='concat', attention_dropout=0.5, activation='elu', kernel_regularizer=l2(5e-4))([X_input, A_input, graph_conv_filters_input])
output = Dropout(0.2)(output)
output = MultiGraphAttentionCNN(100, num_filters=num_filters, num_attention_heads=1, attention_combine='average', attention_dropout=0.5, activation='elu', kernel_regularizer=l2(5e-4))([output, A_input, graph_conv_filters_input])
output = Dropout(0.2)(output)
output = Lambda(lambda x: K.mean(x, axis=1))(output)  # adding a node invariant layer to make sure output does not depends upon the node order in a graph.
output = Dense(Y.shape[1], activation='elu')(output)
output = Activation('softmax')(output)

nb_epochs = 500
batch_size = 169

model = Model(inputs=[X_input, A_input, graph_conv_filters_input], outputs=output)
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['acc'])
model.fit([X, A_orig, graph_conv_filters], Y, batch_size=batch_size, validation_split=0.1, epochs=nb_epochs, shuffle=True, verbose=1)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11649122807017544, 'fraction_numerical': 0.044678362573099414, 'mean_word_length': 5.7398843930635834, 'pattern_counts': {'":': 0, '<': 20, '<?xml version=': 0, '>': 20, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '255379', 'license': 'Unlicense', 'n_tokens_mistral': 4511, 'n_tokens_neox': 3982, 'n_words': 1370, 'path': '1.0.0 - Graph Convolution Networks - Graph Classification.ipynb', 'repo_name': 'harashm/ExpertDL', 'size': '12863'}