<jupyter_start><jupyter_text>Node Representation Learning with attri2vec on Citeseer This is the python implementation of the attri2vec algorithm outlined in paper ***[Attributed Network Embedding Via Subspace Discovery](https://arxiv.org/abs/1901.04095)*** <NAME>, <NAME>, <NAME> and <NAME>, arXiv:1901.04095, [cs.SI], 2019. The implementation uses the stellargraph libraries. attri2vecattri2vec learns node representations by performing a linear/non-linear mapping on node content attributes. To make the learned node representations respect structural similarity, [`DeepWalk`](https://dl.acm.org/citation.cfm?id=2623732)/[`node2vec`](https://snap.stanford.edu/node2vec) learning mechanism is used to make nodes sharing similar random walk context nodes represented closely in the subspace, which is achieved by maximizing the occurrence probability of context nodes conditioned on the representation of the target nodes. The probability is modelled by Softmax and negative sampling is used to speed up its calculation. This makes attri2vec equivalent to predict whether a node occurs in the given target node's context in random walks with the representation of the target node, by minimizing the cross-entropy loss. In implementation, node embeddings are learnt by solving a simple classification task: given a large set of "positive" `(target, context)` node pairs generated from random walks performed on the graph (i.e., node pairs that co-occur within a certain context window in random walks), and an equally large set of "negative" node pairs that are randomly selected from the graph according to a certain distribution, learn a binary classifier that predicts whether arbitrary node pairs are likely to co-occur in a random walk performed on the graph. Through learning this simple binary node-pair-classification task, the model automatically learns an inductive mapping from attributes of nodes to node embeddings in a low-dimensional vector space, which preserves structural and feature similarities of the nodes. To train the attri2vec model, we first construct a training set of nodes, which is composed of an equal number of positive and negative `(target, context)` pairs from the graph. The positive `(target, context)` pairs are the node pairs co-occurring on random walks over the graph whereas the negative node pairs are the sampled randomly from the global node degree distribution of the graph. In attri2vec, each node is attached with two kinds of embeddings: 1) the inductive 'input embedding', i.e, the objective embedding, obtained by perform a linear/non-linear transformation on node content features, and 2) 'output embedding', i.e., the parameter vector used to predict its occurrence as a context node, obtained by looking up a parameter table. Given a `(target, context)` pair, attri2vec outputs a predictive value to indicate whether it is positive or negative, which is obtained by performing the dot product of the 'input embedding' of the target node and the 'output embedding' of the context node, followed by a sigmoid activation. The entire model is trained end-to-end by minimizing the binary cross-entropy loss function with regards to predicted node pair labels and true node pair labels, using stochastic gradient descent (SGD) updates of the model parameters, with minibatches of 'training' node pairs generated on demand and fed into the model.<jupyter_code>import networkx as nx
import pandas as pd
import numpy as np
import os
import random

import stellargraph as sg
from stellargraph.data import UnsupervisedSampler
from stellargraph.mapper import Attri2VecLinkGenerator, Attri2VecNodeGenerator
from stellargraph.layer import Attri2Vec, link_classification

from tensorflow import keras

from pandas.core.indexes.base import Index

import matplotlib.pyplot as plt
from sklearn.manifold import TSNE
from sklearn.decomposition import PCA

from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegressionCV
from sklearn.metrics import accuracy_score<jupyter_output><empty_output><jupyter_text>Dataset The dataset is the citation network Citeseer.It can be downloaded by clicking [here](https://linqs-data.soe.ucsc.edu/public/lbc/citeseer.tgz)The following is the description of the dataset from the publisher,> The CiteSeer dataset consists of 3312 scientific publications classified into one of six classes. The citation network consists of 4732 links. Each publication in the dataset is described by a 0/1-valued word vector indicating the absence/presence of the corresponding word from the dictionary. The dictionary consists of 3703 unique words. The README file in the dataset provides more details.Download and unzip the citeseer.tgz file to a location on your computer. We assume that the dataset is stored in the directory`~/data/citeseer/`where the files `citeseer.cites` and `citeseer.content` can be located.We are going to load the data into a networkx object.<jupyter_code>data_dir = "~/data/citeseer"<jupyter_output><empty_output><jupyter_text>Load edges in order 'cited-paper' <- 'citing-paper'.<jupyter_code>citeseer_location = os.path.expanduser(os.path.join(data_dir, "citeseer.cites"))
g_nx = nx.read_edgelist(path=citeseer_location, create_using=nx.DiGraph()).reverse()<jupyter_output><empty_output><jupyter_text>Convert the graph to undirected graph.<jupyter_code>g_nx = g_nx.to_undirected()<jupyter_output><empty_output><jupyter_text>Load the node attribute data.<jupyter_code>citeseer_data_location = os.path.expanduser(os.path.join(data_dir, "citeseer.content"))
attr_names = ["w_{}".format(ii) for ii in range(3703)]
node_column_names =  attr_names + ["subject"]
node_attr = pd.read_csv(citeseer_data_location, sep='\t', header=None, names=node_column_names)<jupyter_output><empty_output><jupyter_text>Change the type of the indexes of node_attr to str.<jupyter_code>node_attr.index = Index(list(map(str, list(node_attr.index))))<jupyter_output><empty_output><jupyter_text>The original graph contains some nodes with no attributes. We remove them here.<jupyter_code>g_nx = g_nx.subgraph(list(node_attr.index))<jupyter_output><empty_output><jupyter_text>Select the largest connected component. For clarity we ignore isolated nodes and subgraphs.<jupyter_code>g_nx_ccs = (g_nx.subgraph(c).copy() for c in nx.connected_components(g_nx))
g_nx = max(g_nx_ccs, key=len)
print("Largest subgraph statistics: {} nodes, {} edges".format(
    g_nx.number_of_nodes(), g_nx.number_of_edges()))<jupyter_output>Largest subgraph statistics: 2110 nodes, 3720 edges<jupyter_text>Specify node and edge types.<jupyter_code>nx.set_node_attributes(g_nx, "paper", "label")
nx.set_edge_attributes(g_nx, "cites", "label")<jupyter_output><empty_output><jupyter_text>Get the ids of the nodes in the selected largest connected component.<jupyter_code>node_ids = sorted(list(g_nx.nodes))<jupyter_output><empty_output><jupyter_text>Get node features.<jupyter_code>node_features = node_attr[attr_names].reindex(node_ids)<jupyter_output><empty_output><jupyter_text>Create the Stellargraph with node features.<jupyter_code>G = sg.StellarGraph(g_nx, node_features=node_features)
print(G.info())<jupyter_output>NetworkXStellarGraph: Undirected multigraph
 Nodes: 2110, Edges: 3720

 Node types:
  paper: [2110]
    Edge types: paper-cites->paper

 Edge types:
    paper-cites->paper: [3720]<jupyter_text>Train attri2vec on Citeseer Specify the other optional parameter values: root nodes, the number of walks to take per node, the length of each walk.<jupyter_code>nodes = list(G.nodes())
number_of_walks = 4
length = 5<jupyter_output><empty_output><jupyter_text>Create the UnsupervisedSampler instance with the relevant parameters passed to it.<jupyter_code>unsupervised_samples = UnsupervisedSampler(G, nodes=nodes, length=length, number_of_walks=number_of_walks)<jupyter_output><empty_output><jupyter_text>Set the batch size and the number of epochs.<jupyter_code>batch_size = 50
epochs = 4<jupyter_output><empty_output><jupyter_text>Define an attri2vec training generator, which generates a batch of (feature of target node, index of context node, label of node pair) pairs per iteration.<jupyter_code>generator = Attri2VecLinkGenerator(G, batch_size)
train_gen = generator.flow(unsupervised_samples)<jupyter_output><empty_output><jupyter_text>Building the model: a 1-hidden-layer node representation ('input embedding') of the `target` node and the parameter vector ('output embedding') for predicting the existence of `context node` for each `(target context)` pair, with a link classification layer performed on the dot product of the 'input embedding' of the `target` node and the 'output embedding' of the `context` node.Attri2Vec part of the model, with a 128-dimenssion hidden layer, no bias term and no normalization. (Normalization can be set to 'l2').<jupyter_code>layer_sizes = [128]
attri2vec = Attri2Vec(
    layer_sizes=layer_sizes, generator=generator, bias=False, normalize=None
)
# Build the model and expose input and output sockets of attri2vec, for node pair inputs:
x_inp, x_out = attri2vec.build()<jupyter_output><empty_output><jupyter_text>Use the link_classification function to generate the prediction, with the 'ip' edge embedding generation method and the 'sigmoid' activation, which actually performs the dot product of the 'input embedding' of the target node and the 'output embedding' of the context node followed by a sigmoid activation.<jupyter_code>prediction = link_classification(
    output_dim=1, output_act="sigmoid", edge_embedding_method='ip'
)(x_out)<jupyter_output>link_classification: using 'ip' method to combine node embeddings into edge embeddings<jupyter_text>Stack the Attri2Vec encoder and prediction layer into a Keras model, and specify the loss.<jupyter_code>model = keras.Model(inputs=x_inp, outputs=prediction)

model.compile(
    optimizer=keras.optimizers.Adam(lr=1e-3),
    loss=keras.losses.binary_crossentropy,
    metrics=[keras.metrics.binary_accuracy],
)<jupyter_output><empty_output><jupyter_text>Train the model.<jupyter_code>history = model.fit_generator(
    train_gen,
    epochs=epochs,
    verbose=2,
    use_multiprocessing=False,
    workers=1,
    shuffle=True,
)<jupyter_output>Epoch 1/4
1351/1351 - 33s - loss: 0.6828 - binary_accuracy: 0.5571
Epoch 2/4
1351/1351 - 41s - loss: 0.5447 - binary_accuracy: 0.7269
Epoch 3/4
1351/1351 - 38s - loss: 0.3652 - binary_accuracy: 0.8618
Epoch 4/4
1351/1351 - 37s - loss: 0.2667 - binary_accuracy: 0.9101<jupyter_text>Visualise Node Embeddings Build the node based model for predicting node representations from node content attributes with the learned parameters. Below a Keras model is constructed, with x_inp[0] as input and x_out[0] as output. Note that this model's weights are the same as those of the corresponding node encoder in the previously trained node pair classifier.<jupyter_code>x_inp_src = x_inp[0]
x_out_src = x_out[0]
embedding_model = keras.Model(inputs=x_inp_src, outputs=x_out_src)<jupyter_output><empty_output><jupyter_text>Get the node embeddings by applying the learned mapping function to node content features.<jupyter_code>node_gen = Attri2VecNodeGenerator(G, batch_size).flow(node_ids)
node_embeddings = embedding_model.predict_generator(node_gen, workers=1, verbose=1)<jupyter_output>43/43 [==============================] - 0s 3ms/step<jupyter_text>Get node subjects.<jupyter_code>node_targets = [ node_attr["subject"][node_id] for node_id in node_ids ]<jupyter_output><empty_output><jupyter_text>Transform the embeddings to 2d space for visualisation.<jupyter_code>transform = TSNE # PCA

trans = transform(n_components=2)
node_embeddings_2d = trans.fit_transform(node_embeddings)
# draw the embedding points, coloring them by the target label (paper subject)
alpha = 0.7
label_map = { l: i for i, l in enumerate(np.unique(node_targets)) }
node_colours = [ label_map[target] for target in node_targets ]

plt.figure(figsize=(7,7))
plt.axes().set(aspect="equal")
plt.scatter(node_embeddings_2d[:,0], 
            node_embeddings_2d[:,1], 
            c=node_colours, cmap="jet", alpha=alpha)
plt.title('{} visualization of node embeddings'.format(transform.__name__))
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08112745098039216, 'fraction_numerical': 0.01838235294117647, 'mean_word_length': 6.4232868405094, 'pattern_counts': {'":': 0, '<': 97, '<?xml version=': 0, '>': 99, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '24', 'dir': 'jupyter-structured-clean-dedup', 'id': '76526', 'license': 'Apache-2.0', 'n_tokens_mistral': 3795, 'n_tokens_neox': 3560, 'n_words': 1260, 'path': 'demos/embeddings/stellargraph-attri2vec-citeseer.ipynb', 'repo_name': 'zblumen/stellargraph', 'size': '12244'}