<commit_before>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Inductive Node Representation Learning through attri2vec"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the python implementation of the attri2vec algorithm outlined in paper ***[Attributed Network Embedding Via Subspace Discovery](https://arxiv.org/abs/1901.04095)*** <NAME>, <NAME>, <NAME> and <NAME>, arXiv:1901.04095, [cs.SI], 2019. The implementation uses the stellargraph libraries.\n",
    "\n",
    "## attri2vec\n",
    "\n",
    "attri2vec learns node representations by performing a linear/non-linear mapping on node content attributes. To make the learned node representations respect structural similarity, [`DeepWalk`](https://dl.acm.org/citation.cfm?id=2623732)/[`node2vec`](https://snap.stanford.edu/node2vec) learning mechanism is used to make nodes sharing similar random walk context nodes represented closely in the subspace, which is achieved by maximizing the occurrence probability of context nodes conditioned on the representation of the target nodes. The probability is modelled by Softmax and negative sampling is used to speed up its calculation. This makes attri2vec equivalent to predict whether a node occurs in the given target node's context in random walks with the representation of the target node, by minimizing the cross-entropy loss. \n",
    "\n",
    "In implementation, node embeddings are learnt by solving a simple classification task: given a large set of \"positive\" `(target, context)` node pairs generated from random walks performed on the graph (i.e., node pairs that co-occur within a certain context window in random walks), and an equally large set of \"negative\" node pairs that are randomly selected from the graph according to a certain distribution, learn a binary classifier that predicts whether arbitrary node pairs are likely to co-occur in a random walk performed on the graph. Through learning this simple binary node-pair-classification task, the model automatically learns an inductive mapping from attributes of nodes to node embeddings in a low-dimensional vector space, which preserves structural and feature similarities of the nodes.\n",
    "\n",
    "To train the attri2vec model, we first construct a training set of nodes, which is composed of an equal number of positive and negative `(target, context)` pairs from the graph. The positive `(target, context)` pairs are the node pairs co-occurring on random walks over the graph whereas the negative node pairs are the sampled randomly from the global node degree distribution of the graph. In attri2vec, each node is attached with two kinds of embeddings: 1) the inductive 'input embedding', i.e, the objective embedding, obtained by perform a non-linear transformation on node content features, and 2) 'output embedding', i.e., the parameter vector used to predict its occurrence as a context node, obtained by looking up a parameter table. Given a `(target, context)` pair, attri2vec outputs a predictive value to indicate whether it is positive or negative, which is obtained by performing the dot product of the 'input embedding' of the target node and the 'output embedding' of the context node, followed by a sigmoid activation. \n",
    "\n",
    "The entire model is trained end-to-end by minimizing the binary cross-entropy loss function with regards to predicted node pair labels and true node pair labels, using stochastic gradient descent (SGD) updates of the model parameters, with minibatches of 'training' node pairs generated on demand and fed into the model.\n",
    "\n",
    "In this demo, we first train the attri2vec model on the in-sample subgraph and obtain a mapping function from node attributes to node representations, then apply the mapping function to the content attributes of out-of-sample nodes and obtain the representations of out-of-sample nodes. We evaluate the quality of inferred out-of-sample node representations by using it to predict the links of out-of-sample nodes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import networkx as nx\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import os\n",
    "import random\n",
    "\n",
    "import stellargraph as sg\n",
    "from stellargraph.data import UnsupervisedSampler\n",
    "from stellargraph.mapper import Attri2VecLinkGenerator, Attri2VecNodeGenerator\n",
    "from stellargraph.layer import Attri2Vec, link_classification\n",
    "\n",
    "from tensorflow import keras\n",
    "\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.metrics import roc_auc_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading DBLP network data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This demo uses a DBLP citation network, a subgraph extracted from [DBLP-Citation-network V3](https://aminer.org/citation). To form this subgraph, papers from four subjects are extracted according to their venue information: *Database, Data Mining, Artificial Intelligence and Computer Vision*, and papers with no citations are removed. The DBLP network contains 18,448 papers and 45,661 citation relations. From paper titles, we construct 2,476-dimensional binary node feature vectors, with each element indicating the presence/absence of the corresponding word. By ignoring the citation direction, we take the DBLP subgraph as an undirected network.\n",
    "\n",
    "As papers in DBLP are attached with publication year, the DBLP network with the dynamic property can be used to study the problem of out-of-sample node representation learning. From the DBLP network, we construct four in-sample subgraphs using papers published before 2006, 2007, 2008 and 2009, and denote the four subgraphs as DBLP2006, DBLP2007, DBLP2008, and DBLP2009. For each subgraph, the remaining papers are taken as out-of-sample nodes. We consider the case where new coming nodes have no links. We predict the links of out-of-sample nodes using the learned out-of-sample node representations and compare its performance with the node content feature baseline.\n",
    "\n",
    "The dataset used in this demo can be downloaded from https://www.kaggle.com/daozhang/dblp-subgraph.\n",
    "The following is the description of the dataset:\n",
    "\n",
    "> The content.txt file contains descriptions of the papers in the following format:\n",
    "\n",
    " \t\t<paper_id> <word_attributes> <class_label> <publication_year>\n",
    "        \n",
    "> The first entry in each line contains the unique integer ID (ranging from 0 to 18,447) of the paper followed by binary values indicating whether each word in the vocabulary is present (indicated by 1) or absent (indicated by 0) in the paper. Finally, the last two entries in the line are the class label and the publication year of the paper.\n",
    "> The edgeList.txt file contains the citation relations. Each line describes a link in the following format:\n",
    "\t\t\n",
    "        <ID of paper1> <ID of paper2>\n",
    "        \n",
    "> Each line contains two paper IDs, with paper2 citing paper1 or paper1 citing paper2.\n",
    "\n",
    "\n",
    "Download and unzip the dblp-subgraph.zip file to a location on your computer and set the `data_dir` variable to\n",
    "point to the location of the dataset (the \"DBLP\" directory containing \"content.txt\" and \"edgeList.txt\")."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_dir = \"~/data/DBLP\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the graph from the edgelist."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "edgelist = pd.read_csv(os.path.join(data_dir, \"edgeList.txt\"), sep='\\t', header=None, names=[\"source\", \"target\"])\n",
    "edgelist[\"label\"] = \"cites\"  # set the edge type"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load paper content features, subjects and publishing years."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "feature_names = [\"w_{}\".format(ii) for ii in range(2476)]\n",
    "node_column_names =  feature_names + [\"subject\", \"year\"]\n",
    "node_data = pd.read_csv(os.path.join(data_dir, \"content.txt\"), sep='\\t', header=None, names=node_column_names)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Construct the whole graph from edge list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "G_all_nx = nx.from_pandas_edgelist(edgelist, edge_attr=\"label\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Specify node types."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "nx.set_node_attributes(G_all_nx, \"paper\", \"label\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get node features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_node_features = node_data[feature_names]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create the Stellargraph with node features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "G_all = sg.StellarGraph(G_all_nx, node_features=all_node_features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "StellarGraph: Undirected multigraph\n",
      " Nodes: 18448, Edges: 45611\n",
      "\n",
      " Node types:\n",
      "  paper: [18448]\n",
      "    Edge types: paper-cites->paper\n",
      "\n",
      " Edge types:\n",
      "    paper-cites->paper: [45611]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(G_all.info())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get DBLP Subgraph \n",
    "### with papers published before a threshold year"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the edge list connecting in-sample nodes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "year_thresh = 2006 # the threshold year for in-sample and out-of-sample set split, which can be 2007, 2008 and 2009\n",
    "subgraph_edgelist = []\n",
    "for ii in range(len(edgelist)):\n",
    "    source_index = edgelist[\"source\"][ii]\n",
    "    target_index = edgelist[\"target\"][ii]\n",
    "    source_year = int(node_data[\"year\"][source_index])\n",
    "    target_year = int(node_data[\"year\"][target_index])\n",
    "    if source_year < year_thresh and target_year < year_thresh:\n",
    "        subgraph_edgelist.append([source_index, target_index])\n",
    "subgraph_edgelist = pd.DataFrame(np.array(subgraph_edgelist), columns=[\"source\", \"target\"])\n",
    "subgraph_edgelist[\"label\"] = \"cites\" # set the edge type"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Construct the network from the selected edge list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "G_sub_nx = nx.from_pandas_edgelist(subgraph_edgelist, edge_attr=\"label\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Specify node types."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "nx.set_node_attributes(G_sub_nx, \"paper\", \"label\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the ids of the nodes in the selected subgraph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "subgraph_node_ids = sorted(list(G_sub_nx.nodes))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the node features of the selected subgraph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "subgraph_node_features = node_data[feature_names].reindex(subgraph_node_ids)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create the Stellargraph with node features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "G_sub = sg.StellarGraph(G_sub_nx, node_features=subgraph_node_features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "StellarGraph: Undirected multigraph\n",
      " Nodes: 11776, Edges: 28937\n",
      "\n",
      " Node types:\n",
      "  paper: [11776]\n",
      "    Edge types: paper-cites->paper\n",
      "\n",
      " Edge types:\n",
      "    paper-cites->paper: [28937]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(G_sub.info())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train attri2vec on the DBLP Subgraph"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Specify the other optional parameter values: root nodes, the number of walks to take per node, the length of each walk."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "nodes = list(G_sub.nodes())\n",
    "number_of_walks = 2\n",
    "length = 5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create the UnsupervisedSampler instance with the relevant parameters passed to it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "unsupervised_samples = UnsupervisedSampler(G_sub, nodes=nodes, length=length, number_of_walks=number_of_walks)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set the batch size and the number of epochs. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 50\n",
    "epochs = 6"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define an attri2vec training generator, which generates a batch of (feature of target node, index of context node, label of node pair) pairs per iteration."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running Attri2VecLinkGenerator with an estimated 4710 batches generated on the fly per epoch.\n"
     ]
    }
   ],
   "source": [
    "train_gen = Attri2VecLinkGenerator(G_sub, batch_size).flow(unsupervised_samples)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Building the model: a 1-hidden-layer node representation ('input embedding') of the `target` node and the parameter vector ('output embedding') for predicting the existence of `context node` for each `(target context)` pair, with a link classification layer performed on the dot product of the 'input embedding' of the `target` node and the 'output embedding' of the `context` node.\n",
    "\n",
    "Attri2Vec part of the model, with a 128-dimenssion hidden layer, no bias term, no dropout and no normalization. (Dropout can be switched on by specifying a positive dropout rate, 0 < dropout < 1 and normalization can be set to 'l2'). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "layer_sizes = [128]\n",
    "attri2vec = Attri2Vec(\n",
    "    layer_sizes=layer_sizes, generator=train_gen, bias=False, normalize=None\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: Logging before flag parsing goes to stderr.\n",
      "W0925 10:21:27.091693 140736388883392 deprecation.py:506] From /anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/init_ops.py:1251: calling VarianceScaling.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Call initializer instance with the dtype argument instead of passing it to the constructor\n",
      "W0925 10:21:27.113364 140736388883392 deprecation.py:506] From /anaconda3/lib/python3.6/site-packages/tensorflow/python/keras/initializers.py:119: calling RandomUniform.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Call initializer instance with the dtype argument instead of passing it to the constructor\n"
     ]
    }
   ],
   "source": [
    "# Build the model and expose input and output sockets of attri2vec, for node pair inputs:\n",
    "x_inp, x_out = attri2vec.build()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the link_classification function to generate the prediction, with the 'ip' edge embedding generation method and the 'sigmoid' activation, which actually performs the dot product of the 'input embedding' of the target node and the 'output embedding' of the context node followed by a sigmoid activation. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "link_classification: using 'ip' method to combine node embeddings into edge embeddings\n"
     ]
    }
   ],
   "source": [
    "prediction = link_classification(\n",
    "    output_dim=1, output_act=\"sigmoid\", edge_embedding_method='ip'\n",
    ")(x_out)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Stack the Attri2Vec encoder and prediction layer into a Keras model, and specify the loss."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.Model(inputs=x_inp, outputs=prediction)\n",
    "\n",
    "model.compile(\n",
    "    optimizer=keras.optimizers.Adam(lr=1e-3),\n",
    "    loss=keras.losses.binary_crossentropy,\n",
    "    metrics=[keras.metrics.binary_accuracy],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "W0925 10:21:27.574651 140736388883392 deprecation.py:323] From /anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/math_grad.py:1250: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.where in 2.0, which has the same broadcast rule as np.where\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/6\n",
      "4711/4711 [==============================] - 72s 15ms/step - loss: 0.7680 - binary_accuracy: 0.5814\n",
      "Epoch 2/6\n",
      "4711/4711 [==============================] - 72s 15ms/step - loss: 0.6447 - binary_accuracy: 0.6101\n",
      "Epoch 3/6\n",
      "4711/4711 [==============================] - 75s 16ms/step - loss: 0.5866 - binary_accuracy: 0.6483\n",
      "Epoch 4/6\n",
      "4711/4711 [==============================] - 75s 16ms/step - loss: 0.4925 - binary_accuracy: 0.7284\n",
      "Epoch 5/6\n",
      "4711/4711 [==============================] - 76s 16ms/step - loss: 0.3574 - binary_accuracy: 0.8300\n",
      "Epoch 6/6\n",
      "4711/4711 [==============================] - 76s 16ms/step - loss: 0.2242 - binary_accuracy: 0.9105\n"
     ]
    }
   ],
   "source": [
    "history = model.fit_generator(\n",
    "    train_gen,\n",
    "    epochs=epochs,\n",
    "    verbose=1,\n",
    "    use_multiprocessing=True,\n",
    "    workers=4,\n",
    "    shuffle=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Predicting links of out-of-sample nodes with the learned attri2vec model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Build the node based model for predicting node representations from node content attributes with the learned parameters. Below a Keras model is constructed, with x_inp[0] as input and x_out[0] as output. Note that this model's weights are the same as those of the corresponding node encoder in the previously trained node pair classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_inp_src = x_inp[0]\n",
    "x_out_src = x_out[0]\n",
    "embedding_model = keras.Model(inputs=x_inp_src, outputs=x_out_src)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the node embeddings, for both in-sample and out-of-sample nodes, by applying the learned mapping function to node content features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "369/369 [==============================] - 1s 1ms/step\n"
     ]
    }
   ],
   "source": [
    "node_ids = node_data.index\n",
    "node_gen = Attri2VecNodeGenerator(G_all, batch_size).flow(node_ids)\n",
    "node_embeddings = embedding_model.predict_generator(node_gen, workers=4, verbose=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the positive and negative edges for in-sample nodes and out-of-sample nodes. The edges of the in-sample nodes only include the edges between in-sample nodes, and the edges of out-of-sample nodes are referred to all the edges linked to out-of-sample nodes, including the edges connecting in-sample and out-of-sample edges."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "year_thresh = 2006\n",
    "in_sample_edges = []\n",
    "out_of_sample_edges = []\n",
    "for ii in range(len(edgelist)):\n",
    "    source_index = edgelist[\"source\"][ii]\n",
    "    target_index = edgelist[\"target\"][ii]\n",
    "    if source_index > target_index: # neglect edge direction for the undirected graph\n",
    "        continue\n",
    "    source_year = int(node_data[\"year\"][source_index])\n",
    "    target_year = int(node_data[\"year\"][target_index])\n",
    "    if source_year < year_thresh and target_year < year_thresh:\n",
    "        in_sample_edges.append([source_index, target_index, 1]) # get the positive edge\n",
    "        negative_target_index = unsupervised_samples.random.choices(node_data.index.tolist(), k=1) # generate negative node\n",
    "        in_sample_edges.append([source_index, negative_target_index[0], 0]) # get the negative edge\n",
    "    else:\n",
    "        out_of_sample_edges.append([source_index, target_index, 1]) # get the positive edge\n",
    "        negative_target_index = unsupervised_samples.random.choices(node_data.index.tolist(), k=1) # generate negative node\n",
    "        out_of_sample_edges.append([source_index, negative_target_index[0], 0]) # get the negative edge\n",
    "in_sample_edges = np.array(in_sample_edges)\n",
    "out_of_sample_edges = np.array(out_of_sample_edges)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Construct the edge features from the learned node representations with l2 normed difference, where edge features are the element-wise square of the difference between the embeddings of two head nodes. Other strategy like element-wise product can also be used to construct edge features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "in_sample_edge_feat_from_emb = (node_embeddings[in_sample_edges[:,0]]-node_embeddings[in_sample_edges[:,1]])**2\n",
    "out_of_sample_edge_feat_from_emb = (node_embeddings[out_of_sample_edges[:,0]]-node_embeddings[out_of_sample_edges[:,1]])**2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train the Logistic Regression classifier from in-sample edges with the edge features constructed from attri2vec embeddings. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "                   intercept_scaling=1, l1_ratio=None, max_iter=500,\n",
       "                   multi_class='auto', n_jobs=None, penalty='l2',\n",
       "                   random_state=None, solver='lbfgs', tol=0.0001, verbose=0,\n",
       "                   warm_start=False)"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf_edge_pred_from_emb = LogisticRegression(verbose=0, solver='lbfgs', multi_class=\"auto\", max_iter=500)\n",
    "clf_edge_pred_from_emb.fit(in_sample_edge_feat_from_emb, in_sample_edges[:,2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Predict the edge existence probability with the trained Logistic Regression classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "edge_pred_from_emb = clf_edge_pred_from_emb.predict_proba(out_of_sample_edge_feat_from_emb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the positive class index of `edge_pred_from_emb`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "if clf_edge_pred_from_emb.classes_[0] == 1:\n",
    "    positive_class_index = 0\n",
    "else:\n",
    "    positive_class_index = 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Evaluate the AUC score for the prediction with attri2vec embeddings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7842243817182476"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "roc_auc_score(out_of_sample_edges[:,2], edge_pred_from_emb[:,positive_class_index])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As the baseline, we also investigate the performance of node content features in predicting the edges of out-of-sample nodes. Firstly, we construct edge features from node content features with the same strategy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "in_sample_edge_rep_from_feat = (node_data[feature_names].values[in_sample_edges[:,0]]-node_data[feature_names].values[in_sample_edges[:,1]])**2\n",
    "out_of_sample_edge_rep_from_feat = (node_data[feature_names].values[out_of_sample_edges[:,0]]-node_data[feature_names].values[out_of_sample_edges[:,1]])**2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we train the Logistic Regression classifier from in-sample edges with the edge features constructed from node content features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "                   intercept_scaling=1, l1_ratio=None, max_iter=500,\n",
       "                   multi_class='auto', n_jobs=None, penalty='l2',\n",
       "                   random_state=None, solver='lbfgs', tol=0.0001, verbose=0,\n",
       "                   warm_start=False)"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf_edge_pred_from_feat = LogisticRegression(verbose=0, solver='lbfgs', multi_class=\"auto\", max_iter=500)\n",
    "clf_edge_pred_from_feat.fit(in_sample_edge_rep_from_feat, in_sample_edges[:,2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Predict the edge existence probability with the trained Logistic Regression classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "edge_pred_from_feat = clf_edge_pred_from_feat.predict_proba(out_of_sample_edge_rep_from_feat)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get positive class index of `clf_edge_pred_from_feat`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "if clf_edge_pred_from_feat.classes_[0] == 1:\n",
    "    positive_class_index = 0\n",
    "else:\n",
    "    positive_class_index = 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Evaluate the AUC score for the prediction with node content features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6666971120688185"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "roc_auc_score(out_of_sample_edges[:,2], edge_pred_from_feat[:,positive_class_index])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "attri2vec can inductively infer the representations of out-of-sample nodes from their content attributes. As the inferred node representations well capture both structure and node content information, they perform much better than node content features in predicting the links of out-of-sample nodes."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
<commit_msg>rerun attri2vec out-of-sample link prediction demo on DBLP<commit_after>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Inductive Node Representation Learning through attri2vec"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the python implementation of the attri2vec algorithm outlined in paper ***[Attributed Network Embedding Via Subspace Discovery](https://arxiv.org/abs/1901.04095)*** <NAME>, <NAME>, <NAME> and <NAME>, arXiv:1901.04095, [cs.SI], 2019. The implementation uses the stellargraph libraries.\n",
    "\n",
    "## attri2vec\n",
    "\n",
    "attri2vec learns node representations by performing a linear/non-linear mapping on node content attributes. To make the learned node representations respect structural similarity, [`DeepWalk`](https://dl.acm.org/citation.cfm?id=2623732)/[`node2vec`](https://snap.stanford.edu/node2vec) learning mechanism is used to make nodes sharing similar random walk context nodes represented closely in the subspace, which is achieved by maximizing the occurrence probability of context nodes conditioned on the representation of the target nodes. The probability is modelled by Softmax and negative sampling is used to speed up its calculation. This makes attri2vec equivalent to predict whether a node occurs in the given target node's context in random walks with the representation of the target node, by minimizing the cross-entropy loss. \n",
    "\n",
    "In implementation, node embeddings are learnt by solving a simple classification task: given a large set of \"positive\" `(target, context)` node pairs generated from random walks performed on the graph (i.e., node pairs that co-occur within a certain context window in random walks), and an equally large set of \"negative\" node pairs that are randomly selected from the graph according to a certain distribution, learn a binary classifier that predicts whether arbitrary node pairs are likely to co-occur in a random walk performed on the graph. Through learning this simple binary node-pair-classification task, the model automatically learns an inductive mapping from attributes of nodes to node embeddings in a low-dimensional vector space, which preserves structural and feature similarities of the nodes.\n",
    "\n",
    "To train the attri2vec model, we first construct a training set of nodes, which is composed of an equal number of positive and negative `(target, context)` pairs from the graph. The positive `(target, context)` pairs are the node pairs co-occurring on random walks over the graph whereas the negative node pairs are the sampled randomly from the global node degree distribution of the graph. In attri2vec, each node is attached with two kinds of embeddings: 1) the inductive 'input embedding', i.e, the objective embedding, obtained by perform a non-linear transformation on node content features, and 2) 'output embedding', i.e., the parameter vector used to predict its occurrence as a context node, obtained by looking up a parameter table. Given a `(target, context)` pair, attri2vec outputs a predictive value to indicate whether it is positive or negative, which is obtained by performing the dot product of the 'input embedding' of the target node and the 'output embedding' of the context node, followed by a sigmoid activation. \n",
    "\n",
    "The entire model is trained end-to-end by minimizing the binary cross-entropy loss function with regards to predicted node pair labels and true node pair labels, using stochastic gradient descent (SGD) updates of the model parameters, with minibatches of 'training' node pairs generated on demand and fed into the model.\n",
    "\n",
    "In this demo, we first train the attri2vec model on the in-sample subgraph and obtain a mapping function from node attributes to node representations, then apply the mapping function to the content attributes of out-of-sample nodes and obtain the representations of out-of-sample nodes. We evaluate the quality of inferred out-of-sample node representations by using it to predict the links of out-of-sample nodes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import networkx as nx\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import os\n",
    "import random\n",
    "\n",
    "import stellargraph as sg\n",
    "from stellargraph.data import UnsupervisedSampler\n",
    "from stellargraph.mapper import Attri2VecLinkGenerator, Attri2VecNodeGenerator\n",
    "from stellargraph.layer import Attri2Vec, link_classification\n",
    "\n",
    "from tensorflow import keras\n",
    "\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.metrics import roc_auc_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading DBLP network data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This demo uses a DBLP citation network, a subgraph extracted from [DBLP-Citation-network V3](https://aminer.org/citation). To form this subgraph, papers from four subjects are extracted according to their venue information: *Database, Data Mining, Artificial Intelligence and Computer Vision*, and papers with no citations are removed. The DBLP network contains 18,448 papers and 45,661 citation relations. From paper titles, we construct 2,476-dimensional binary node feature vectors, with each element indicating the presence/absence of the corresponding word. By ignoring the citation direction, we take the DBLP subgraph as an undirected network.\n",
    "\n",
    "As papers in DBLP are attached with publication year, the DBLP network with the dynamic property can be used to study the problem of out-of-sample node representation learning. From the DBLP network, we construct four in-sample subgraphs using papers published before 2006, 2007, 2008 and 2009, and denote the four subgraphs as DBLP2006, DBLP2007, DBLP2008, and DBLP2009. For each subgraph, the remaining papers are taken as out-of-sample nodes. We consider the case where new coming nodes have no links. We predict the links of out-of-sample nodes using the learned out-of-sample node representations and compare its performance with the node content feature baseline.\n",
    "\n",
    "The dataset used in this demo can be downloaded from https://www.kaggle.com/daozhang/dblp-subgraph.\n",
    "The following is the description of the dataset:\n",
    "\n",
    "> The content.txt file contains descriptions of the papers in the following format:\n",
    "\n",
    " \t\t<paper_id> <word_attributes> <class_label> <publication_year>\n",
    "        \n",
    "> The first entry in each line contains the unique integer ID (ranging from 0 to 18,447) of the paper followed by binary values indicating whether each word in the vocabulary is present (indicated by 1) or absent (indicated by 0) in the paper. Finally, the last two entries in the line are the class label and the publication year of the paper.\n",
    "> The edgeList.txt file contains the citation relations. Each line describes a link in the following format:\n",
    "\t\t\n",
    "        <ID of paper1> <ID of paper2>\n",
    "        \n",
    "> Each line contains two paper IDs, with paper2 citing paper1 or paper1 citing paper2.\n",
    "\n",
    "\n",
    "Download and unzip the dblp-subgraph.zip file to a location on your computer and set the `data_dir` variable to\n",
    "point to the location of the dataset (the \"DBLP\" directory containing \"content.txt\" and \"edgeList.txt\")."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_dir = \"~/data/DBLP\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the graph from the edgelist."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "edgelist = pd.read_csv(os.path.join(data_dir, \"edgeList.txt\"), sep='\\t', header=None, names=[\"source\", \"target\"])\n",
    "edgelist[\"label\"] = \"cites\"  # set the edge type"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load paper content features, subjects and publishing years."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "feature_names = [\"w_{}\".format(ii) for ii in range(2476)]\n",
    "node_column_names =  feature_names + [\"subject\", \"year\"]\n",
    "node_data = pd.read_csv(os.path.join(data_dir, \"content.txt\"), sep='\\t', header=None, names=node_column_names)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Construct the whole graph from edge list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "G_all_nx = nx.from_pandas_edgelist(edgelist, edge_attr=\"label\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Specify node types."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "nx.set_node_attributes(G_all_nx, \"paper\", \"label\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get node features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_node_features = node_data[feature_names]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create the Stellargraph with node features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "G_all = sg.StellarGraph(G_all_nx, node_features=all_node_features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "StellarGraph: Undirected multigraph\n",
      " Nodes: 18448, Edges: 45611\n",
      "\n",
      " Node types:\n",
      "  paper: [18448]\n",
      "    Edge types: paper-cites->paper\n",
      "\n",
      " Edge types:\n",
      "    paper-cites->paper: [45611]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(G_all.info())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get DBLP Subgraph \n",
    "### with papers published before a threshold year"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the edge list connecting in-sample nodes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "year_thresh = 2006 # the threshold year for in-sample and out-of-sample set split, which can be 2007, 2008 and 2009\n",
    "subgraph_edgelist = []\n",
    "for ii in range(len(edgelist)):\n",
    "    source_index = edgelist[\"source\"][ii]\n",
    "    target_index = edgelist[\"target\"][ii]\n",
    "    source_year = int(node_data[\"year\"][source_index])\n",
    "    target_year = int(node_data[\"year\"][target_index])\n",
    "    if source_year < year_thresh and target_year < year_thresh:\n",
    "        subgraph_edgelist.append([source_index, target_index])\n",
    "subgraph_edgelist = pd.DataFrame(np.array(subgraph_edgelist), columns=[\"source\", \"target\"])\n",
    "subgraph_edgelist[\"label\"] = \"cites\" # set the edge type"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Construct the network from the selected edge list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "G_sub_nx = nx.from_pandas_edgelist(subgraph_edgelist, edge_attr=\"label\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Specify node types."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "nx.set_node_attributes(G_sub_nx, \"paper\", \"label\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the ids of the nodes in the selected subgraph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "subgraph_node_ids = sorted(list(G_sub_nx.nodes))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the node features of the selected subgraph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "subgraph_node_features = node_data[feature_names].reindex(subgraph_node_ids)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create the Stellargraph with node features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "G_sub = sg.StellarGraph(G_sub_nx, node_features=subgraph_node_features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "StellarGraph: Undirected multigraph\n",
      " Nodes: 11776, Edges: 28937\n",
      "\n",
      " Node types:\n",
      "  paper: [11776]\n",
      "    Edge types: paper-cites->paper\n",
      "\n",
      " Edge types:\n",
      "    paper-cites->paper: [28937]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(G_sub.info())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train attri2vec on the DBLP Subgraph"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Specify the other optional parameter values: root nodes, the number of walks to take per node, the length of each walk."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "nodes = list(G_sub.nodes())\n",
    "number_of_walks = 2\n",
    "length = 5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create the UnsupervisedSampler instance with the relevant parameters passed to it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "unsupervised_samples = UnsupervisedSampler(G_sub, nodes=nodes, length=length, number_of_walks=number_of_walks)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set the batch size and the number of epochs. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 50\n",
    "epochs = 6"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define an attri2vec training generator, which generates a batch of (feature of target node, index of context node, label of node pair) pairs per iteration."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running Attri2VecLinkGenerator with an estimated 4710 batches generated on the fly per epoch.\n"
     ]
    }
   ],
   "source": [
    "train_gen = Attri2VecLinkGenerator(G_sub, batch_size).flow(unsupervised_samples)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Building the model: a 1-hidden-layer node representation ('input embedding') of the `target` node and the parameter vector ('output embedding') for predicting the existence of `context node` for each `(target context)` pair, with a link classification layer performed on the dot product of the 'input embedding' of the `target` node and the 'output embedding' of the `context` node.\n",
    "\n",
    "Attri2Vec part of the model, with a 128-dimenssion hidden layer, no bias term, no dropout and no normalization. (Dropout can be switched on by specifying a positive dropout rate, 0 < dropout < 1 and normalization can be set to 'l2'). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "layer_sizes = [128]\n",
    "attri2vec = Attri2Vec(\n",
    "    layer_sizes=layer_sizes, generator=train_gen, bias=False, normalize=None\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: Logging before flag parsing goes to stderr.\n",
      "W0925 17:46:06.513718 140736388883392 deprecation.py:506] From /anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/init_ops.py:1251: calling VarianceScaling.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Call initializer instance with the dtype argument instead of passing it to the constructor\n",
      "W0925 17:46:06.534775 140736388883392 deprecation.py:506] From /anaconda3/lib/python3.6/site-packages/tensorflow/python/keras/initializers.py:119: calling RandomUniform.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Call initializer instance with the dtype argument instead of passing it to the constructor\n"
     ]
    }
   ],
   "source": [
    "# Build the model and expose input and output sockets of attri2vec, for node pair inputs:\n",
    "x_inp, x_out = attri2vec.build()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the link_classification function to generate the prediction, with the 'ip' edge embedding generation method and the 'sigmoid' activation, which actually performs the dot product of the 'input embedding' of the target node and the 'output embedding' of the context node followed by a sigmoid activation. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "link_classification: using 'ip' method to combine node embeddings into edge embeddings\n"
     ]
    }
   ],
   "source": [
    "prediction = link_classification(\n",
    "    output_dim=1, output_act=\"sigmoid\", edge_embedding_method='ip'\n",
    ")(x_out)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Stack the Attri2Vec encoder and prediction layer into a Keras model, and specify the loss."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.Model(inputs=x_inp, outputs=prediction)\n",
    "\n",
    "model.compile(\n",
    "    optimizer=keras.optimizers.Adam(lr=1e-3),\n",
    "    loss=keras.losses.binary_crossentropy,\n",
    "    metrics=[keras.metrics.binary_accuracy],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "W0925 17:46:07.069445 140736388883392 deprecation.py:323] From /anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/math_grad.py:1250: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.where in 2.0, which has the same broadcast rule as np.where\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/6\n",
      "4711/4711 [==============================] - 75s 16ms/step - loss: 0.7018 - binary_accuracy: 0.6224\n",
      "Epoch 2/6\n",
      "4711/4711 [==============================] - 76s 16ms/step - loss: 0.6562 - binary_accuracy: 0.6042\n",
      "Epoch 3/6\n",
      "4711/4711 [==============================] - 75s 16ms/step - loss: 0.5475 - binary_accuracy: 0.6931\n",
      "Epoch 4/6\n",
      "4711/4711 [==============================] - 76s 16ms/step - loss: 0.4017 - binary_accuracy: 0.8048\n",
      "Epoch 5/6\n",
      "4711/4711 [==============================] - 76s 16ms/step - loss: 0.2637 - binary_accuracy: 0.8900\n",
      "Epoch 6/6\n",
      "4711/4711 [==============================] - 76s 16ms/step - loss: 0.1570 - binary_accuracy: 0.9473\n"
     ]
    }
   ],
   "source": [
    "history = model.fit_generator(\n",
    "    train_gen,\n",
    "    epochs=epochs,\n",
    "    verbose=1,\n",
    "    use_multiprocessing=True,\n",
    "    workers=4,\n",
    "    shuffle=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Predicting links of out-of-sample nodes with the learned attri2vec model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Build the node based model for predicting node representations from node content attributes with the learned parameters. Below a Keras model is constructed, with x_inp[0] as input and x_out[0] as output. Note that this model's weights are the same as those of the corresponding node encoder in the previously trained node pair classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_inp_src = x_inp[0]\n",
    "x_out_src = x_out[0]\n",
    "embedding_model = keras.Model(inputs=x_inp_src, outputs=x_out_src)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the node embeddings, for both in-sample and out-of-sample nodes, by applying the learned mapping function to node content features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "369/369 [==============================] - 1s 2ms/step\n"
     ]
    }
   ],
   "source": [
    "node_ids = node_data.index\n",
    "node_gen = Attri2VecNodeGenerator(G_all, batch_size).flow(node_ids)\n",
    "node_embeddings = embedding_model.predict_generator(node_gen, workers=4, verbose=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the positive and negative edges for in-sample nodes and out-of-sample nodes. The edges of the in-sample nodes only include the edges between in-sample nodes, and the edges of out-of-sample nodes are referred to all the edges linked to out-of-sample nodes, including the edges connecting in-sample and out-of-sample edges."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "year_thresh = 2006\n",
    "in_sample_edges = []\n",
    "out_of_sample_edges = []\n",
    "for ii in range(len(edgelist)):\n",
    "    source_index = edgelist[\"source\"][ii]\n",
    "    target_index = edgelist[\"target\"][ii]\n",
    "    if source_index > target_index: # neglect edge direction for the undirected graph\n",
    "        continue\n",
    "    source_year = int(node_data[\"year\"][source_index])\n",
    "    target_year = int(node_data[\"year\"][target_index])\n",
    "    if source_year < year_thresh and target_year < year_thresh:\n",
    "        in_sample_edges.append([source_index, target_index, 1]) # get the positive edge\n",
    "        negative_target_index = unsupervised_samples.random.choices(node_data.index.tolist(), k=1) # generate negative node\n",
    "        in_sample_edges.append([source_index, negative_target_index[0], 0]) # get the negative edge\n",
    "    else:\n",
    "        out_of_sample_edges.append([source_index, target_index, 1]) # get the positive edge\n",
    "        negative_target_index = unsupervised_samples.random.choices(node_data.index.tolist(), k=1) # generate negative node\n",
    "        out_of_sample_edges.append([source_index, negative_target_index[0], 0]) # get the negative edge\n",
    "in_sample_edges = np.array(in_sample_edges)\n",
    "out_of_sample_edges = np.array(out_of_sample_edges)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Construct the edge features from the learned node representations with l2 normed difference, where edge features are the element-wise square of the difference between the embeddings of two head nodes. Other strategy like element-wise product can also be used to construct edge features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "in_sample_edge_feat_from_emb = (node_embeddings[in_sample_edges[:,0]]-node_embeddings[in_sample_edges[:,1]])**2\n",
    "out_of_sample_edge_feat_from_emb = (node_embeddings[out_of_sample_edges[:,0]]-node_embeddings[out_of_sample_edges[:,1]])**2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train the Logistic Regression classifier from in-sample edges with the edge features constructed from attri2vec embeddings. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "                   intercept_scaling=1, l1_ratio=None, max_iter=500,\n",
       "                   multi_class='auto', n_jobs=None, penalty='l2',\n",
       "                   random_state=None, solver='lbfgs', tol=0.0001, verbose=0,\n",
       "                   warm_start=False)"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf_edge_pred_from_emb = LogisticRegression(verbose=0, solver='lbfgs', multi_class=\"auto\", max_iter=500)\n",
    "clf_edge_pred_from_emb.fit(in_sample_edge_feat_from_emb, in_sample_edges[:,2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Predict the edge existence probability with the trained Logistic Regression classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "edge_pred_from_emb = clf_edge_pred_from_emb.predict_proba(out_of_sample_edge_feat_from_emb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the positive class index of `edge_pred_from_emb`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "if clf_edge_pred_from_emb.classes_[0] == 1:\n",
    "    positive_class_index = 0\n",
    "else:\n",
    "    positive_class_index = 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Evaluate the AUC score for the prediction with attri2vec embeddings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8085538458796013"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "roc_auc_score(out_of_sample_edges[:,2], edge_pred_from_emb[:,positive_class_index])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As the baseline, we also investigate the performance of node content features in predicting the edges of out-of-sample nodes. Firstly, we construct edge features from node content features with the same strategy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "in_sample_edge_rep_from_feat = (node_data[feature_names].values[in_sample_edges[:,0]]-node_data[feature_names].values[in_sample_edges[:,1]])**2\n",
    "out_of_sample_edge_rep_from_feat = (node_data[feature_names].values[out_of_sample_edges[:,0]]-node_data[feature_names].values[out_of_sample_edges[:,1]])**2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we train the Logistic Regression classifier from in-sample edges with the edge features constructed from node content features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "                   intercept_scaling=1, l1_ratio=None, max_iter=500,\n",
       "                   multi_class='auto', n_jobs=None, penalty='l2',\n",
       "                   random_state=None, solver='lbfgs', tol=0.0001, verbose=0,\n",
       "                   warm_start=False)"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf_edge_pred_from_feat = LogisticRegression(verbose=0, solver='lbfgs', multi_class=\"auto\", max_iter=500)\n",
    "clf_edge_pred_from_feat.fit(in_sample_edge_rep_from_feat, in_sample_edges[:,2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Predict the edge existence probability with the trained Logistic Regression classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "edge_pred_from_feat = clf_edge_pred_from_feat.predict_proba(out_of_sample_edge_rep_from_feat)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get positive class index of `clf_edge_pred_from_feat`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "if clf_edge_pred_from_feat.classes_[0] == 1:\n",
    "    positive_class_index = 0\n",
    "else:\n",
    "    positive_class_index = 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Evaluate the AUC score for the prediction with node content features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6651603215420048"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "roc_auc_score(out_of_sample_edges[:,2], edge_pred_from_feat[:,positive_class_index])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "attri2vec can inductively infer the representations of out-of-sample nodes from their content attributes. As the inferred node representations well capture both structure and node content information, they perform much better than node content features in predicting the links of out-of-sample nodes."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}

{'fraction_non_alphanumeric': 0.15511273420133376, 'fraction_numerical': 0.02154652270562083, 'mean_word_length': 3.1593580768722758, 'pattern_counts': {'":': 755, '<': 37, '<?xml version=': 0, '>': 43, 'https://': 10, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6271969', 'n_tokens_mistral': 21902, 'n_tokens_neox': 20588, 'n_words': 5511}