<jupyter_start><jupyter_text>Causal Inference In Statistics - A Primer        1.4 Graphs        <NAME>        www.data4sci.com            @bgoncalves, @data4sci<jupyter_code>from collections import Counter
from pprint import pprint

import pandas as pd
import numpy as np

import matplotlib
import matplotlib.pyplot as plt 

import networkx as nx
from networkx.drawing.nx_pydot import graphviz_layout

import watermark

%load_ext watermark
%matplotlib inline<jupyter_output><empty_output><jupyter_text>We start by print out the versions of the libraries we're using for future reference<jupyter_code>%watermark -n -v -m -g -iv<jupyter_output>pandas     1.0.1
watermark  2.0.2
json       2.0.9
networkx   2.4
matplotlib 3.1.3
autopep8   1.5
numpy      1.18.1
Fri Oct 16 2020 

CPython 3.7.3
IPython 6.2.1

compiler   : Clang 4.0.1 (tags/RELEASE_401/final)
system     : Darwin
release    : 19.6.0
machine    : x86_64
processor  : i386
CPU cores  : 8
interpreter: 64bit
Git hash   : 4eecd30344d5677356267e69b3559341d7f43a0d<jupyter_text>Load default figure style<jupyter_code>plt.style.use('./d4sci.mplstyle')
colors = plt.rcParams['axes.prop_cycle'].by_key()['color']<jupyter_output><empty_output><jupyter_text>Graphs Graphs allow us to represent relationships (edges) between entities (nodes). We will use the [NetworkX](https://networkx.github.io/) Python package to handle all of our graphs for the rest of the course. We start by instanciating a new Graph object<jupyter_code>G = nx.Graph()<jupyter_output><empty_output><jupyter_text>We can add nodes one by one or from a list of nodes. Any Python hashable object can be a node:<jupyter_code>G.add_nodes_from(['X', 'Y', 'Z'])<jupyter_output><empty_output><jupyter_text>Similarly, we can add edges one by one of from a list. Edges are a pair of nodes with an optional dictionary contianing data about the edge:<jupyter_code>G.add_edges_from([('X', 'Y', {'label':'A'}), ('Y', 'Z', {'label':'B'})])<jupyter_output><empty_output><jupyter_text>NetworkX has an extensive set of drawing functionality. We can simply draw the graph directly:<jupyter_code>nx.draw(G)<jupyter_output>/Users/bgoncalves/anaconda3/lib/python3.7/site-packages/IPython/core/events.py:73: UserWarning: This figure includes Axes that are not compatible with tight_layout, so results might be incorrect.
  func(*args, **kwargs)
/Users/bgoncalves/anaconda3/lib/python3.7/site-packages/IPython/core/pylabtools.py:122: UserWarning: This figure includes Axes that are not compatible with tight_layout, so results might be incorrect.
  fig.canvas.print_figure(bytes_io, **kw)<jupyter_text>Or customize the plot by definining our own layout (node coordinates):<jupyter_code>pos = {'X':(-1, 0), 'Y':(0, 0), 'Z':(1, 0)}
labels = {(node_i, node_j) : label for node_i, node_j, label in G.edges(data='label', default='')}<jupyter_output><empty_output><jupyter_text>And the way we want the graph to be plotted using Matplotlib.<jupyter_code>fig, ax = plt.subplots(1, figsize=(4,1))
nx.draw(G, pos, ax=ax, node_color=colors[0])
nx.draw_networkx_labels(G, pos, ax=ax)
nx.draw_networkx_edge_labels(G, pos, labels, ax=ax)
fig.tight_layout()<jupyter_output><empty_output><jupyter_text>To create a directed graph, we simply use DiGraph()<jupyter_code>G = nx.DiGraph()
G.add_nodes_from(['X', 'Y', 'Z'])
G.add_edges_from([('X', 'Y', {'label':'A'}), ('Y', 'Z', {'label':'B'})])<jupyter_output><empty_output><jupyter_text>And we can plot it in just the same way<jupyter_code>pos = {'X':(-1, 0), 'Y':(0, 0), 'Z':(1, 0)}
labels = {(node_i, node_j) : label for node_i, node_j, label in G.edges(data='label', default='')}

fig, ax = plt.subplots(1, figsize=(4,1))
nx.draw(G, pos, ax=ax, node_color=colors[0])
nx.draw_networkx_labels(G, pos, ax=ax)
nx.draw_networkx_edge_labels(G, pos, labels, ax=ax)
fig.tight_layout()<jupyter_output><empty_output><jupyter_text>And if we add mode edges, the layout naturally needs to change as well<jupyter_code>G = nx.DiGraph()
G.add_edges_from([
    ('U', 'X', {'label':'A'}),
    ('U', 'Y', {'label':'C'}),
    ('X', 'Y', {'label':'B'})])
pos = {'U': (0, 1), 'X': (-1, 0), 'Y': (1, 0)}
labels = {(node_i, node_j) : label for node_i, node_j, label in G.edges(data='label', default='')}

fig, ax = plt.subplots(1, figsize=(3,2.2))
nx.draw(G, pos, ax=ax, node_color=colors[0])
nx.draw_networkx_labels(G, pos, ax=ax)
#nx.draw_networkx_edge_labels(G, pos, labels, ax=ax)
fig.tight_layout()<jupyter_output><empty_output><jupyter_text>And yet another example<jupyter_code>G = nx.DiGraph()
G.add_edges_from([
    ('X', 'Y'),
    ('X', 'W'),
    ('W', 'Y'),
    ('W', 'Z'),
    ('Y', 'T'),
    ('Y', 'Z'),
    ('Z', 'T')])
pos = {'X': (-1, 0), 'Y': (0, 0), 'Z': (1, 0), 'W': (0, 1), 'T' : (0.5, -0.5)}

fig, ax = plt.subplots(1, figsize=(4,2.3))
nx.draw(G, pos, ax=ax, node_color=colors[0])
nx.draw_networkx_labels(G, pos, ax=ax)
fig.tight_layout()<jupyter_output><empty_output><jupyter_text>NetworkX also gives us simple ways to directly ask questions of this DAG We can get the parents (predecessors) and ancestors of Z<jupyter_code>print('Parents:', list(G.predecessors('Z')), 'Ancestors:', list(nx.ancestors(G, 'Z')))<jupyter_output>Parents: ['W', 'Y'] Ancestors: ['W', 'Y', 'X']<jupyter_text>And the children (successors)and decendents of W:<jupyter_code>print('Childrens:', list(G.successors('W')), 'Descendants:', list(nx.descendants(G, 'W')))<jupyter_output>Childrens: ['Y', 'Z'] Descendants: ['Y', 'T', 'Z']<jupyter_text>We can easily find all the directed paths between X and T<jupyter_code>directed_paths = {tuple(path) for path in nx.all_simple_paths(G, 'X', 'T')}
print(directed_paths)<jupyter_output>{('X', 'Y', 'T'), ('X', 'W', 'Y', 'Z', 'T'), ('X', 'Y', 'Z', 'T'), ('X', 'W', 'Z', 'T'), ('X', 'W', 'Y', 'T')}<jupyter_text>And write a helper function to plot them all out on top of our DAG:<jupyter_code>def plot_path(G, pos, path, ax=None):
    fig = None
    
    if ax == None:
        fig, ax = plt.subplots(1, figsize=(4,2.3))
    
    edgelist = {(path[i], path[i+1]) for i in range(len(path)-1)}
    edges = set(G.edges()) - set(edgelist)
    
    nx.draw(G, pos, node_color=colors[0], ax=ax, edgelist=[])
    nx.draw_networkx_labels(G, pos, ax=ax)
    nx.draw_networkx_edges(G, pos,
                       edgelist=edgelist,
                       width=3, edge_color=colors[1], ax=ax)
    nx.draw_networkx_edges(G, pos,
                       edgelist=edges,
                       width=1, ax=ax)
    if fig is not None:
        fig.tight_layout()<jupyter_output><empty_output><jupyter_text>And easily plot them all side by side<jupyter_code>fig, axs = plt.subplots(3, 2, figsize=(8, 6.9))
axs = np.array(axs).flatten()

for i, path in enumerate(directed_paths):
    plot_path(G, pos, path, axs[i])

axs[-1].axis('off')
fig.tight_layout()<jupyter_output><empty_output><jupyter_text>In some cases, we're also interested in all the paths, even if they break the direction of the edges. We can extract them all by converting our __DiGraph__ into a regular __Graph__ before calling __nx.all_simple_paths__<jupyter_code>all_paths = {tuple(path) for path in nx.all_simple_paths(G.to_undirected(G), 'X', 'T')}<jupyter_output><empty_output><jupyter_text>And we see that we find two extra paths<jupyter_code>print(all_paths-directed_paths)<jupyter_output>{('X', 'W', 'Z', 'Y', 'T'), ('X', 'Y', 'W', 'Z', 'T')}<jupyter_text>That we can also visualize<jupyter_code>fig, axs = plt.subplots(1, 2, figsize=(8, 2.3))
axs = np.array(axs).flatten()

new_paths = all_paths-directed_paths

for i, path in enumerate(new_paths):
    plot_path(G, pos, path, axs[i])

axs[-1].axis('off')
fig.tight_layout()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1662531017369727, 'fraction_numerical': 0.021940707849027033, 'mean_word_length': 5.251428571428572, 'pattern_counts': {'":': 0, '<': 80, '<?xml version=': 0, '>': 80, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '21', 'dir': 'jupyter-structured-clean-dedup', 'id': '299240', 'license': 'MIT', 'n_tokens_mistral': 3077, 'n_tokens_neox': 2963, 'n_words': 730, 'path': '1.4 - Graphs.ipynb', 'repo_name': 'snowdj/Causality-1', 'size': '7666'}