<jupyter_start><jupyter_text>****** Working with RDF Graphs*****Author:** [TJCallahan](http://tiffanycallahan.com/)  **GitHub Repository:** [PheKnowLator](https://github.com/callahantiff/PheKnowLator/wiki)  **Wiki Page:** [OWL-NETS-2.0](https://github.com/callahantiff/PheKnowLator/wiki/OWL-NETS-2.0)  **Release:** **[v2.0.0](https://github.com/callahantiff/PheKnowLator/wiki/v2.0.0)**      Purpose  The goal of this notebook is to provide some examples of how to manipulate RDF ([Resource Description Framework](https://www.w3.org/RDF/)) graphs using the Python libraries [RDFLib](https://rdflib.readthedocs.io/en/stable/_modules/rdflib/compare.html) and [NetworkX](https://networkx.org/). This tutorial will cover the following concepts:  - [Environment Set-Up](setup)   - [Loading Data](data-load)     - [Exploring Graphs](exploring-graphs)  - [Saving, Serializing, and Writing Graphs](saving-output)    - [Try it Yourself](try-it-yourself)   Resources   - RDFLib    - [Getting started with RDFLib](https://rdflib.readthedocs.io/en/stable/gettingstarted.html)   - NetworkX  - [Introduction to NetworkX](https://networkx.org/documentation/stable//reference/introduction.html)    - [Cambridge NetworkX Tutorial](https://www.cl.cam.ac.uk/teaching/1314/L109/tutorial.pdf)  - SPARQL Query  - [Constructing SPARQL Queries](https://www.w3.org/2009/Talks/0615-qbe/)     - [SPARQL - Medium Article](https://medium.com/wallscope/constructing-sparql-queries-ca63b8b9ac02) ***   Set-Up Environment ***   Dependencies: [`pkt_kg`](https://pypi.org/project/pkt-kg/), [`networkx`](https://pypi.org/project/networkx/), [`rdflib`](https://pypi.org/project/rdflib/)To prepare for the tutorial we need to make sure that the all needed libraries are downloaded and imported. If you don't already have `pkt_kg`, `networkx` and `rdflib` installed, you can extend the code chunk below to include any libraries that you need to download. In addition to downloading needed libraries, you will also need to download graph data. Each data source is briefly described in the next section.<jupyter_code># # uncomment and run to install any required modules from notebooks/requirements.txt
# import sys
# !{sys.executable} -m pip install -r requirements.txt
# # if running a local version of pkt_kg, uncomment the code below
# import sys
# sys.path.append('../')
# import needed libraries
import networkx as nx
import os

from rdflib import Graph, Namespace, URIRef, BNode, Literal
from tqdm import tqdm

from pkt_kg.utils import *  # provides access to helper functions
# import built-in namespaces
from rdflib.namespace import OWL, RDF, RDFS

# create namespaces
obo = Namespace('http://purl.obolibrary.org/obo/')
entrez = Namespace('http://www.ncbi.nlm.nih.gov/gene/')<jupyter_output><empty_output><jupyter_text>Tutorial Data  This tutorial utilizes two separate graphs. The first graph is the [Vaccine Ontology (VO)](http://www.obofoundry.org/ontology/vo.html) and the second graph is one of the [`PheKnowlator`](https://github.com/callahantiff/PheKnowLator) builds. Each of these data sources are briefly described below:  **[`Vaccine Ontology`](http://purl.obolibrary.org/obo/vo.owl)**: For this resource, we will download and utilize the [OWL](https://www.w3.org/TR/owl-features/) file provided by the Open Biomedical Ontologies Foundry. The link used to download this ontology will ensure that no matter when    **[`PheKnowLator Build`](https://storage.googleapis.com/pheknowlator/archived_builds/release_v2.0.0/build_11FEB2021/knowledge_graphs/instance_builds/relations_only/owlnets/PheKnowLator_v2.0.0_full_instance_relationsOnly_noOWL_OWLNETS.nt)**: For this resources, we will utilize data from the `PheKnowLator` instance build, with directed relations, and [`OWL-NETS`](https://github.com/callahantiff/PheKnowLator/wiki/OWL-NETS-2.0) property graph conversion (`PheKnowLator_v2.0.0_full_instance_relationsOnly_noOWL_OWLNETS.nt`). To demonstrate different ways that data can be processed, we will be downloading the data in the [`ntriples`](https://www.w3.org/TR/n-triples/) format.<jupyter_code># Set global variables
write_location = '../temp_dir/'
if not os.path.exists(write_location):
    os.mkdir(write_location)
# download data to the temp_dir directory
data_urls = [
    'http://purl.obolibrary.org/obo/vo.owl',
    'https://storage.googleapis.com/pheknowlator/archived_builds/release_v2.0.0/build_25JAN2021/knowledge_graphs/instance_builds/relations_only/owlnets/PheKnowLator_v2.0.0_full_instance_relationsOnly_noOWL_OWLNETS.nt'
]

for url in data_urls:
    file_name = url.split('/')[-1]
    if not os.path.exists(write_location + file_name):
        data_downloader(url, write_location)<jupyter_output><empty_output><jupyter_text>*** Loading Data *****Relevant Documentation:** [`https://rdflib.readthedocs.io/en/stable/intro_to_parsing.html`](https://rdflib.readthedocs.io/en/stable/intro_to_parsing.html)  This section demonstrates how to load data using `RDFLib`. Please note that there are many different ways that one can load data, which largely depends on the the format of the data you are trying to import. See the link above for details regarding the different types of formats that the library can import.In this tutorial we will be importing data to an `RDFLib` `Graph()` object using two different imports formats:1. `vo.owl` using `xml` format  2. `PheKnowLator_v2.0.0_full_instance_relationsOnly_noOWL_OWLNETS.nt` using `nt` format  Loading `vo.owl`<jupyter_code>vo_graph = Graph()
vo_graph.parse(write_location + 'vo.owl', format='xml')<jupyter_output><empty_output><jupyter_text>*** Exploring Graphs *** Before diving into this tutorial, it's useful to review some of the tutorials on accessing and manipulating `RDFLib` `Graph()` objects. Specifically, it's useful to review the following:  - **[RDFLib Terms](https://rdflib.readthedocs.io/en/stable/rdf_terms.html?highlight=terms)**: `URIRef()`, `Literal()`, and `BNode()`  - **[Triple Pattern Matching](https://rdflib.readthedocs.io/en/stable/intro_to_graphs.htmlbasic-triple-matching)**: `graph.triples()`, `graph.subjects()`, `graph.objects()`, and `graph.predicates()`  - **[Namespace Utilities](https://rdflib.readthedocs.io/en/stable/apidocs/rdflib.html?highlight=namespacenamespace-utilities)**: Provides description of the built-in namespaces (e.g. `OWL`, `RDF`, `RDFS`, etc...)  - **[SPARQL Queries](https://rdflib.readthedocs.io/en/stable/intro_to_sparql.html):** `graph.query()`  Graph Descriptives The code chunks below provide different examples of how to obtain counts of different parts of the graph. Specifically, the code below demonstrates how to obtain counts of `owl:Class` and `owl:ObjectProperty`. *Approach 1:* Obtain counts using a SPARQL query. `RDFLib` documentation for writing SPARQL queries can be found [here](https://rdflib.readthedocs.io/en/stable/intro_to_sparql.html).<jupyter_code># Approach 1: SPARQL query -- note the use of OWL and RDF built-in namespaces
class_query = vo_graph.query(
    """SELECT DISTINCT ?class
    WHERE {
      ?class rdf:type owl:Class .}
    """, initNs={'rdf': RDF,
                 'owl': OWL}) 

print('There are {} owl:Class objects'.format(len(class_query)))
object_query = vo_graph.query(
    """SELECT DISTINCT ?object_properties
    WHERE {
      ?object_properties rdf:type owl:ObjectProperty .}
    """, initNs={'rdf': RDF,
                 'owl': OWL}) 

print('There are {} owl:ObjectProperties objects'.format(len(object_query)))<jupyter_output><empty_output><jupyter_text>*Approach 2:* Obtain counts using `RDFLib` built-in methods. Specifically, the examples below demonstrate the `subjects()` and `triples()` methods. `RDFLib` documentation for these built-in functions can be found [here](https://rdflib.readthedocs.io/en/stable/intro_to_graphs.htmlgraph-methods-for-accessing-triples).<jupyter_code># Approach 2: Iterate over RDFLib graph object
classes = list(vo_graph.subjects(RDF.type, OWL.Class))
objects = list(vo_graph.subjects(RDF.type, OWL.ObjectProperty))
triples = list(vo_graph.triples((None, None, None)))

print('There are {} triples, {} owl:Class objects, and {} owl:ObjectProperties objects'.format(len(triples),
                                                                                               len(classes),
                                                                                               len(objects)))<jupyter_output><empty_output><jupyter_text>Helpful FunctionsThe next code sections provide examples of how to access different types of information from an `RDF` graph.<jupyter_code># get labels for the URI VO_0002186 -- see two different ways to reference a URI
# using obo namespace
vo_0002186_label = vo_graph.label(obo.VO_0002186)
print(str(vo_0002186_label))
# using RDFLIb URIRef term
vo_0002186_label = vo_graph.label(URIRef('http://purl.obolibrary.org/obo/VO_0002186'))
print(str(vo_0002186_label))
# get all triples that VO_0002186 participates in
VO_0002186_triples = list(vo_graph.triples((obo.VO_0002186, None, None))) +\
                     list(vo_graph.triples((None, None, obo.VO_0002186)))

for s, p, o in tqdm(VO_0002186_triples):
#     print(s, p, o)
    print(str(s).split('/')[-1], str(p).split('/')[-1], str(o).split('/')[-1] + '\n')  # converting entities to str
# get all owl:ObjectProperty objects
vo_obj_props = list(vo_graph.subjects(RDF.type, OWL.ObjectProperty))

for p in tqdm(vo_obj_props):
    print(p)<jupyter_output><empty_output><jupyter_text>Obtaining Detailed Network Statistics  While `RDFLib` is a great library for building `RDF` graphs, it lacks functionality to obtain some of the more traditional network statistics. In order to be able to use the `NetworkX` functions, we first need to convert the `RDFLib` `Graph()` object into a `NetworkX` [MultiDiGraph](https://networkx.github.io/documentation/stable/reference/classes/multidigraph.html). You will notice, if you read the `RDFLib` documentation that there are methods that will do this for you. In my experience, they are incredibly slow, it's much faster to use the code shown below.From [this](https://networkx.org/documentation/stable/reference/classes/multidigraph.html) `NetworkX` documentation:  > A directed graph class that can store multiedges.  >  > Multiedges are multiple edges between two nodes. Each edge can hold optional data or attributes.  >  > A MultiDiGraph holds directed edges. Self loops are allowed.  >  > Nodes can be arbitrary (hashable) Python objects with optional key/value attributes. By convention None is not used as a node. >  > Edges are represented as links between nodes with optional key/value attributes.<jupyter_code># convert RDFLib graph to Networkx MultiDiGraph
nx_graph = nx.MultiDiGraph()

for s, p, o in tqdm(vo_graph):
    nx_graph.add_edge(s, o, **{'key': p})
# get the number of nodes, edges, and self-loops
nodes = nx.number_of_nodes(nx_graph)
edges = nx.number_of_edges(nx_graph)
self_loops = nx.number_of_selfloops(nx_graph)

print('There are {} nodes, {} edges, and {} self-loop(s)'.format(nodes, edges, self_loops))
# get degree information
avg_degree = float(edges) / nodes

print('The Average Degree is {}'.format(avg_degree))
# get 5 nodes with the highest degress
n_deg = sorted([(str(x[0]), x[1]) for x in  nx_graph.degree], key=lambda x: x[1], reverse=1)[:6]

for x in n_deg:
    print('{} (degree={})'.format(x[0], x[1]))
# get network density
density = nx.density(nx_graph)

print('The density of the graph is: {}'.format(density))
# get connected components -- have to convert MultiDiGraph to undirected graph
nx_graph_und = nx_graph.to_undirected()

# get connected components
components = sorted(list(nx.connected_components(nx_graph_und)), key=len, reverse=True)
cc_content = {x: str(len(components[x])) + ' nodes: ' + ' | '.join(components[x]) if len(components[x]) < 500
              else len(components[x]) for x in range(len(components))}

for k, v in cc_content.items():
    if isinstance(v, int):
        print('COMPONENT: {} Consists of {} nodes'.format(str(k), str(v)))
    else:
        print('\nCOMPONENT: {} Consists of the following nodes:'.format(str(k)))
        for node in v.split(': ')[-1].split(' | '):
            print(node)<jupyter_output><empty_output><jupyter_text>Shortest PathThis chunk shows how to find the shortest path between a starting node and all nodes reachable from that node. From [this](https://networkx.org/documentation/stable//reference/algorithms/generated/networkx.algorithms.shortest_paths.unweighted.single_source_shortest_path.htmlnetworkx.algorithms.shortest_paths.unweighted.single_source_shortest_path) `NetworkX` page:  > Compute shortest path between source and all other nodes reachable from source.> > **Parameters:**>   - G (NetworkX graph)>   - source (node label) – Starting node for path>   - cutoff (integer, optional) – Depth to stop the search. Only paths of length <= cutoff are returned.> > **Returns:**   >   lengths – Dictionary, keyed by target, of shortest paths.  >> **Return type:**  > dictionary<jupyter_code># get shortest path from VO_0002186
vo_0002186_path = nx.single_source_shortest_path(nx_graph, obo.VO_0002186)

for k, v in vo_0002186_path.items():
    if k != obo.VO_0002186:
        print('\n{} - {} Path:'.format(str(obo.VO_0002186).split('/')[-1], str(k).split('/')[-1]))
        for i in v:
            print(i)<jupyter_output><empty_output><jupyter_text>*** Saving, Serializing, and Writing Graphs ***Below, I provide examples for how to save data using `NetworkX` and `RDFLib`.  NetworkX  **Documentation:** [Reading and Writing Data](https://networkx.org/documentation/stable//reference/readwrite/index.html?highlight=write%20data)Here, I provide an example of how to save the `MultiDiGraph` version of the `vo_graph` object so it can be used in the future without having to create it from the `RDFLib` graph object.<jupyter_code># save multidigraph version of graph
nx.write_gpickle(nx_graph, write_location + 'vo_NetworkxMultiDiGraph.gpickle')

# read in multidigraph version of graph
# nx_graph = nx.read_gpickle(write_location + 'vo_NetworkxMultiDiGraph.gpickle')<jupyter_output><empty_output><jupyter_text>RDFLib  **Documentation:** [Graph Serialization](https://rdflib.readthedocs.io/en/latest/apidocs/rdflib.htmlrdflib.graph.Graph.serialize)Here, I provide an example of how to save the `vo_graph` object as an `OWL` file and as an `n-triples` file.<jupyter_code># save vo_graph as `ntriple` format
vo_graph.serialize(write_location + 'vo_graph_data.nt', format='nt')
# save vo_graph as OWL (i.e. RDF/XML)
vo_graph.serialize(write_location + 'vo_graph_data.owl', format='xml')<jupyter_output><empty_output><jupyter_text>*** Try it Out for Yourself - Explore the PheKnowLator Graph Build ***You will notice that while we downloaded the `pkt_kg` graph data `PheKnowLator_v2.0.0_full_instance_relationsOnly_noOWL_OWLNETS.nt`, we never used it. The remainder of this tutorial is left for you. I encourage you to use the functionality introduced above to explore this graph. If you do something interesting and want to share it, please make a pull-request and add it to this tutorial!  Loading `PheKnowLator_v2.0.0_full_instance_relationsOnly_noOWL_OWLNETS.nt`  Please note that running the chunk below can take up to 45 min as the file is 2.0 GB.<jupyter_code>pkt_graph = Graph()
pkt_graph.parse(write_location + 'PheKnowLator_v2.0.0_full_instance_relationsOnly_noOWL_OWLNETS.nt', format='nt')
# get basic descriptive statistics
nodes = set(list(pkt_graph.subjects()) + list(pkt_graph.objects()))
rels = set(list(pkt_graph.predicates()))

# print stats
print('Graph Stats: {} triples, {} nodes, {} predicates'.format(len(pkt_graph), len(nodes), len(rels)))<jupyter_output><empty_output><jupyter_text>Create Subgraph  Create a subgraph from the `pkt_kg` build that only contains the following types of edges: `gene-drug`, `drug-disease`, `gene-disease`. To do this, we filter the graph first by relation and then by node types. Examples for each relation type are listed below:  `drug-gene`: - Find all triples that include the predicate: *interacts with* (`obo:RO_0002434`)    - Only keep subjects with the following URI: `obo:CHEBI_XXXXXXX`    - Only keep objects with the following URI: `http://www.ncbi.nlm.nih.gov/gene/``drug-disease`:  - Find all triples that include the predicate: *substance that treats* (`obo:RO_0002606`)    - Only keep subjects with the following URI: `obo:CHEBI_XXXXXXX`    - Only keep objects with the following URI: `obo:MONDO_XXXXXXX``gene-disease`:  - Find all triples that include the predicate: *causes or contributes to* (`obo.RO_0003302`)    - Only keep subjects with the following URI: `http://www.ncbi.nlm.nih.gov/gene/X`    - Only keep objects with the following URI: `obo:MONDO_XXXXXXX`<jupyter_code>def filters_edge_data(graph, relation, subj, obj):
    """Method takes an input RDFLib graph and filters it using an the input relation, subj, and obj variables.
    
    Args:
        graph: An RDFLib Graph object.
        relation: A URIRef object containing a Relation Ontology relation. 
        subj: A URIRef object containing information to filter subjects by.
        obj: A URIRef object containing information to filter objects by.
    
    Returns:
        filtered_edges: 
    """
    
    filtered_triples = []

    for s, p, o in tqdm(graph):
        if p == relation:
            if str(s).startswith(str(subj)) and str(o).startswith(str(obj)):
                filtered_triples += [(s, p, o)]
    
    return filtered_triples
# get gene-drug edges - then keep subjects with 
drug_gene_triples = filters_edge_data(graph=pkt_graph,
                                      relation=obo.RO_0002434,
                                      subj=obo.CHEBI,
                                      obj=entrez)

print('There are {} drug-gene edges'.format(len(drug_gene_triples)))
# get drug-disease edges
drug_disease_triples = filters_edge_data(graph=pkt_graph,
                                         relation=obo.RO_0002606,
                                         subj=obo.CHEBI,
                                         obj=obo.MONDO)

print('There are {} drug-disease edges'.format(len(drug_disease_triples)))
# get gene-disease edges
gene_disease_triples = filters_edge_data(graph=pkt_graph,
                                         relation=obo.RO_0003302,
                                         subj=entrez,
                                         obj=obo.MONDO)

print('There are {} gene-disease edges'.format(len(gene_disease_triples)))
# combine triples into single graph
filtered_edges = drug_gene_triples + drug_disease_triples + gene_disease_triples
gene_drug_disease_graph = adds_edges_to_graph(Graph(), filtered_edges)

print('The drug-gene-disease Subgraph contains {} triples'.format(len(gene_drug_disease_graph)))<jupyter_output><empty_output><jupyter_text>Convert Subgraph to NetworkX MultiDiGraph<jupyter_code># convert RDFLib graph to Networkx MultiDiGraph
nx_graph_dgd = nx.MultiDiGraph()

for s, p, o in tqdm(gene_drug_disease_graph):
    nx_graph_dgd.add_edge(s, o, **{'key': p})<jupyter_output><empty_output><jupyter_text>Get Graph Descriptives<jupyter_code># get the number of nodes, edges, and self-loops
nodes = nx.number_of_nodes(nx_graph_dgd)
edges = nx.number_of_edges(nx_graph_dgd)
avg_degree = float(edges) / nodes

print('There are {} nodes, {} edges, and has an average degree of {}'.format(nodes, edges, avg_degree))
# get 5 nodes with the highest degress
n_deg = sorted([(str(x[0]), x[1]) for x in  nx_graph_dgd.degree], key=lambda x: x[1], reverse=1)[:6]

for x in n_deg:
    print('{} (degree={})'.format(x[0], x[1]))
# get connected components -- have to convert MultiDiGraph to undirected graph
nx_graph_und = nx_graph_dgd.to_undirected()

# get connected components
components = sorted(list(nx.connected_components(nx_graph_und)), key=len, reverse=True)
cc_content = {x: str(len(components[x])) + ' nodes: ' + ' | '.join(components[x]) if len(components[x]) < 500
              else len(components[x]) for x in range(len(components))}

for k, v in cc_content.items():
    if isinstance(v, int):
        print('COMPONENT: {} Consists of {} nodes'.format(str(k), str(v)))
    else:
        print('\nCOMPONENT: {} Consists of the following nodes:'.format(str(k)))
        for node in v.split(': ')[-1].split(' | '):
            print(node)<jupyter_output><empty_output><jupyter_text>Explore GraphExample shown below searches for all shortest paths between valporic acid ([CHEBI_39867](https://www.ebi.ac.uk/chebi/searchId.do?chebiId=CHEBI:39867)) and all other targets in the graph. This search returns several results, one example is shown below:> **Path between CHEBI_39867 - MONDO_0012489:**  > http://purl.obolibrary.org/obo/CHEBI_39867  > http://www.ncbi.nlm.nih.gov/gene/1413  > http://purl.obolibrary.org/obo/MONDO_0012489  Where [entrez gene 1413](http://www.ncbi.nlm.nih.gov/gene/1413) is CRYBA4 (crystallin beta A4) and [MONDO_0012489](http://www.ontobee.org/ontology/MONDO?iri=http://purl.obolibrary.org/obo/MONDO_0012489) is early-onset non-syndromic cataract caused by mutation in `CRYBA4`. A quick Googling returns several relevant results to connect these entities.<jupyter_code># perform bidirectional search for path between -- epilepsy and valporic acid
nx.bidirectional_shortest_path(nx_graph_dgd, obo.CHEBI_39867, obo.MONDO_0005027)
# look at shortest path between 
shortest_paths = nx.shortest_path(nx_graph_dgd, source=obo.CHEBI_39867)

for k, v in shortest_paths.items():
    if k != obo.CHEBI_39867:
        print('\n{} - {} Path:'.format(str(obo.CHEBI_39867).split('/')[-1], str(k).split('/')[-1]))
        for i in v:
            print(i)<jupyter_output><empty_output><jupyter_text>Save Output<jupyter_code># save subgraph
gene_drug_disease_graph.serialize(write_location + 'pkt_DrugGeneDisease_subgraph.nt', format='nt')
nx.write_gpickle(nx_graph_dgd, write_location + 'pkt_DrugGeneDisease_NetworkxMultiDiGraph.gpickle')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11948571168960974, 'fraction_numerical': 0.017173231566035164, 'mean_word_length': 4.482440846824408, 'pattern_counts': {'":': 0, '<': 68, '<?xml version=': 0, '>': 90, 'https://': 34, 'lorem ipsum': 0, 'www.': 13, 'xml': 3}, 'pii_count': 0, 'substrings_counts': 4, 'word_list_counts': {'cursed_substrings.json': 33, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '385908', 'license': 'Apache-2.0', 'n_tokens_mistral': 7469, 'n_tokens_neox': 6939, 'n_words': 1893, 'path': 'notebooks/RDF_Graph_Processing_Example.ipynb', 'repo_name': 'Bancherd-DeLong/PheKnowLator', 'size': '22011'}