<jupyter_start><jupyter_text>IntroductionThis notebook demonstrates basic usage of BioThings Explorer, an engine for autonomously querying a distributed knowledge graph. BioThings Explorer can answer two classes of queries -- "PREDICT" and "EXPLAIN".  PREDICT queries are described in [PREDICT_demo.ipynb](PREDICT_demo.ipynb). Here, we describe EXPLAIN queries and how to use BioThings Explorer to execute them.  A more detailed overview of the BioThings Explorer systems is provided in [these slides](https://docs.google.com/presentation/d/1QWQqqQhPD_pzKryh6Wijm4YQswv8pAjleVORCPyJyDE/edit?usp=sharing).EXPLAIN queries are designed to **identify plausible reasoning chains to explain the relationship between two entities**.  For example, in this notebook, we explore the question:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"*Why does imatinib have an effect on the treatment of chronic myelogenous leukemia (CML)?*"  Later, we also compare those results to a similar query looking at imatinib's role in treating gastrointestinal stromal tumors (GIST).**To experiment with an executable version of this notebook, [load it in Google Colaboratory](https://colab.research.google.com/github/biothings/biothings_explorer/blob/master/jupyter%20notebooks/EXPLAIN_demo.ipynb).**  Step 0: Load BioThings Explorer modules First, install the `biothings_explorer` and `biothings_schema` packages, as described in this [README](https://github.com/biothings/biothings_explorer/blob/master/jupyter%20notebooks/README.mdprerequisite).  This only needs to be done once (but including it here for compability with [colab](https://colab.research.google.com/)).<jupyter_code>%%capture
!pip install git+https://github.com/biothings/biothings_explorer#egg=biothings_explorer<jupyter_output><empty_output><jupyter_text>Next, import the relevant modules:* **Hint**: Find corresponding bio-entity representation used in BioThings Explorer based on user input (could be any database IDs, symbols, names)* **FindConnection**: Find intermediate bio-entities which connects user specified input and output<jupyter_code># import modules from biothings_explorer
from biothings_explorer.hint import Hint
from biothings_explorer.user_query_dispatcher import FindConnection
import nest_asyncio
nest_asyncio.apply()<jupyter_output><empty_output><jupyter_text>Step 1: Find representation of "chronic myelogenous leukemia" and "imatinib" in BTEIn this step, BioThings Explorer translates our query strings "chronic myelogenous leukemia" and "imatinib" into BioThings objects, which contain mappings to many common identifiers.  Generally, the top result returned by the `Hint` module will be the correct item, but you should confirm that using the identifiers shown.Search terms can correspond to any child of [BiologicalEntity](https://biolink.github.io/biolink-model/docs/BiologicalEntity.html) from the [Biolink Model](https://biolink.github.io/biolink-model/docs/), including `DiseaseOrPhenotypicFeature` (e.g., "lupus"), `ChemicalSubstance` (e.g., "acetaminophen"), `Gene` (e.g., "CDK2"), `BiologicalProcess` (e.g., "T cell differentiation"), and `Pathway` (e.g., "Citric acid cycle").<jupyter_code>ht = Hint()
# find all potential representations of CML
cml_hint = ht.query("chronic myelogenous leukemia")
# select the correct representation of CML
cml = cml_hint['Disease'][0]
cml
# find all potential representations of imatinib
imatinib_hint = ht.query("imatinib")
# select the correct representation of imatinib
imatinib = imatinib_hint['ChemicalSubstance'][0]
imatinib<jupyter_output><empty_output><jupyter_text>Step 2: Find intermediate nodes connecting imatinib and chronic myelogenous leukemiaIn this section, we find all paths in the knowledge graph that connect imatinib and chronic myelogenous leukemia.  To do that, we will use `FindConnection`.  This class is a convenient wrapper around two advanced functions for **query path planning** and **query path execution**. More advanced features for both query path planning and query path execution are in development and will be documented in the coming months. The parameters for `FindConnection` are described below:<jupyter_code>help(FindConnection.__init__)<jupyter_output>Help on function __init__ in module biothings_explorer.user_query_dispatcher:

__init__(self, input_obj, output_obj, intermediate_nodes, registry=None)
    Find relationships in the Knowledge Graph between an Input Object and an Output Object.
    
    Args:
        input_obj (required): must be an object returned from Hint corresponding to a specific biomedical entity.
                            Examples:
                Hint().query("Fanconi anemia")['DiseaseOrPhenotypicFeature'][0]
                Hint().query("acetaminophen")['ChemicalSubstance'][0]
    
        output_obj (required): must EITHER be an object returned from Hint corresponding to a specific biomedical
                            entity, OR be a string or list of strings corresponding to Biolink Entity classes.
                            Examples:
                Hint().query("acetaminophen")['ChemicalSubstance'][0]
                'Gene'
                ['Gene','ChemicalSubstance']
    
        intermediate_nodes ([...]<jupyter_text>Here, we formulate a `FindConnection` query with "CML" as the `input_ojb`, "imatinib" as the `output_obj`.  We further specify with the `intermediate_nodes` parameter that we are looking for paths joining chronic myelogenous leukemia and imatinib with *one* intermediate node that is a Gene.  (The ability to search for longer reasoning paths that include additional intermediate nodes will be added shortly.)<jupyter_code>fc = FindConnection(input_obj=cml, output_obj=imatinib, intermediate_nodes='Gene')<jupyter_output><empty_output><jupyter_text>We next execute the `connect` method, which performs the **query path planning** and **query path execution** process.  In short, BioThings Explorer is deconstructing the query into individual API calls, executing those API calls, then assembling the results.A verbose log of this process is displayed below:<jupyter_code># set verbose=True will display all steps which BTE takes to find the connection
fc.connect(verbose=True)<jupyter_output>==========
========== QUERY PARAMETER SUMMARY ==========
==========

BTE will find paths that join 'chronic myelogenous leukemia' and 'imatinib'. Paths will have 1 intermediate node.

Intermediate node #1 will have these type constraints: Gene


==========
========== QUERY #1 -- fetch all Gene entities linked to 'chronic myelogenous leukemia' ==========
==========

==== Step #1: Query path planning ====

Because chronic myelogenous leukemia is of type 'Disease', BTE will query our meta-KG for APIs that can take 'Disease' as input and 'Gene' as output

BTE found 10 apis:

API 1. semmed_disease(15 API calls)
API 2. cord_disease(1 API call)
API 3. mgi_gene2phenotype(1 API call)
API 4. scibite(1 API call)
API 5. hetio(1 API call)
API 6. mydisease(1 API call)
API 7. DISEASES(1 API call)
API 8. pharos(1 API call)
API 9. scigraph(1 API call)
API 10. biolink(1 API call)


==== Step #2: Query path execution ====
NOTE: API requests are dispatched in parallel, so the list of APIs below is ordered[...]<jupyter_text>Step 3: Display and Filter resultsThis section demonstrates post-query filtering done in Python. Later, more advanced filtering functions will be added to the **query path execution** module for interleaved filtering, thereby enabling longer query paths. More details to come...First, all matching paths can be exported to a data frame. Let's examine a sample of those results.<jupyter_code>df = fc.display_table_view()

# because UMLS is not currently well-integrated in our ID-to-object translation system, removing UMLS-only entries here
patternDel = "^UMLS:C\d+"
filter = df.node1_id.str.contains(patternDel)
df = df[~filter]

print(df.shape)
df.sample(10)<jupyter_output>(1621, 16)<jupyter_text>While most results are based on edges from [semmed](https://skr3.nlm.nih.gov/SemMed/), edges from [DGIdb](http://www.dgidb.org/), [biolink](https://monarchinitiative.org/), [disgenet](http://www.disgenet.org/), [mydisease.info](https://mydisease.info) and [drugcentral](http://drugcentral.org/) were also retrieved from their respective APIs.  Next, let's look to see which genes are mentioned the most.<jupyter_code>df.node1_name.value_counts().head(10)<jupyter_output><empty_output><jupyter_text>Not surprisingly, the top two genes that BioThings Explorer found that join imatinib to CML are *ABL1* and *BCR*, the two genes that are fused in the "Philadelphia chromosome", the genetic abnormality that underlies CML, *and* the validate target of imatinib.Let's examine some of the PubMed articles linking **CML to *ABL1*** and ***ABL1* to imatinib**.<jupyter_code># fetch all articles connecting 'chronic myelogenous leukemia' and 'ABL1'
articles = []
for info in fc.display_edge_info('chronic myelogenous leukemia', 'ABL1').values():
    if 'pubmed' in info['info']:
        articles += info['info']['pubmed']
print("There are "+str(len(articles))+" articles supporting the edge between CML and ABL1. Sampling of 10 of those:")
x = [print("http://pubmed.gov/"+str(x)) for x in articles[0:10] ]
# fetch all articles connecting 'ABL1' and 'Imatinib
articles = []
for info in fc.display_edge_info('ABL1', 'imatinib').values():
    if 'pubmed' in info['info']:
        articles += info['info']['pubmed']
print("There are "+str(len(articles))+" articles supporting the edge between ABL1 and imatinib. Sampling of 10 of those:")
x = [print("http://pubmed.gov/"+str(x)) for x in articles[0:10] ]<jupyter_output>There are 32 articles supporting the edge between ABL1 and imatinib. Sampling of 10 of those:
http://pubmed.gov/15799618
http://pubmed.gov/15917650
http://pubmed.gov/15949566
http://pubmed.gov/16153117
http://pubmed.gov/16205964
http://pubmed.gov/15713800
http://pubmed.gov/15713800
http://pubmed.gov/15713800
http://pubmed.gov/19166098
http://pubmed.gov/26030291<jupyter_text>Comparing results between CML and GISTLet's perform another BioThings Explorer query, this time looking to EXPLAIN the relationship between imatinib and gastrointestinal stromal tumors (GIST), another disease treated by imatinib.<jupyter_code>ht = Hint()
# find all potential representations of CML
gist_hint = ht.query("gastrointestinal stromal tumor")
# select the correct representation of CML
gist = gist_hint['Disease'][0]
gist
fc = FindConnection(input_obj=gist, output_obj=imatinib, intermediate_nodes='Gene')
fc.connect(verbose=False) # skipping the verbose log here
df = fc.display_table_view()

# because UMLS is not currently well-integrated in our ID-to-object translation system, removing UMLS-only entries here
patternDel = "^UMLS:C\d+"
filter = df.node1_id.str.contains(patternDel)
df = df[~filter]

print(df.shape)
df.sample(10)
df.node1_name.value_counts().head(10)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10132480584741892, 'fraction_numerical': 0.015714938328003653, 'mean_word_length': 5.412419449326303, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 37, 'https://': 10, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '132867', 'license': 'Apache-2.0', 'n_tokens_mistral': 3448, 'n_tokens_neox': 3094, 'n_words': 1135, 'path': 'jupyter notebooks/EXPLAIN_demo.ipynb', 'repo_name': 'AlexanderPico/biothings_explorer', 'size': '10945'}