<jupyter_start><jupyter_text>Begin Queries  Taxon Causes Disease (2 syntax, 2 approches)Theyre are two syntax for taxon causes disease that exist in the WikiData data model:1. Direct statements:      Taxon has-effect Disease... or Disease has-cause Taxon 2. Qualifier Statements:      Disease has-cause infection (qual: of Taxon)     There are also two approaches:1. Direct links:          Taxon has-effect Disease2. Punning down to a specific taxonomic level:      Partent_taxon has-effect Disease      Taxon has-parent* Parent_taxon      Taxon has-rank Species  Syntax 1: disease has-cause infection - qualifier: of TAXON<jupyter_code>q = """    SELECT DISTINCT ?disease ?taxon ?taxonLabel ?tax_id
    WHERE 
    {
     {?disease wdt:P31 wd:Q12136}UNION{?disease wdt:P699 ?doid}.

     ?disease p:P828 [ps:P828 wd:Q166231;
                       pq:P642 ?taxon;].
      OPTIONAL{?taxon wdt:P685 ?tax_id}.
          
      SERVICE wikibase:label { bd:serviceParam wikibase:language "[AUTO_LANGUAGE],en". }
    }"""
qr = execute_sparql_query(q)
tax_nodes = process_taxa(qr)
edge_res = standardize_edges(qr, 'taxon', 'disease', 'causes')

nodes.append(tax_nodes)
edges.append(edge_res)

edge_res.head()
tax_nodes.head()<jupyter_output><empty_output><jupyter_text>Syntax 2: disease has-cause TAXON or TAXON has-effect Disease<jupyter_code>q = """    SELECT DISTINCT ?disease ?diseaseLabel ?doid ?taxon ?taxonLabel ?tax_id
    WHERE 
    {
       {?disease wdt:P31 wd:Q12136}UNION{?disease wdt:P699 ?doid}.
      
        #?taxon wdt:P31 wd:Q16521.
        ?taxon wdt:P685 ?tax_id. 
      
       {?disease wdt:P828 ?taxon}UNION{?taxon wdt:P1542 ?disease}.
      
        OPTIONAL {?disease wdt:P699 ?doid.}
      
      SERVICE wikibase:label { bd:serviceParam wikibase:language "[AUTO_LANGUAGE],en". }
    }"""
q = """    SELECT DISTINCT ?disease ?diseaseLabel ?doid ?taxon ?taxonLabel ?tax_id
    WHERE 
    {
       {?disease wdt:P31 wd:Q12136}UNION{?disease wdt:P699 ?doid}.
      
        #?taxon wdt:P31 wd:Q16521.
        ?taxon wdt:P685 ?tax_id. 
      
       {?disease wdt:P828 ?taxon}UNION{?taxon wdt:P1542 ?disease}.
      
        OPTIONAL {?disease wdt:P699 ?doid.}
      
      SERVICE wikibase:label { bd:serviceParam wikibase:language "[AUTO_LANGUAGE],en". }
    }"""
qr = execute_sparql_query(q)
tax_nodes = process_taxa(qr)
edge_res = standardize_edges(qr, 'taxon', 'disease', 'causes')

nodes.append(tax_nodes)
edges.append(edge_res)

edge_res.head()
tax_nodes.head(2)<jupyter_output><empty_output><jupyter_text>Syntax 1, mode 2: disease has-cause infection - qualifier: of TAXON -> extend down parent taxa to SPECIES LEVEL<jupyter_code>q = """    SELECT DISTINCT ?disease ?diseaseLabel ?doid ?parent_tax ?parent_taxLabel ?par_taxid ?taxon ?taxonLabel ?tax_id
    WHERE 
    {
      {?disease wdt:P31 wd:Q12136}UNION{?disease wdt:P699 ?doid}.

      
      ?disease p:P828 [ps:P828 wd:Q166231;
                       pq:P642 ?parent_tax;].
      OPTIONAL{?disease wdt:P699 ?doid}.
      OPTIONAL{?parent_tax wdt:P685 ?par_taxid}.
      
      # No kingdoms
      FILTER NOT EXISTS {?parent_tax wdt:P105 wd:Q36732}.
      # No Superphyla
      FILTER NOT EXISTS {?parent_tax wdt:P105 wd:Q3978005}.

      
      {?taxon wdt:P171+ ?parent_tax}UNION{?parent_tax wdt:P171+ ?taxon}
      ?taxon wdt:P105 wd:Q7432 .
      ?taxon wdt:P685 ?tax_id
          
      SERVICE wikibase:label { bd:serviceParam wikibase:language "[AUTO_LANGUAGE],en". }
    }"""
qr = execute_sparql_query(q)
tax_nodes = process_taxa(qr)
edge_res = standardize_edges(qr, 'taxon', 'disease', 'causes', 'computed')
edge_res['comp_type'] = 'punning'

nodes.append(tax_nodes)
edges.append(edge_res)

edge_res.head()
tax_nodes.head()<jupyter_output><empty_output><jupyter_text>Syntax 2, mode 2: disease has-cause TAXON  or TAXON has-effct DISEASE -> extend down parent taxa to SPECIES LEVEL<jupyter_code>q = """    SELECT DISTINCT ?disease ?diseaseLabel ?doid ?parent_tax ?parent_taxLabel ?parent_tax_id ?taxon ?taxonLabel ?tax_id
    WHERE 
    {
       {?disease wdt:P31 wd:Q12136}UNION{?disease wdt:P699 ?doid}.
      
        #?tax wdt:P31 wd:Q16521.
        ?parent_tax wdt:P685 ?parent_tax_id. 
      
      # No kingdoms
      FILTER NOT EXISTS {?parent_tax wdt:P105 wd:Q36732}.
      # No Superphyla
      FILTER NOT EXISTS {?parent_tax wdt:P105 wd:Q3978005}.      
       {?disease wdt:P828 ?parent_tax}UNION{?parent_tax wdt:P1542 ?disease}.
      
        OPTIONAL {?disease wdt:P699 ?doid.}
      {?taxon wdt:P171+ ?parent_tax}UNION{?parent_tax wdt:P171+ ?taxon}
      ?taxon wdt:P685 ?tax_id .
      ?taxon wdt:P105 wd:Q7432 .
      
      SERVICE wikibase:label { bd:serviceParam wikibase:language "[AUTO_LANGUAGE],en". }
    }
"""
qr = execute_sparql_query(q)
tax_nodes = process_taxa(qr)
edge_res = standardize_edges(qr, 'taxon', 'disease', 'causes', 'computed')
edge_res['comp_type'] = 'punning'

nodes.append(tax_nodes)
edges.append(edge_res)

edge_res.head()
tax_nodes.head()
tax_nodes = pd.concat(nodes, sort=False, ignore_index=True).drop_duplicates(subset=['id'])
nodes = [tax_nodes]<jupyter_output><empty_output><jupyter_text>Go back and re-run taxon limited to protein and gene queries with each of the infectious taxa<jupyter_code>len(tax_nodes)<jupyter_output><empty_output><jupyter_text>This is too many to queryThis is a very large number of Taxa to have to query for... However, WikiData may have far fewer taxa that actually have annotations to genes or proteins, so lets see how many of each are in WD.<jupyter_code>q = """    SELECT DISTINCT ?taxon
    WHERE 
    {
      ?protein wdt:P31 wd:Q8054.
      ?protein wdt:P703 ?taxon.
    }"""
qr = execute_sparql_query(q)
prot_taxa = set(qr['taxon'])
len(prot_taxa)<jupyter_output><empty_output><jupyter_text>This is already a managable number, but lets also check the genes, as they're separate queries...<jupyter_code>q = """    SELECT DISTINCT ?taxon
    WHERE 
    {
      ?gene wdt:P31 wd:Q7187.
      ?gene wdt:P703 ?taxon.

    }"""
qr = execute_sparql_query(q)
gene_taxa = set(qr['taxon'])
len(gene_taxa)<jupyter_output><empty_output><jupyter_text>Again, another manageable number, lets see how they overlap with what we've recieved for infectious taxa<jupyter_code>len(prot_taxa & set(tax_nodes['id']))<jupyter_output><empty_output><jupyter_text>Awesome, very queryable<jupyter_code>len(gene_taxa & set(tax_nodes['id']))<jupyter_output><empty_output><jupyter_text>Similarly querable  Querying proteins of infectious taxa<jupyter_code>q = """SELECT DISTINCT ?protein ?proteinLabel ?uniprot
        WHERE {{

          # Initial typing for Protein
          ?protein wdt:P31 wd:Q8054.
          ?protein wdt:P703 wd:{tax}.

          OPTIONAL{{?protein wdt:P352 ?uniprot .}}
          SERVICE wikibase:label {{ bd:serviceParam wikibase:language "[AUTO_LANGAGE],en" }}
        }}"""
tax_prot_edges = []

for tax_id in prot_taxa & set(tax_nodes['id']):
    this_q = q.format(tax=tax_id)
    res = node_query_pipeline(this_q, {'uniprot':'UniProt'}, 'protein')
    if res is None:
        continue
    
    nodes.append(res[['id', 'name', 'label', 'xrefs']].copy())
    
    res['tax'] = tax_id
    res_edges = standardize_edges(res, 'id', 'tax', 'in_taxon')
    tax_prot_edges.append(res_edges)
prot_tax = pd.concat(tax_prot_edges, sort=False, ignore_index=True)
edges.append(prot_tax)
prot_tax.head()
prot_tax['end_id'].nunique()<jupyter_output><empty_output><jupyter_text>Querying genes of infectious taxa<jupyter_code>q = """SELECT DISTINCT ?gene ?geneLabel ?entrez ?symbol ?hgnc ?omim ?ensembl
        WHERE {{

          # Initial typing for Gene
          ?gene wdt:P31 wd:Q7187.
          ?gene wdt:P703 wd:{tax}.

          OPTIONAL{{?gene wdt:P351 ?entrez .}}
          OPTIONAL{{?gene wdt:P353 ?symbol .}}
          OPTIONAL{{?gene wdt:P354 ?hgnc .}}
          OPTIONAL{{?gene wdt:P492 ?omim .}}
          OPTIONAL{{?gene wdt:P594 ?ensembl .}}

          SERVICE wikibase:label {{ bd:serviceParam wikibase:language "[AUTO_LANGAGE],en" }}
        }}"""
tax_gene_edges = []
gene_curi_map = {'entrez': 'NCBIGene', 'symbol': 'SYM', 'hgnc':'HGNC', 'omim':'OMIM', 'ensembl':'ENSG'}

for tax_id in gene_taxa & set(tax_nodes['id']):
    this_q = q.format(tax=tax_id)
    
    res = node_query_pipeline(this_q, gene_curi_map, 'gene')
    if res is None:
        continue
    
    nodes.append(res[['id', 'name', 'label', 'xrefs']].copy())
    
    res['tax'] = tax_id
    res_edges = standardize_edges(res, 'id', 'tax', 'in_taxon')
    tax_gene_edges.append(res_edges)
gene_tax = pd.concat(tax_gene_edges, sort=False, ignore_index=True)
edges.append(gene_tax)
gene_tax.head()
gene_tax['end_id'].nunique()<jupyter_output><empty_output><jupyter_text>Gene ENCODES ProteinThis edge will be used to map Genes to Proteins... our final network will likely not distinguish between the two, and therefore sources that only map to one (e.g. gene) will need this to map to the other (protein).<jupyter_code>q = """SELECT DISTINCT ?gene ?protein 
WHERE {{

  # Initial typing for Gene
  ?gene wdt:P31 wd:Q7187.
  ?gene wdt:P703 wd:{tax}.
  # Initial typing for Protein
  ?protein wdt:P31 wd:Q8054.
  ?protein wdt:P703 wd:{tax}.
  
  {{?gene wdt:P688 ?protein}}UNION{{?protein wdt:P702 ?gene}}
}}"""
human_tax_id = 'Q15978631'
encodes_edges = []

infectious_tax = list(set(gene_tax['end_id']) & set(prot_tax['end_id']))

for tax in infectious_tax + [human_tax_id]:
    this_q = q.format(tax=tax)
    
    qr = execute_sparql_query(this_q)
    if qr is not None:
        this_edge = standardize_edges(qr, 'gene', 'protein', 'encodes')
        encodes_edges.append(this_edge)
encodes_edges = pd.concat(encodes_edges, sort=False, ignore_index=True)
edges.append(encodes_edges)
len(encodes_edges)<jupyter_output><empty_output><jupyter_text>Protein MF Molecular Function<jupyter_code>q = """SELECT DISTINCT ?protein ?molecular_function
WHERE {

  ?protein wdt:P31 wd:Q8054.
  ?molecular_function wdt:P686 ?goid.
  
  {?protein wdt:P680 ?molecular_function}
}"""
qr = execute_sparql_query(q)
edge_res = standardize_edges(qr, 'protein', 'molecular_function', 'enables')

edges.append(edge_res)
edge_res.head(2)<jupyter_output><empty_output><jupyter_text>Protein BP Biological Process<jupyter_code>q = """SELECT DISTINCT ?protein ?biological_process
WHERE {

  ?protein wdt:P31 wd:Q8054.
  ?biological_process wdt:P686 ?goid.
  
  {?protein wdt:P682 ?biological_process}
}"""
qr = execute_sparql_query(q)
edge_res = standardize_edges(qr, 'protein', 'biological_process', 'involved_in')

edges.append(edge_res)
edge_res.head()<jupyter_output><empty_output><jupyter_text>Protein CC Cellular Component<jupyter_code>q = """SELECT DISTINCT ?protein ?cell_component
WHERE {

  ?protein wdt:P31 wd:Q8054.
  ?cell_component wdt:P686 ?goid.
  
  {?protein wdt:P681 ?cell_component}
}"""
qr = execute_sparql_query(q)
edge_res = standardize_edges(qr, 'protein', 'cell_component', 'part_of')

edges.append(edge_res)
edge_res.head()<jupyter_output><empty_output><jupyter_text>Compound TREATS Disease<jupyter_code>q = """SELECT DISTINCT ?compound ?compoundLabel ?disease ?diseaseLabel
  WHERE {

    # Initial typing for Compound
    ?compound wdt:P31 wd:Q11173 .

    # Initial typing for Disease 
    # Either instance of Disease of has a Disease Ontology ID
    {?disease wdt:P31 wd:Q12136}UNION{?disease wdt:P699 ?doid}.

    {?compound wdt:P2175 ?disease}UNION{?disease wdt:P2176 ?compound}
    SERVICE wikibase:label { bd:serviceParam wikibase:language "[AUTO_LANGAGE],en" }

  }"""
qr = execute_sparql_query(q)
edge_res = standardize_edges(qr, 'compound', 'disease', 'treats')

edges.append(edge_res)
edge_res.head()<jupyter_output><empty_output><jupyter_text>Compound IW Protein<jupyter_code>q = """SELECT DISTINCT ?compound ?compoundLabel ?qualifier ?qualifierLabel ?protein ?proteinLabel 
WHERE {

    # Initial typing for Compound
    ?compound wdt:P31 wd:Q11173 .

    # Initial typing for Protein
    ?protein wdt:P31 wd:Q8054 .

    { ?compound p:P129 [ps:P129 ?protein;
                  pq:P2868 ?qualifier] }
    UNION { ?protein p:P129 [ps:P129 ?compound;
                 pq:P366 ?qualifier] }

    SERVICE wikibase:label { bd:serviceParam wikibase:language "[AUTO_LANGAGE],en" }
}"""
qr = execute_sparql_query(q)
edge_res = standardize_edges(qr, 'compound', 'protein', 'qualifierLabel')

edges.append(edge_res)
edge_res.head()<jupyter_output><empty_output><jupyter_text>Anatomy SITE OF Disease<jupyter_code>q = """SELECT DISTINCT ?disease ?diseaseLabel ?anatomy ?anatomyLabel 
WHERE {

    # Initial typing for Disease
    {?disease wdt:P31 wd:Q12136}UNION{?disease wdt:P699 ?doid}.
    {?disease1 wdt:P31 wd:Q12136}UNION{?disease1 wdt:P699 ?doid1}.

    # Initial typing for Anatomical Structure
    ?anatomy wdt:P1554 ?uberon .

    {?disease1 wdt:P927 ?anatomy} {?disease wdt:P279? ?disease1}

    SERVICE wikibase:label { bd:serviceParam wikibase:language "[AUTO_LANGAGE],en" }
}"""
qr = execute_sparql_query(q)
edge_res = standardize_edges(qr, 'anatomy', 'disease', 'site_of')

edges.append(edge_res)
edge_res.head()<jupyter_output><empty_output><jupyter_text>Gene AW Disease<jupyter_code>q = """SELECT DISTINCT ?disease ?diseaseLabel ?gene ?geneLabel 
WHERE {

    # Initial typing for Disease
    {?disease wdt:P31 wd:Q12136}UNION{?disease wdt:P699 ?doid}.
  
    # Initial typing for Gene
    ?gene wdt:P31 wd:Q7187 .

    ?disease wdt:P2293 ?gene

    SERVICE wikibase:label { bd:serviceParam wikibase:language "[AUTO_LANGAGE],en" }
}"""
qr = execute_sparql_query(q)
edge_res = standardize_edges(qr, 'gene', 'disease', 'associated_with')

edges.append(edge_res)
edge_res.head()<jupyter_output><empty_output><jupyter_text>Pathway HP Gene<jupyter_code>q = """SELECT DISTINCT ?pathway ?gene 
WHERE {

    # Initial typing for Pathway
    ?pathway wdt:P31 wd:Q4915012 .
    FILTER NOT EXISTS{?pathway wdt:P686 ?goid}
  
    # Initial typing for Gene
    ?gene wdt:P31 wd:Q7187 .

    ?pathway wdt:P527 ?gene
} """
qr = execute_sparql_query(q)
edge_res = standardize_edges(qr, 'pathway', 'gene', 'has_part')

edges.append(edge_res)
edge_res.head()<jupyter_output><empty_output><jupyter_text>Pathway Has Part Compound<jupyter_code>q = """SELECT DISTINCT ?pathway ?compound 
WHERE {

    # Initial typing for Pathway
    ?pathway wdt:P31 wd:Q4915012 .
    FILTER NOT EXISTS{?pathway wdt:P686 ?goid}
  
    # Initial typing for Compound
    ?compound wdt:P31 wd:Q11173 .

    ?pathway wdt:P527 ?compound
} """
qr = execute_sparql_query(q)
edge_res = standardize_edges(qr, 'pathway', 'compound', 'has_part')

edges.append(edge_res)
edge_res.head()<jupyter_output><empty_output><jupyter_text>Pathway Medical_condition Disease<jupyter_code>q = """SELECT DISTINCT ?pathway ?disease 
WHERE {

    # Initial typing for Pathway
    ?pathway wdt:P31 wd:Q4915012 .
    FILTER NOT EXISTS{?pathway wdt:P686 ?goid}
  
    # Initial typing for Disease
    {?disease wdt:P31 wd:Q12136 .}UNION{?disease wdt:P699 ?doid}

    {?pathway wdt:P1050 ?disease}
} """
qr = execute_sparql_query(q)
edge_res = standardize_edges(qr, 'pathway', 'disease', 'associated_with')

edges.append(edge_res)
edge_res.head()<jupyter_output><empty_output><jupyter_text>Pathway to BP<jupyter_code>q = """SELECT DISTINCT ?pathway ?bio_process 
WHERE {

    # Initial typing for Pathway
    ?pathway wdt:P31 wd:Q4915012 .
    FILTER NOT EXISTS{?pathway wdt:P686 ?goid}
  
    # Initial typing for Bio Process
    ?bio_process wdt:P31 wd:Q2996394 .
    ?bio_process wdt:P686 ?goid .

    {?pathway wdt:P31 ?bio_process}UNION{?bio_process wdt:P31 ?pathway}
} """
qr = execute_sparql_query(q)
edge_res = standardize_edges(qr, 'pathway', 'bio_process', 'involved_in')

edges.append(edge_res)
edge_res.head()<jupyter_output><empty_output><jupyter_text>Disease to Phenotype<jupyter_code>q = """  SELECT DISTINCT ?disease ?pheno
  WHERE {

      # Initial typing for Symptom
      {?disease wdt:P31 wd:Q12136}UNION{?disease wdt:P699 ?doid}
    
      # Initial typing for Phenotype
      {?pheno wdt:P31 wd:Q169872.}UNION{?pheno wdt:P3841 ?hpo}

      {?pheno wdt:P780 ?disease}UNION{?disease wdt:P780 ?pheno}

  }"""
qr = execute_sparql_query(q)
edge_res = standardize_edges(qr, 'disease', 'pheno', 'presents')

edges.append(edge_res)
edge_res.head()<jupyter_output><empty_output><jupyter_text>Compound TREATS Phenotype<jupyter_code>q = """SELECT DISTINCT ?compound ?pheno
  WHERE {

      # Initial typing for Compound
      ?compound wdt:P31 wd:Q11173 .
    
      # Initial typing for Phenotype
      {?pheno wdt:P31 wd:Q169872.}UNION{?pheno wdt:P3841 ?hpo}

      {?pheno wdt:P2176 ?compound}UNION{?compound wdt:P2175 ?pheno}
  } """
qr = execute_sparql_query(q)
edge_res = standardize_edges(qr, 'compound', 'pheno', 'treats')

edges.append(edge_res)
edge_res.head()<jupyter_output><empty_output><jupyter_text>Subclass Edges for later Punning  Disease is a Subclass of disease<jupyter_code>q = """SELECT DISTINCT ?disease ?diseaseLabel ?disease1 ?disease1Label
WHERE {

    # Initial typing for Disease
    {?disease wdt:P31 wd:Q12136}UNION{?disease wdt:P699 ?doid}.
    {?disease1 wdt:P31 wd:Q12136}UNION{?disease1 wdt:P699 ?doid1}.

    ?disease wdt:P279? ?disease1

    SERVICE wikibase:label { bd:serviceParam wikibase:language "[AUTO_LANGAGE],en" }
}"""
qr = execute_sparql_query(q)
edge_res = standardize_edges(qr, 'disease', 'disease1', 'subclass_of')

edges.append(edge_res)
edge_res.head()<jupyter_output><empty_output><jupyter_text>Anatomy subclass of Anatomy<jupyter_code>q = """SELECT DISTINCT ?anatomy ?anatomyLabel ?anatomy1 ?anatomy1Label
WHERE {


    # Initial typing for Anatomical Structure
    ?anatomy wdt:P1554 ?uberon .
    ?anatomy1 wdt:P1554 ?uberon1 .
    
    ?anatomy wdt:P279? ?anatomy1

    SERVICE wikibase:label { bd:serviceParam wikibase:language "[AUTO_LANGAGE],en" }
}"""
qr = execute_sparql_query(q)
edge_res = standardize_edges(qr, 'anatomy', 'anatomy1', 'subclass_of')

edges.append(edge_res)
edge_res.head()<jupyter_output><empty_output><jupyter_text>Combine new edge and node results<jupyter_code>nodes.append(prev_nodes)
nodes = pd.concat(nodes, sort=False, ignore_index=True)
edges = pd.concat(edges, sort=False, ignore_index=True)
len(nodes)
len(edges)
count_plot_h(nodes['label'])
count_plot_h(edges['type'].value_counts().head(10))
count_plot_h(edges['type'].value_counts().head(20).tail(10))<jupyter_output><empty_output><jupyter_text>Save the new info out<jupyter_code>out_dir = Path('../results/')

out_dir.mkdir(parents=True, exist_ok=True)

nodes.to_csv(out_dir.joinpath('01b_nodes.csv'), index=False)
edges.to_csv(out_dir.joinpath('01b_edges.csv'), index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.126970509383378, 'fraction_numerical': 0.03613941018766756, 'mean_word_length': 4.6110108303249095, 'pattern_counts': {'":': 0, '<': 117, '<?xml version=': 0, '>': 119, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '29', 'dir': 'jupyter-structured-clean-dedup', 'id': '199460', 'license': 'CC0-1.0', 'n_tokens_mistral': 7968, 'n_tokens_neox': 7092, 'n_words': 1409, 'path': 'old/src/metapaths/.ipynb_checkpoints/01b_WikiData_Edges-checkpoint.ipynb', 'repo_name': 'andrawaag/WRP', 'size': '18650'}