<jupyter_start><jupyter_text>IntroductionAn ontology is a hierarchical arrangement of two types of nodes: (1)genes at the leaves of the hierarchy and (2) terms at intermediatelevels of the hierarchy. The hierarchy can be thought of as directedacyclic graph (DAG), in which each node can have multiple children ormultiple parent nodes. DAGs are a generalization of trees(a.k.a. dendogram), where each node has at most one parent.The DDOT Python library provides many functions for assembling,analyzing, and visualizing ontologies.  The main functionalities areimplemented in an object-oriented manner by an "Ontology" class.  Thisclass can handle both ontologies that are data-driven as well as thosethat are manually curated like the Gene Ontology.<jupyter_code># Import Ontology class from DDOT package
from ddot import Ontology<jupyter_output><empty_output><jupyter_text>Creating an Ontology objectAn object of the Ontology class can be created in several ways. To demonstratethis, we will build the following ontology  Through the \_\_init\_\_ constructor<jupyter_code># Connections from child terms to parent terms
hierarchy = [('S3', 'S1'),
             ('S4', 'S1'),
             ('S5', 'S1'),
             ('S5', 'S2'),
             ('S6', 'S2'),
             ('S1', 'S0'),
             ('S2', 'S0')]

# Connections from genes to terms
mapping = [('A', 'S3'),
           ('B', 'S3'),
           ('C', 'S3'),
           ('C', 'S4'),
           ('D', 'S4'),
           ('E', 'S5'),
           ('F', 'S5'),
           ('G', 'S6'),
           ('H', 'S6')]

# Construct ontology
ont = Ontology(hierarchy, mapping)<jupyter_output><empty_output><jupyter_text>To and from a tab-delimited table or Pandas dataframe<jupyter_code>ont.to_table('toy_ontology.txt')
ont = Ontology.from_table('toy_ontology.txt')<jupyter_output><empty_output><jupyter_text>From the Network Data Exchange (NDEx). Requires creating a free user account at http://ndexbio.org/<jupyter_code># Replace with your own NDEx user account
ndex_server, ndex_user, ndex_pass = 'http://test.ndexbio.org', 'scratch', 'scratch'
# ndex_user, ndex_pass = '<PASSWORD>', '<PASSWORD>'

url, _ = ont.to_ndex(ndex_server=ndex_server, ndex_user=ndex_user, ndex_pass=ndex_pass)
print(url)
ont2 = Ontology.from_ndex(url)
print(ont2)<jupyter_output>8 genes, 7 terms, 9 gene-term relations, 7 term-term relations
node_attributes: ['name', 'y_pos', 'Vis:Fill Color', 'Vis:Border Paint', 'x_pos', 'Label', 'Vis:Shape', 'NodeType', 'Size', 'Vis:Size', 'isRoot']
edge_attributes: ['EdgeType', 'Vis:Visible', 'Is_Tree_Edge', '2']<jupyter_text>Inspecting the structure of an ontology An Ontology object contains seven attributes:* ``genes`` : List of gene names* ``terms`` : List of term names* ``gene_2_term`` : dictionary mapping a gene name to a list of terms connected to that gene. Terms are represented as their 0-based index in ``terms``.* ``term_2_gene`` : dictionary mapping a term name to a list or genes connected to that term. Genes are represented as their 0-based index in ``genes``.* ``child_2_parent`` : dictionary mapping a child term to its parent terms.* ``parent_2_child`` : dictionary mapping a parent term to its children terms.* ``term_sizes`` : A list of each term's size, i.e. the number of unique genes contained within this term and its descendants. The order of this list is the same as ``terms``. For every ``i``, it holds that ``term_sizes[i] = len(self.term_2_gene[self.terms[i]])``<jupyter_code>ont.genes
ont.terms
ont.gene_2_term
ont.term_2_gene
ont.child_2_parent
ont.parent_2_child<jupyter_output><empty_output><jupyter_text>Alternatively, the hierarchical connections can be viewed as a binary matrix, using `Ontology.connected()`<jupyter_code>conn = ont.connected()
np.array(conn, dtype=np.int32)<jupyter_output><empty_output><jupyter_text>A summary of an Ontology’s object, i.e. the number of genes, terms, and connections, can be printed `print(ont)`<jupyter_code>print(ont)<jupyter_output>8 genes, 7 terms, 9 gene-term relations, 7 term-term relations
node_attributes: []
edge_attributes: [2]<jupyter_text>Manipulating the structure of an ontology DDOT provides several convenience functions for processing Ontologies into a desirable structure. Currently, there are no functions for adding genes and terms. If this is needed, then we recommend creating a new Ontology or manipulating the contents in a different library, such as NetworkX or igraph, and transforming the results into Ontology.<jupyter_code># Renaming genes and terms.
ont2 = ont.rename(genes={'A' : 'A_alias'}, terms={'S0':'S0_alias'})
ont2.to_table()<jupyter_output><empty_output><jupyter_text>Delete S1 and G while preserving transitive connections<jupyter_code>ont2 = ont.delete(to_delete=['S1', 'G'])
print(ont2)<jupyter_output>7 genes, 6 terms, 8 gene-term relations, 6 term-term relations
node_attributes: []
edge_attributes: [2]<jupyter_text>Delete S1 and G (don't preserve transitive connections)<jupyter_code>ont2 = ont.delete(to_delete=['S1', 'G'], preserve_transitivity=False)
print(ont2)<jupyter_output>7 genes, 6 terms, 8 gene-term relations, 3 term-term relations
node_attributes: []
edge_attributes: [2]<jupyter_text>Propagate gene-term connections<jupyter_code>ont2 = ont.propagate(direction='forward', gene_term=True, term_term=False)
print(ont2)

# Remove all transitive connections, and maintain only a parsimonious set of connections
ont3 = ont2.propagate(direction='reverse', gene_term=True, term_term=False)<jupyter_output>8 genes, 7 terms, 27 gene-term relations, 7 term-term relations
node_attributes: []
edge_attributes: [2]<jupyter_text>Propagate term-term connections<jupyter_code>ont2 = ont.propagate(direction='forward', gene_term=False, term_term=True)
print(ont2)

# Remove all transitive connections, and maintain only a parsimonious set of connections
ont3 = ont2.propagate(direction='reverse', gene_term=False, term_term=True)<jupyter_output>8 genes, 7 terms, 9 gene-term relations, 11 term-term relations
node_attributes: []
edge_attributes: [2]<jupyter_text>Take the subbranch consisting of all term and genes under S1<jupyter_code>ont2 = ont.focus(branches=['S1'])
print(ont2)<jupyter_output>Genes and Terms to keep: 10
6 genes, 4 terms, 7 gene-term relations, 3 term-term relations
node_attributes: ['Original_Size']
edge_attributes: [2]<jupyter_text>Inferring a data-driven ontologyAn ontology can also be inferred in a data-driven manner based on an input set of node-node similarities.<jupyter_code>sim, genes = ont.flatten()
print(genes)
print(sim)
ont2 = Ontology.run_clixo(sim, 0.0, 1.0, square=True, square_names=genes)
print(ont2)<jupyter_output>8 genes, 7 terms, 9 gene-term relations, 7 term-term relations
node_attributes: []
edge_attributes: ['CLIXO_score']<jupyter_text>Ontology alignment<jupyter_code>## Make a second ontology

# Connections from child terms to parent terms
hierarchy = [('T3', 'T1'),
             ('T4', 'T1'),
             ('T1', 'T0'),
             ('T5', 'T0')]

# Connections from genes to terms
mapping = [('A', 'T3'),
           ('B', 'T3'),
           ('C', 'T3'),
           ('D', 'T4'),
           ('E', 'T4'),
           ('F', 'T4'),
           ('G', 'T5'),
           ('H', 'T5')]

# Construct ontology
ont_B = Ontology(hierarchy, mapping)
ont.align(ont_B)<jupyter_output>collapse command: /cellar/users/mikeyu/DeepTranslate/ddot/ddot/alignOntology/collapseRedundantNodes /tmp/tmp69tzhltw
collapse command: /cellar/users/mikeyu/DeepTranslate/ddot/ddot/alignOntology/collapseRedundantNodes /tmp/tmpq7jbs_ag
Alignment command: /cellar/users/mikeyu/DeepTranslate/ddot/ddot/alignOntology/calculateFDRs /tmp/tmpdleaetmk /tmp/tmpwvbp55c8 0.05 criss_cross /tmp/tmp8bvabn36 100 40 gene<jupyter_text>Construct ontotypes<jupyter_code># Genotypes can be represented as tuples of mutated genes
genotypes = [('A', 'B'),
             ('A', 'E'),
             ('A', 'H'),
             ('B', 'E'),
             ('B', 'H'),
             ('C', 'F'),
             ('D', 'E'),
             ('D', 'H'),
             ('E', 'H'),
             ('G', 'H')]

ontotypes = ont.get_ontotype(genotypes)
print(ontotypes)
# Genotypes can also be represented a genotype-by-gene matrix
import pandas as pd, numpy as np
genotypes_df = pd.DataFrame(np.zeros((len(genotypes), len(ont.genes)), np.float64),
                            index=['Genotype%s' % i for i in range(len(genotypes))],
                            columns=ont.genes)
for i, (g1, g2) in enumerate(genotypes):
    genotypes_df.loc['Genotype%s' % i, g1] = 1.0
    genotypes_df.loc['Genotype%s' % i, g2] = 1.0
print(genotypes_df)

ontotypes = ont.get_ontotype(genotypes_df, input_format='matrix')
print(ontotypes)<jupyter_output>A    B    C    D    E    F    G    H
Genotype0  1.0  1.0  0.0  0.0  0.0  0.0  0.0  0.0
Genotype1  1.0  0.0  0.0  0.0  1.0  0.0  0.0  0.0
Genotype2  1.0  0.0  0.0  0.0  0.0  0.0  0.0  1.0
Genotype3  0.0  1.0  0.0  0.0  1.0  0.0  0.0  0.0
Genotype4  0.0  1.0  0.0  0.0  0.0  0.0  0.0  1.0
Genotype5  0.0  0.0  1.0  0.0  0.0  1.0  0.0  0.0
Genotype6  0.0  0.0  0.0  1.0  1.0  0.0  0.0  0.0
Genotype7  0.0  0.0  0.0  1.0  0.0  0.0  0.0  1.0
Genotype8  0.0  0.0  0.0  0.0  1.0  0.0  0.0  1.0
Genotype9  0.0  0.0  0.0  0.0  0.0  0.0  1.0  1.0
            S0   S1   S2   S3   S4   S5   S6
Genotype0  0.0  0.0  0.0  2.0  0.0  0.0  0.0
Genotype1  0.0  0.0  0.0  1.0  0.0  1.0  0.0
Genotype2  0.0  0.0  0.0  1.0  0.0  0.0  1.0
Genotype3  0.0  0.0  0.0  1.0  0.0  1.0  0.0
Genotype4  0.0  0.0  0.0  1.0  0.0  0.0  1.0
Genotype5  0.0  0.0  0.0  1.0  1.0  1.0  0.0
Genotype6  0.0  0.0  0.0  0.0  1.0  1.0  0.0
Genotype7  0.0  0.0  0.0  0.0  1.0  0.0  1.0
Genotype8  0.0  0.0  0.0  0.0  0.0  1.0  1.0
[...]<jupyter_text>Conversions to NetworkX and igraph<jupyter_code>G = ont.to_igraph()
print(G)
G = ont.to_networkx()
print(G.nodes())
print(G.edges())<jupyter_output>['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'S0', 'S1', 'S2', 'S3', 'S4', 'S5', 'S6']
[('A', 'S3'), ('B', 'S3'), ('C', 'S3'), ('C', 'S4'), ('D', 'S4'), ('E', 'S5'), ('F', 'S5'), ('G', 'S6'), ('H', 'S6'), ('S1', 'S0'), ('S2', 'S0'), ('S3', 'S1'), ('S4', 'S1'), ('S5', 'S1'), ('S5', 'S2'), ('S6', 'S2')]<jupyter_text>Visualization in HiView (http://hiview.ucsd.edu)<jupyter_code>url, _ = ont.to_ndex(ndex_server=ndex_server, ndex_user=ndex_user, ndex_pass=ndex_pass, layout='bubble-collect')

print('Enter this into the "NDEx Sever URL" field')
print(ndex_server)

print('Enter this into the "UUID of the main hierarchy" field at http://hiview.ucsd.edu:')
print(url.split('/')[-1])<jupyter_output>Enter this into the "NDEx Sever URL" field
http://test.ndexbio.org
Enter this into the "UUID of the main hierarchy" field at http://hiview.ucsd.edu:
23f542c5-3a0a-11e8-9da1-0660b7976219
{'fraction_non_alphanumeric': 0.14143110364772835, 'fraction_numerical': 0.04879186491277172, 'mean_word_length': 3.9355432780847144, 'pattern_counts': {'":': 0, '<': 63, '<?xml version=': 0, '>': 63, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '23006', 'license': 'MIT', 'n_tokens_mistral': 4113, 'n_tokens_neox': 3956, 'n_words': 1125, 'path': 'examples/Tutorial.ipynb', 'repo_name': 'pupster90/ddot2', 'size': '10717'}