<jupyter_start><jupyter_text>Introduction: DDOT tutorial* __What is DDOT?__ The DDOT Python package provides many functions for assembling,analyzing, and visualizing ontologies.  The main functionalities areimplemented in an object-oriented manner by an "Ontology" class, which handles ontologies that are data-driven as well as thosethat are manually curated like the Gene Ontology.* __What is an ontology?__ An ontology is a hierarchical arrangement of two types of nodes: (1)genes at the leaves of the hierarchy and (2) terms at intermediatelevels of the hierarchy. The hierarchy can be thought of as directedacyclic graph (DAG), in which each node can have multiple children ormultiple parent nodes. DAGs are a generalization of trees(a.k.a. dendogram), where each node has at most one parent.* __What to do after reading this tutorial__ Check out a complete list of functions in the [Ontology class](http://ddot.readthedocs.io/en/latest/ontology.html) and a list of [utility functions](http://ddot.readthedocs.io/en/latest/utils.html) that may help you build more concise pipelines. Also check out [example Jupyter notebooks](https://github.com/michaelkyu/ddot/tree/master/examples) that contain pipelines for downloading and processing the Gene Ontology and for inferring data-driven gene ontologies of diseases<jupyter_code># Import Ontology class from DDOT package
import ddot
from ddot import Ontology

import numpy as np<jupyter_output><empty_output><jupyter_text>Creating an Ontology object* An object of the Ontology class can be created in several ways.* In this tutorial, we will construct and analyze the toy ontology shown below.  Create an ontology through the \_\_init\_\_ constructor<jupyter_code># Connections from child terms to parent terms
hierarchy = [('S3', 'S1'),
             ('S4', 'S1'),
             ('S5', 'S1'),
             ('S5', 'S2'),
             ('S6', 'S2'),
             ('S1', 'S0'),
             ('S2', 'S0')]

# Connections from genes to terms
mapping = [('A', 'S3'),
           ('B', 'S3'),
           ('C', 'S3'),
           ('C', 'S4'),
           ('D', 'S4'),
           ('E', 'S5'),
           ('F', 'S5'),
           ('G', 'S6'),
           ('H', 'S6')]

# Construct ontology
ont = Ontology(hierarchy, mapping)

# Prints a summary of the ontology's structure
print(ont)<jupyter_output>8 genes, 7 terms, 9 gene-term relations, 7 term-term relations
node_attributes: []
edge_attributes: []<jupyter_text>Create an ontology from a tab-delimited table or Pandas dataframe<jupyter_code># Write ontology to a tab-delimited table
ont.to_table('toy_ontology.txt')
# Reconstruct the ontology from the table
ont2 = Ontology.from_table('toy_ontology.txt')
ont2<jupyter_output><empty_output><jupyter_text>From the Network Data Exchange (NDEx).* It is strongly recommended that you create a free account on NDEx in order to keep track of your own ontologies.* Note that there are two NDEx servers: the main server at http://public.ndexbio.org/ and a test server for prototyping your code at http://test.ndexbio.org (also aliased as http://dev2.ndexbio.org). Each server requires a separate user account. Because the main server contains networks from publications, we recommend that you use an account on the test server while you become familiar with DDOT<jupyter_code># Note: change the server to http://public.ndexbio.org, if this is where you created your NDEx account
ndex_server = 'http://test.ndexbio.org' 

# Set the NDEx server and the user account (replace with your own account)
ndex_user, ndex_pass = '<enter your username>', '<enter your account password>'
# Upload ontology to NDEx. The string after "v2/network/" is a unique identifier, which is called the UUID, of the ontology in NDEx
url, _ = ont.to_ndex(ndex_server=ndex_server, ndex_user=ndex_user, ndex_pass=ndex_pass)
print(url)
# Download the ontology from NDEx
ont2 = Ontology.from_ndex(url)
print(ont2)<jupyter_output>8 genes, 7 terms, 9 gene-term relations, 7 term-term relations
node_attributes: ['Vis:Border Paint', 'Vis:Shape', 'name', 'Vis:Fill Color']
edge_attributes: ['Vis:Visible']<jupyter_text>Inspecting the structure of an ontology An Ontology object contains seven attributes:* ``genes`` : List of gene names* ``terms`` : List of term names* ``gene_2_term`` : dictionary mapping a gene name to a list of terms connected to that gene. Terms are represented as their 0-based index in ``terms``.* ``term_2_gene`` : dictionary mapping a term name to a list or genes connected to that term. Genes are represented as their 0-based index in ``genes``.* ``child_2_parent`` : dictionary mapping a child term to its parent terms.* ``parent_2_child`` : dictionary mapping a parent term to its children terms.* ``term_sizes`` : A list of each term's size, i.e. the number of unique genes contained within this term and its descendants. The order of this list is the same as ``terms``. For every ``i``, it holds that ``term_sizes[i] = len(self.term_2_gene[self.terms[i]])``<jupyter_code>ont.genes
ont.terms
ont.gene_2_term
ont.term_2_gene
ont.child_2_parent
ont.parent_2_child<jupyter_output><empty_output><jupyter_text>Alternatively, the hierarchical connections can be viewed as a binary matrix, using `Ontology.connected()`<jupyter_code>conn = ont.connected()
np.array(conn, dtype=np.int32)<jupyter_output><empty_output><jupyter_text>A summary of an Ontology’s object, i.e. the number of genes, terms, and connections, can be printed `print(ont)`<jupyter_code>print(ont)<jupyter_output>8 genes, 7 terms, 9 gene-term relations, 7 term-term relations
node_attributes: []
edge_attributes: []<jupyter_text>Manipulating the structure of an ontology DDOT provides several convenience functions for processing Ontologies into a desirable structure. Currently, there are no functions for adding genes and terms. If this is needed, then we recommend creating a new Ontology or manipulating the contents in a different library, such as NetworkX or igraph, and transforming the results into Ontology.  Renaming nodes<jupyter_code># Renaming genes and terms.
ont2 = ont.rename(genes={'A' : 'A_alias'}, terms={'S0':'S0_alias'})
ont2.to_table()<jupyter_output><empty_output><jupyter_text>Delete S1 and G while preserving transitive connections<jupyter_code>ont2 = ont.delete(to_delete=['S1', 'G'])
print(ont2)<jupyter_output>7 genes, 6 terms, 8 gene-term relations, 6 term-term relations
node_attributes: []
edge_attributes: []<jupyter_text>Delete S1 and G (don't preserve transitive connections)<jupyter_code>ont2 = ont.delete(to_delete=['S1', 'G'], preserve_transitivity=False)
print(ont2)<jupyter_output>7 genes, 6 terms, 8 gene-term relations, 3 term-term relations
node_attributes: []
edge_attributes: []<jupyter_text>Propagate gene-term connections* Often times, it is convenient to explicitly include all transitive connections in the hierarchy. That is, if a hierarchy has edges A-->B and B-->C, then the hierarchy also has A-->C. This can be done by calling `Ontology.propagate(direction='forward')` function.* On the other hand, all transitive connections can be removed with `Ontology.propagate(direction='reverse')`. This is useful as a parsimonious set of connections.<jupyter_code># Include all transitive connections between genes and terms
ont2 = ont.propagate(direction='forward', gene_term=True, term_term=False)
print(ont2)

# Remove all transitive connections between genes and terms, retaining only a parsimonious set of connections
ont3 = ont2.propagate(direction='reverse', gene_term=True, term_term=False)
print(ont3)<jupyter_output>8 genes, 7 terms, 27 gene-term relations, 7 term-term relations
node_attributes: []
edge_attributes: []
8 genes, 7 terms, 9 gene-term relations, 7 term-term relations
node_attributes: []
edge_attributes: []<jupyter_text>Propagate term-term connections<jupyter_code># Include all transitive connections between terms
ont2 = ont.propagate(direction='forward', gene_term=False, term_term=True)
print(ont2)

# Remove all transitive connections between terms, retaining only a parsimonious set of connections
ont3 = ont2.propagate(direction='reverse', gene_term=False, term_term=True)
print(ont3)<jupyter_output>8 genes, 7 terms, 9 gene-term relations, 11 term-term relations
node_attributes: []
edge_attributes: []
8 genes, 7 terms, 9 gene-term relations, 7 term-term relations
node_attributes: []
edge_attributes: []<jupyter_text>Take the subbranch consisting of all term and genes under S1<jupyter_code>ont2 = ont.focus(branches=['S1'])
print(ont2)<jupyter_output>Genes and Terms to keep: 10
6 genes, 4 terms, 7 gene-term relations, 3 term-term relations
node_attributes: []
edge_attributes: []<jupyter_text>Inferring a data-driven ontology* Given a set of genes and a gene similarity network, we can hierarchically cluster the genes to infer cellular subsystems using the CliXO algorithm. The resulting hierarchy of subsystems defines a "data-driven gene ontology". For more information about the CLIXO algorithm, see Kramer et al. Bioinformatics, 30(12), pp.i34-i42. 2014.* Conversely, we can also "flatten" the ontology structure to infer a gene-by-gene similarity network. In particular, the similarity between two genes is calculated as the size of the smallest common subsystem, known as "Resnik semantic similarity".* The CLIXO algorithm has been designed to reconstruct the original hierarchy from the Resnik score.<jupyter_code># Flatten ontology to gene-by-gene network
sim, genes = ont.flatten()
print('Similarity matrix')
print(np.round(sim, 2))
print('Row/column names of similarity matrix')
print(genes)
# Reconstruct the ontology using the CLIXO algorithm.
# In general, you may feed any kind of gene-gene similarities,
# e.g. measurements of protein-protein interactions, gene co-expression, or genetic interactions.

ont2 = Ontology.run_clixo(sim, alpha=0.0, beta=1.0, square=True, square_names=genes)
print(ont2)
ont2.to_table(edge_attr=True)<jupyter_output><empty_output><jupyter_text>Ontology alignment* The structures of two ontologies can be compared through a procedure known as ontology alignment. Ontology.align() implements the ontology alignment described in (Dutkowski et al. Nature biotechnology, 31(1), 2013), in which terms are matched if they contain similar sets of genes and if their parents and children terms are also similar.* Ontology alignment is particularly useful for annotating a data-driven gene ontology by aligning it to a curated ontology such as the Gene Ontology (GO). For instance, if a data-driven term is identified to have a similar set of genes as the GO term for DNA repair, then the data-driven subsystem can be annotated as being involved in DNA repair. Moreover, data-driven terms with no matches in the ontology alignment may represent new molecular mechanisms.<jupyter_code>## Make a second ontology (the ontology to the right in the above diagram)

# Connections from child terms to parent terms
hierarchy = [('T3', 'T1'),
             ('T4', 'T1'),
             ('T1', 'T0'),
             ('T5', 'T0')]

# Connections from genes to terms
mapping = [('A', 'T3'),
           ('B', 'T3'),
           ('C', 'T3'),
           ('D', 'T4'),
           ('E', 'T4'),
           ('F', 'T4'),
           ('G', 'T5'),
           ('H', 'T5')]

# Construct ontology
ont_B = Ontology(hierarchy, mapping)
ont.align(ont_B)<jupyter_output>collapse command: /cellar/users/mikeyu/anaconda2/envs/ddot_py36/lib/python3.6/site-packages/ddot/alignOntology/collapseRedundantNodes /tmp/tmpwp1dge56
collapse command: /cellar/users/mikeyu/anaconda2/envs/ddot_py36/lib/python3.6/site-packages/ddot/alignOntology/collapseRedundantNodes /tmp/tmp1lc3e9yo
Alignment command: /cellar/users/mikeyu/anaconda2/envs/ddot_py36/lib/python3.6/site-packages/ddot/alignOntology/calculateFDRs /tmp/tmp9eqtgf9r /tmp/tmpwsvypmdo 0.05 criss_cross /tmp/tmp39uq4flt 100 40 gene<jupyter_text>Construct ontotypes* A major goal of genetics is to understand how genotype translates to phenotype. An ontology represents biological structure through which this genotype-phenotype translation happens. * Given a set of mutations comprising a genotype, DDOT allows you to propagate the impact of these mutations to the subsystems containing these genes in the ontology. In particular, the impact on a subsystem is estimated by the number of its genes that have been mutated. These subsystem activities, which we have called an “ontotype”, enables more accurate and interpretable predictions of phenotype from genotype (Yu et al. Cell Systems 2016, 2(2), pp.77-88. 2016).<jupyter_code># Genotypes can be represented as tuples of mutated genes
genotypes = [('A', 'B'),
             ('A', 'E'),
             ('A', 'H'),
             ('B', 'E'),
             ('B', 'H'),
             ('C', 'F'),
             ('D', 'E'),
             ('D', 'H'),
             ('E', 'H'),
             ('G', 'H')]

# Calculate the ontotypes, represented a genotype-by-term matrix. Each value represents the functional impact on a term in a genotype.
ontotypes = ont.get_ontotype(genotypes)
print(ontotypes)
# Genotypes can also be represented a genotype-by-gene matrix as an alternative input format
import pandas as pd, numpy as np
genotypes_df = pd.DataFrame(np.zeros((len(genotypes), len(ont.genes)), np.float64),
                            index=['Genotype%s' % i for i in range(len(genotypes))],
                            columns=ont.genes)
for i, (g1, g2) in enumerate(genotypes):
    genotypes_df.loc['Genotype%s' % i, g1] = 1.0
    genotypes_df.loc['Genotype%s' % i, g2] = 1.0
print('Genotype matrix')
print(genotypes_df)
print("")

ontotypes = ont.get_ontotype(genotypes_df, input_format='matrix')
print('Ontotype matrix')
print(ontotypes)<jupyter_output>Genotype matrix
             A    B    C    D    E    F    G    H
Genotype0  1.0  1.0  0.0  0.0  0.0  0.0  0.0  0.0
Genotype1  1.0  0.0  0.0  0.0  1.0  0.0  0.0  0.0
Genotype2  1.0  0.0  0.0  0.0  0.0  0.0  0.0  1.0
Genotype3  0.0  1.0  0.0  0.0  1.0  0.0  0.0  0.0
Genotype4  0.0  1.0  0.0  0.0  0.0  0.0  0.0  1.0
Genotype5  0.0  0.0  1.0  0.0  0.0  1.0  0.0  0.0
Genotype6  0.0  0.0  0.0  1.0  1.0  0.0  0.0  0.0
Genotype7  0.0  0.0  0.0  1.0  0.0  0.0  0.0  1.0
Genotype8  0.0  0.0  0.0  0.0  1.0  0.0  0.0  1.0
Genotype9  0.0  0.0  0.0  0.0  0.0  0.0  1.0  1.0

Ontotype matrix
            S0   S1   S2   S3   S4   S5   S6
Genotype0  0.0  0.0  0.0  2.0  0.0  0.0  0.0
Genotype1  0.0  0.0  0.0  1.0  0.0  1.0  0.0
Genotype2  0.0  0.0  0.0  1.0  0.0  0.0  1.0
Genotype3  0.0  0.0  0.0  1.0  0.0  1.0  0.0
Genotype4  0.0  0.0  0.0  1.0  0.0  0.0  1.0
Genotype5  0.0  0.0  0.0  1.0  1.0  1.0  0.0
Genotype6  0.0  0.0  0.0  0.0  1.0  1.0  0.0
Genotype7  0.0  0.0  0.0  0.0  1.0  0.0  1.0
Genotype8  0[...]<jupyter_text>Conversions to NetworkX and igraph<jupyter_code># Convert to an igraph object
G = ont.to_igraph()
print(G)
# Reconstruct the Ontology object from the igraph object
Ontology.from_igraph(G)
# Convert to a NetworkX object
G = ont.to_networkx()
print(G.nodes())
print(G.edges())
# Reconstruct the Ontology object from the NetworkX object
tmp = Ontology.from_networkx(G)
print(tmp)<jupyter_output>8 genes, 7 terms, 9 gene-term relations, 7 term-term relations
node_attributes: []
edge_attributes: []<jupyter_text>Ontology visualization using HiView (http://hiview.ucsd.edu)* HiView is a web application for general visualization of the hierarchical structure in ontologies.* To use HiView, you must first upload your ontology into NDEx using the [Ontology.to_ndex()](http://ddot.readthedocs.io/en/latest/ontology.htmlddot.Ontology.to_ndex) function, and then input the NDEx URL for the ontology to HiView* In contrast to almost all other hierarchical visualization tools, which are limited to simple tree structures, HiView also supports more complicated hierarchies in the form of directed acyclic graphs, in which nodes may have multiple parents.  A simple upload to NDEx and visualization in HiView* Upload ontologies to NDEx using the `Ontology.to_ndex()` function.* Setting the parameter `layout="bubble"` (default value) will identify a spanning tree of the DAG and then lay this tree in a space-compact manner. When viewing in HiView, only the edges in the spanning tree are shown, while the other edges can be chosen to be shown.<jupyter_code>url, _ = ont.to_ndex(ndex_server=ndex_server,
                     ndex_user=ndex_user,
                     ndex_pass=ndex_pass,
                     layout='bubble')

print('To visualize in HiView, go to http://hiview.ucsd.edu in your web browser, and then')
print('\t--Enter this into the "NDEx Sever URL" field: %s' % ddot.parse_ndex_server(url))
print('\t--Enter this into the "UUID of the main hierarchy" field: %s' % ddot.parse_ndex_uuid(url))
print('Alternatively, go to %s' % ddot.to_hiview_url(url))<jupyter_output>To visualize in HiView, go to http://hiview.ucsd.edu in your web browser, and then
	--Enter this into the "NDEx Sever URL" field: http://dev2.ndexbio.org/
	--Enter this into the "UUID of the main hierarchy" field: 29c16a02-fa71-11e8-ad43-0660b7976219
Alternatively, go to http://hiview.ucsd.edu/29c16a02-fa71-11e8-ad43-0660b7976219?type=test&server=http://dev2.ndexbio.org<jupyter_text>An alternative layout by duplicating nodes* Setting the parameter `layout="bubble-collect"` will convert the DAG into a tree by duplicating nodes.* This transformation enables the ontology structure to be visualized without edges crossing.<jupyter_code>url, _ = ont.to_ndex(ndex_server=ndex_server,
                     ndex_user=ndex_user,
                     ndex_pass=ndex_pass,
                     layout='bubble-collect')

print('To visualize in HiView, go to http://hiview.ucsd.edu in your web browser, and then')
print('\t--Enter this into the "NDEx Sever URL" field: %s' % ddot.parse_ndex_server(url))
print('\t--Enter this into the "UUID of the main hierarchy" field: %s' % ddot.parse_ndex_uuid(url))
print('Alternatively, go to %s' % ddot.to_hiview_url(url))<jupyter_output>To visualize in HiView, go to http://hiview.ucsd.edu in your web browser, and then
	--Enter this into the "NDEx Sever URL" field: http://dev2.ndexbio.org/
	--Enter this into the "UUID of the main hierarchy" field: 29ec98b4-fa71-11e8-ad43-0660b7976219
Alternatively, go to http://hiview.ucsd.edu/29ec98b4-fa71-11e8-ad43-0660b7976219?type=test&server=http://dev2.ndexbio.org<jupyter_text>Visualizing metadata by modifying node labels, colors, and sizes* An Ontology object has a `node_attr` field that is a pandas DataFrame. The rows of the dataframe are genes or terms, and the columns are node attributes.* HiView understands special node attributes to control the node labels, colors, and sizes.<jupyter_code># Set the node labels (default is the gene and term names, as found in Ontology.genes and Ontology.terms)
ont.node_attr.loc['S4', 'Label'] = 'S4 alias'
ont.node_attr.loc['S5', 'Label'] = 'S5 alias'

# Set the fill color of nodes
ont.node_attr.loc['C', 'Vis:Fill Color'] = '#7fc97f'
ont.node_attr.loc['S1', 'Vis:Fill Color'] = '#beaed4'
ont.node_attr.loc['S0', 'Vis:Fill Color'] = '#fdc086'

# Set the node sizes (if not set, the default is the term size, as found in Ontology.term_sizes)
ont.node_attr.loc['C', 'Size'] = 10

ont.node_attr
url, _ = ont.to_ndex(ndex_server=ndex_server,
                     ndex_user=ndex_user,
                     ndex_pass=ndex_pass,
                     layout='bubble-collect')

print('To visualize in HiView, go to http://hiview.ucsd.edu in your web browser, and then')
print('\t--Enter this into the "NDEx Sever URL" field: %s' % ddot.parse_ndex_server(url))
print('\t--Enter this into the "UUID of the main hierarchy" field: %s' % ddot.parse_ndex_uuid(url))
print('Alternatively, go to %s' % ddot.to_hiview_url(url))
# Clear node attributes (optional)
ont.clear_node_attr()
ont.node_attr<jupyter_output><empty_output><jupyter_text>Visualize gene-gene interaction networks alongside the ontology* Every term in an ontology represents a biological function shared among the term's genes. Based on this intuition, those genes should be interacting in different ways, e.g. protein-protein interactions, RNA expression, or genetic interactions.* Gene-gene interaction networks can be uploaded with the ontology to NDEx, so that they can be visualized at the same time in HiView<jupyter_code># Calculate a gene-by-gene similarity matrix using the Resnik semantic similarity definition (see section "Inferring a data-driven ontology")
sim, genes = ont.flatten()
print(genes)
print(np.round(sim, 2))
# Convert the gene-by-gene similarity matrix into a dataframe with a "long" format, where rows represent gene pairs. This conversion can be easily done with ddot.melt_square()
import pandas as pd
sim_df = pd.DataFrame(sim, index=genes, columns=genes)
sim_long = ddot.melt_square(sim_df)
sim_long.head()
# Create other gene-gene interactions.
# For example, these can represent protein-protein interactions or gene co-expression. 
# Here, we simulate types of interactions by adding a random noise to the Resnik similarity
sim_long['example_interaction_type1'] = sim_long['similarity'] + np.random.random(sim_long.shape[0]) / 2.
sim_long['example_interaction_type2'] = sim_long['similarity'] + np.random.random(sim_long.shape[0]) / 2.
sim_long.head()
# Include the above gene-gene interactions by setting the `network` and `main_feature` parameters.
url, _ = ont.to_ndex(name="Toy Ontology",
                     ndex_server=ndex_server,
                     ndex_user=ndex_user,
                     ndex_pass=ndex_pass,
                     network=sim_long,
                     main_feature='similarity',
                     layout='bubble-collect')

print('To visualize in HiView, go to http://hiview.ucsd.edu in your web browser, and then')
print('\t--Enter this into the "NDEx Sever URL" field: %s' % ddot.parse_ndex_server(url))
print('\t--Enter this into the "UUID of the main hierarchy" field: %s' % ddot.parse_ndex_uuid(url))
print('Alternatively, go to %s' % ddot.to_hiview_url(url))<jupyter_output>To visualize in HiView, go to http://hiview.ucsd.edu in your web browser, and then
	--Enter this into the "NDEx Sever URL" field: http://dev2.ndexbio.org/
	--Enter this into the "UUID of the main hierarchy" field: 2b0a3dc6-fa71-11e8-ad43-0660b7976219
Alternatively, go to http://hiview.ucsd.edu/2b0a3dc6-fa71-11e8-ad43-0660b7976219?type=test&server=http://dev2.ndexbio.org
{'fraction_non_alphanumeric': 0.10191594837211379, 'fraction_numerical': 0.030547988030905273, 'mean_word_length': 4.198978407244021, 'pattern_counts': {'":': 0, '<': 73, '<?xml version=': 0, '>': 76, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '21', 'dir': 'jupyter-structured-clean-dedup', 'id': '129124', 'license': 'MIT', 'n_tokens_mistral': 7493, 'n_tokens_neox': 7049, 'n_words': 2528, 'path': 'examples/1.0.0/Tutorial.ipynb', 'repo_name': 'antonkratz/ddot', 'size': '22391'}