<jupyter_start><jupyter_text>KEGGutils Tutorial 0: KEGGutils basics and KEGG REST API commands Setup and how to make basic requests to KEGG API using intuitive function calls from within your code***   Getting Started Installing KEGGutils`KEGGutils` is available as a **PyPi** package  and can be easily installed using `pip` via `pip install KEGGutils`,this should be enough to get you going. Alternatively you can visit the project's **Github** https://github.com/filippocastelli/KEGGutils and the **PyPi** page https://pypi.org/project/KEGGutils/ for a manual install.  First things firstLet's import `KEGGutils` with an alias<jupyter_code>import os,sys,inspect
currentdir = os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe())))
parentdir = os.path.dirname(currentdir)
sys.path.insert(0,parentdir) 
import KEGGutils as kg<jupyter_output><empty_output><jupyter_text>check that the latest version of `KEGGutils` is installed<jupyter_code>print(kg.__version__)<jupyter_output>0.3.2<jupyter_text>we should be ready to go!we can set a directory for our downloaded files, or we can use the default one:<jupyter_code>print("Our default download directory is {}".format(kg.DOWNLOAD_DIR))<jupyter_output>Our default download directory is C:\Users\filippo.castelli4\Documents\GitHub\KEGGutils\tutorials\kegg_downloads<jupyter_text>please notice that the default directory is a runtime variable, remember to **change it every time `KEGGutils` is reloaded** (maybe we'll change that in the future).   to ensure that we use the latest available data we can remove all previously downloaded files, if it's your first time using `KEGGutils` you should have no problems.<jupyter_code>kg.delete_cached_files()<jupyter_output>> deleting the following files from C:\Users\filippo.castelli4\Documents\GitHub\KEGGutils\tutorials\kegg_downloads
C:\Users\filippo.castelli4\Documents\GitHub\KEGGutils\tutorials\kegg_downloads\hsa05215-description
C:\Users\filippo.castelli4\Documents\GitHub\KEGGutils\tutorials\kegg_downloads\hsa05215-kgml
C:\Users\filippo.castelli4\Documents\GitHub\KEGGutils\tutorials\kegg_downloads\path-hsa05215-image.png<jupyter_text>The interesting stuff: KEGG REST API KEGGutils provides, among its features, the ability to easily interrogate the KEGG  REST API from within your code without having to explicitly handle HTTP requests: this functionality comes in the form of specific methods for each of the KEGG API commands: + `keggapi_list()` for the `LIST` API command+ `keggapi_info()` for the `INFO` API command+ `keggapi_find()` for the `FIND` API command+ `keggapi_get()` for the `GET` API command+ `keggapi_conv()` for the `CONV` API command+ `keggapi_link()` for the `LINK` API command+ `keggapi_ddi()` for the `DDI` API commandfor further infos on what any of the API commands do you can visit [KEGG's API instructions page](https://www.kegg.jp/kegg/rest/keggapi.html)  KEGG API commands: INFO Let's start from the simplest of them all: the `INFO` instruction that we can easily use with the `keggapi_info()` method.<jupyter_code>print(kg.keggapi_info.__doc__)<jupyter_output>KEGG REST API interface for INFO command
    Displays information on a given database
    
    for further info read https://www.kegg.jp/kegg/rest/keggapi.html
    
    Parameters
    ----------
    database : str
        database of which you want to obtain infos on
    verbose : bool
        if set to False displays only the first 4 lines of text (default is True)
    force_download :  bool
        forces overwriting on previous cached files (default is False)
    retutn_format : str
        optional, specify a return format to return, str | dict (default is None)
        
    Returns
    -------
    info_str : str
        optional, plain text response of API INFO command
    info_dict : dict
        optional, parsed response of API INFO as a dictionary<jupyter_text>It basically displays informations on a given database release, the  databases we can get infos on have to be chosen from the following values:**kegg | pathway | brite | module | ko | genome | genes | vg | ag | ligand | compound | glycan | reaction | rclass | enzyme | network | variant | disease | drug | dgroup | environ** we can also request informations on a particular organism code database: let's try to ask infos on the *Homo Sapiens* gene database using the corresponding KEGG **hsa** organism code.<jupyter_code>kg.keggapi_info("hsa")<jupyter_output>INFO:root:Infos on hsa from KEGG:<jupyter_text>if we wish to have this kind of information packed as a dictionary we can use the  `return_format = "dict" ` option:<jupyter_code>hsa_info_dict = kg.keggapi_info("hsa", return_format = "dict")

hsa_info_dict["linked db"]<jupyter_output>INFO:root:Infos on hsa from KEGG:<jupyter_text>If we want to know what other organism databates are available KEGGutils has a builtin function for displaying a complete list of KEGG organism codes trivially called `get_organism_codes()`<jupyter_code>print(kg.get_organism_codes.__doc__)<jupyter_output>Returns all KEGG Organism name codes<jupyter_text>let's display the first ten organisms<jupyter_code>kg.get_organism_codes()[:10]<jupyter_output><empty_output><jupyter_text>KEGG API commands: LISTthe second command we're gonna see is `LIST`, we can access it using the `keggapi_list()` method.<jupyter_code>print(kg.keggapi_list.__doc__)<jupyter_output>Interface for the KEGG API LIST command

    See https://www.kegg.jp/kegg/rest/keggapi.html for usage

    Parameters
    ----------
    database : str
        Database you wish to obtain a list about, one from pathway | brite | module | ko | genome | <org> | vg | ag | compound | glycan | reaction | rclass | enzyme | network | variant | disease | drug | dgroup | environ | organism | <medicus>
    option : str, optional
        'xl' option is applicable only to the 'brite' database for listing binary relation files (the default is None)
    want_descriptions : bool, optional
        If True returns descriptions for each item (the default is False)
    force_download : bool, optional
        If true replaces any pre-existing downloaded file (the default is False)
    return_url : bool, optional
        If True returns the interrogated URL
    
    Returns
    -------
    item, descritpions
        lists for items and their descriptions<jupyter_text>We can use it to easily get list of all entries in a database, `keggapi_list()` lets you optionally choose wether you want to see all the entries with the respective description or you just need a straightforward list of entries with the `want_descriptions` option (not really masters of choosing option names here).<jupyter_code>hsa_list, descriptions = kg.keggapi_list(database = "hsa", want_descriptions = True)
list(zip(hsa_list, descriptions))[:10]<jupyter_output><empty_output><jupyter_text>we can easily create a dictionary for quick reference with each gene and it's description:<jupyter_code>genes_dictionary = dict(zip(hsa_list, descriptions))

genes_dictionary["hsa:574"]<jupyter_output><empty_output><jupyter_text>please note that everything you request from KEGG gets saved under`kegg_download_dir` in plain text for easy retrieval when you don't have internet connection or simply don't want to wait for another HTTP request.The next time we'll call `keggapi_list()` or any of the methods we're preenting in this tutorial `KEGGutils` will automatically read the same cached copy from disk:<jupyter_code>hsa_list = kg.keggapi_list(database = "hsa")<jupyter_output>WARNING:root:> File hsa-list already present in C:\Users\filippo.castelli4\Documents\GitHub\KEGGutils\tutorials\kegg_downloads
reading from chached file...<jupyter_text>unless we explicitly want a fresh copy, and we can request it using the `force_download` option:<jupyter_code>hsa_list = kg.keggapi_list(database = "hsa", force_download = True)<jupyter_output>INFO:root:> Downloading hsa-list from KEGG at http://rest.kegg.jp/list/hsa
INFO:root:succesfully downloaded hsa-list<jupyter_text>KEGG API commands: FINDThe `FIND` functionality is readily avilable using `keggapi_find()`This is one of the most interesting KEGG API funcionalities: it's a search operation that looks for a query in all of a given database entries an associated fields<jupyter_code>print(kg.keggapi_find.__doc__)<jupyter_output>Interface for the KEGG API FIND command
    
    See https://www.kegg.jp/kegg/rest/keggapi.html for further info
    
    Parameters
    ----------
    database : str
        KEGG database you wish to query on, one from pathway | brite | module | ko | genome | genes | <org> | vg | ag | ligand | compound | glycan | reaction | rclass | enzyme | network | variant | disease | drug | dgroup | environ | <medicus>  
    query : str
        Desired query
    option : str, optional
        if database is "compound" or "drug", possible options are "formula", "exact mass" and "mol_weight" (the default is None)
    want_descriptions : bool, optional
        if set to True returns a list of descriptions for the found items (the default is False,)
    verbose : bool, optional
        if set to True displays additional messages (the default is False)
    force_download : bool, optional
        if set to True replaces any previously downloaded file under the same name (the default is False)
    return[...]<jupyter_text>We may be interested on knowing which *Escherichia Coli* genes code for [*Shiga toxins*](https://en.wikipedia.org/wiki/Shiga_toxin), we can perform a search on all genes for the words *shiga toxin*<jupyter_code>gene_list, descriptions = kg.keggapi_find(database = "genes",
                                          query = "shiga toxin",
                                          want_descriptions = True)

list(zip(gene_list, descriptions))[:10]<jupyter_output><empty_output><jupyter_text>we can easily verify that the found gene codes are from various E. Coli subspecies using `keggapi_info` on the database identifier `ece`<jupyter_code>kg.keggapi_info("ece")<jupyter_output>INFO:root:Infos on ece from KEGG:<jupyter_text>Another useful example of the ways we can use the `FIND` funcionality is searching registered compounds matching the chemical formula of [*Shikimic Acid*](*https://en.wikipedia.org/wiki/Shikimic_acid) *C7H10O5*:<jupyter_code>compounds, formulas = kg.keggapi_find(database = "compound",
                                          query = "C7H10O5",
                                          option = "formula",
                                          want_descriptions = True)

list(zip(compounds, formulas))[:10]<jupyter_output><empty_output><jupyter_text>KEGG API commands: GET This is probably the most used command in the whole series, you can use it by calling `keggapi_get()`.I'ts use to retrieve any KEGG database entry in a variety of formats:<jupyter_code>print(kg.keggapi_get.__doc__)<jupyter_output>Interface for the KEGG API GET command

    for further info read https://www.kegg.jp/kegg/rest/keggapi.html
    
    Parameters
    ----------
    dbentry : str
        KEGG database entry you wish to GET, one from pathway | brite | module | ko | genome | <org> | vg | ag | compound | glycan | reaction | rclass | enzyme | network | variant | disease | drug | dgroup | environ | disease_ja | drug_ja | dgroup_ja | environ_ja | compound_ja
    option : str, optional
        one from aaseq | ntseq | mol | kcf | image | conf | kgml | json (the default is None])
    want_descriptions : bool, optional
        if True returns a list of descriptions for the requested items (the default is False)
    verbose : bool, optional
        is True displays additional messages  (the default is True)
    force_download : bool, optional
        if set to True replaces any file under the same filename (the default is False)
    show_result_image: boo, optional
        if set to True shows the downloaded ima[...]<jupyter_text>easiest case: we want to identify what entry `ecs:ECs1206` could possibly be, we can easy ask the KEGG database using the following command<jupyter_code>kg.keggapi_get(dbentry = "ecs:ECs1206")<jupyter_output>Infos on ecs:ECs1206 from KEGG:

DEFINITION  (RefSeq) Shiga toxin 2 subunit B
ORTHOLOGY   K11007  shiga toxin subunit B
ORGANISM    ecs  Escherichia coli O157:H7 Sakai (EHEC)<jupyter_text>if knowing definition, orthology and organism is not enough for us, we can use the `verbose` option that will tell us life, death and miracles about `ecs:ECs1206`:<jupyter_code>kg.keggapi_get(dbentry = "ecs:ECs1206", verbose = True)<jupyter_output>Infos on ecs:ECs1206 from KEGG:

ENTRY       ECs1206           CDS       T00048
DEFINITION  (RefSeq) Shiga toxin 2 subunit B
ORTHOLOGY   K11007  shiga toxin subunit B
ORGANISM    ecs  Escherichia coli O157:H7 Sakai (EHEC)
PATHWAY     ecs02024  Quorum sensing
            ecs05130  Pathogenic Escherichia coli infection
MODULE      ecs_M00363  EHEC pathogenicity signature, Shiga toxin
BRITE       KEGG Orthology (KO) [BR:ecs00001]
             09140 Cellular Processes
              09145 Cellular community - prokaryotes
               02024 Quorum sensing
                ECs1206
             09160 Human Diseases
              09171 Infectious diseases: Bacterial
               05130 Pathogenic Escherichia coli infection
                ECs1206
             09180 Brite Hierarchies
              09183 Protein families: signaling and cellular processes
               02042 Bacterial toxins [BR:ecs02042]
                ECs1206
            Bacterial toxins [BR:ecs02042]
             Type III t[...]<jupyter_text>the same data is available as a dict selecting the `return_dict` option<jupyter_code>kg.keggapi_get(dbentry = "ecs:ECs1206", return_dict = True)<jupyter_output><empty_output><jupyter_text>among the various perks of the `GET` functionality is the ability to easily request amminoacid sequences using the `aaseq` option<jupyter_code>description, sequence = kg.keggapi_get(dbentry = "ecs:ECs1206",
                                       option = "aaseq",
                                       want_descriptions = True)
print(sequence)<jupyter_output>MKKMFMAVLFALASVNAMAADCAKGKIEFSKYNEDDTFTVKVDGKEYWTSRWNLQPLLQSAQLTGMTVTIKSSTCESGSGFAEVQFNND<jupyter_text>or we can easily see what a given compound formula looks like using the `image option`<jupyter_code>img = kg.keggapi_get(dbentry = "C00002",
              option = "image")<jupyter_output>INFO:root:> Downloading C00002-image from KEGG at http://rest.kegg.jp/get/C00002/image<jupyter_text>graphical descriptions are also available for **pathways**<jupyter_code>e_coli_infection_pathway_img = kg.keggapi_get(dbentry = "hsa05130", option = "image")<jupyter_output>INFO:root:> Downloading hsa05130-image from KEGG at http://rest.kegg.jp/get/hsa05130/image<jupyter_text>The inline image preview is a bit small but you can always use the iPython magic `%matplotlib notebook` to use mainteractive visualization mode. But `GET` can offer us much more than just visualizing pathways, we can obtain (almost) full descriptions in the form of [KGML xml files](https://www.kegg.jp/kegg/xml/)<jupyter_code>kgml = kg.keggapi_get(dbentry = "hsa05130", option = "kgml")<jupyter_output><empty_output><jupyter_text>Once downloaded the `xml` file, `KEGGutils` automatically converts it to an `XML ElementTree` for easy manipulation and interrogation, here we just look for the title of the pathway, the KEGG corresponding description url and a link to a graphical representation.<jupyter_code>print("Pathway Title:")
print(kgml.getroot().get('title'))
print("Pathway link:")
print(kgml.getroot().get('link'))
print("Pathway image link:")
print(kgml.getroot().get('image'))<jupyter_output>Pathway Title:
Pathogenic Escherichia coli infection
Pathway link:
http://www.kegg.jp/kegg-bin/show_pathway?hsa05130
Pathway image link:
http://www.kegg.jp/kegg/pathway/hsa/hsa05130.png<jupyter_text>**note** `KEGGutils` also features a full dedicated class for parsing, storing `.xml` `KGML-style` pathways and converting them to easily usable `networkx` Graph, that's called `KEGGpathway`and we'll see it in a different tutorial.  KEGG API commands: LINKOk, we have very nice lists of stuff and several ways to get information of each and every one of the list elements, but the really interesting part is getting to know how it's all tangled together, here comes to the resecue the `LINK` functionality: KEGG `LINK` functionality is used to view the interaction links between two different databases and it's accessible via `keggapi_link()`.<jupyter_code>print(kg.keggapi_link.__doc__)<jupyter_output>Interface for the KEGG REST API LINK command 
    Given two different database names returns the linked relations between them
    
    for further info read https://www.kegg.jp/kegg/rest/keggapi.html
    
    Parameters
    ----------
    source : str
        source database name
    target : str
        target database name
    verbose : bool, optional
        displays additional infos during the download (the default is True)
    force_download : bool, optional
        forces overwriting over cached files (the default is False)
    return_url : bool, optional
        If True returns the interrogated URL
        
    Raises
    ------
    KEGGKeyError
        if a database key is invalid
    
    Returns
    -------
    link1 : list
        list of source nodes
    link2 : list
        list of target nodes<jupyter_text>Given two different databases `keggapi_link()` will just return every link that exists between their components<jupyter_code>genes, enzymes = kg.keggapi_link("hsa", "enzyme" )<jupyter_output>INFO:root:> Downloading enzyme-hsa-link from KEGG at http://rest.kegg.jp/link/enzyme/hsa
INFO:root:succesfully downloaded enzyme-hsa-link<jupyter_text>in the form of two lists that we can easily zip together to create another list, a dict, or whatever we want to create<jupyter_code>list(zip(genes, enzymes))[:10]<jupyter_output><empty_output><jupyter_text>KEGG API commands: CONVIn an integrated workflow we may want to use different databases other thank just KEGG, how can we make them talk the same language? We use the `CONV` functionality that `KEGGutils` exposes via `keggapi_conv()`.<jupyter_code>print(kg.keggapi_conv.__doc__)<jupyter_output>KEGG REST API interface to CONV command
    Converts KEGG codes to and from NCBI ProteinID, NCBI GeneID, Uniprot, CHEBI    and PubChem name standards
    
    for further info read https://www.kegg.jp/kegg/rest/keggapi.html
    
    Parameters
    ----------
    
    source str :
        source database or dbentry
    target str :
        target database or dbentry
    verbose bool : 
        if set to True displays additional messages (defaultis True)
    force_download bool: 
        forces overwriting cached files (default is False)
    return_url : bool, optional
        If True returns the interrogated URL        
    Returns
    -------
    
    source_codes : list
        list of codes in the original database format
    target_codes : list
        list of codes in the target database format<jupyter_text>The method basically lets you convert to and from external link identifiers single entries or even entire databases: Here we want to get a correspondence between the KEGG style `eco` E. Coli genes and their names in the [NCBI GeneID format](https://www.ncbi.nlm.nih.gov/gene)<jupyter_code>ncbi_genes, kegg_genes_eco = kg.keggapi_conv("ncbi-geneid", "eco")<jupyter_output>INFO:root:> Downloading eco-ncbi-geneid-conv from KEGG at http://rest.kegg.jp/conv/eco/ncbi-geneid
INFO:root:succesfully downloaded eco-ncbi-geneid-conv<jupyter_text>in one line we've retrieved the full correspondences between the two formats<jupyter_code>list(zip(ncbi_genes, kegg_genes_eco))[:10]<jupyter_output><empty_output><jupyter_text>If we don't need a full database we can just request for a single entry conversion<jupyter_code>ncbi, kegg_eco = kg.keggapi_conv("ncbi-geneid:19831932", "genes")
print("{} NCBI gene is called {} in KEGG ".format(ncbi[0], kegg_eco[0]))<jupyter_output>ncbi-geneid:19831932 NCBI gene is called lpn:lpg0366 in KEGG<jupyter_text>KEGG API commands: DDIThe last of the KEGG API commands is called `DDI` that stands for *Drug-Drug Interaction* and searches for adverse drug-drug interaction for a given drug identifier, we can use it with  the `keggapi_ddi()` method.<jupyter_code>print(kg.keggapi_ddi.__doc__)
ddi_list = kg.keggapi_ddi("D00564")
ddi_list[:10]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07963478345783898, 'fraction_numerical': 0.013378253014989502, 'mean_word_length': 4.102640757349278, 'pattern_counts': {'":': 0, '<': 122, '<?xml version=': 0, '>': 129, 'https://': 13, 'lorem ipsum': 0, 'www.': 11, 'xml': 4}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 44, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '35', 'dir': 'jupyter-structured-clean-dedup', 'id': '525995', 'license': 'Unlicense', 'n_tokens_mistral': 6543, 'n_tokens_neox': 5969, 'n_words': 2133, 'path': 'tutorials/Tutorial 0 - Basics and KEGG API.ipynb', 'repo_name': 'filippocastelli/KGutils', 'size': '20481'}