<jupyter_start><jupyter_text>02 - Programming for `UniProt` (browser/notebook)  Table of Contents1. [Introduction](introduction)2. [Python imports](imports)3. [Running a remote `UniProt` query](uniprot)  1. [Connecting to `UniProt`](connect)  2. [Constructing a query](query)  3. [Perform the query](search)  4. [EXAMPLE: Putting it together](example01)4. [Advanced queries](advanced)  1. [`key:value` queries](keyvalue)  2. [Exercise 01](ex01)  3. [Combining queries](combine)  4. [Exercise 02](ex02)  4. [Processing query results](processing)  1. [Tabular](tabular)  2. [Excel](excel)    3. [FASTA sequence](fasta)  4. [`pandas` dataframe](pandas)  IntroductionThe `UniProt` browser interface is very powerful, but you will have noticed from the previous exercises that even the most complex queries can be converted into a single string that describes the search being made of the `UniProt` databases. Using the browser interface, this string is generated for you, and placed into the search field at the top of the `UniProt` webpage every time you run a search.It can be tedious and time-consuming to point-and-click your way through a large number of browser-based searches, but by using the UniProt webservice, the search strings you've already seen, and a Python module called bioservices, we can compose and run as many searches as we like using a small amount of code, and pull the results of those searches down to our local machines.This notebook presents examples of methods for using `UniProt` programmatically, via a *webservice*, and you will be controlling the searches using Python code in this notebook.There are a number of advantages to this approach:             It is easy to set up repeatable searches for many sequences, or collections of sequences         It is easy to read in the search results and conduct downstream analyses that add value to your search    Where it is not practical to submit a large number of simultaneous queries via a web form (because it is tiresome to point-and-click over and over again), this can be handled programmatically instead. You have the opportunity to change custom options to help refine your query, compared to the website interface. If you need to repeat a query, it can be trivial to apply the same settings every time, if you use a programmatic approach.  Python importsTo use the Python programming language to query `UniProt`, we have to *import* helpful *packages* (collections of Python code that perform specialised tasks.<jupyter_code># io is a standard library package that lets us manipulate data
import io

# Import Seaborn for graphics and plotting
import seaborn as sns

# Import bioservices module, to run remote UniProt queries
from bioservices import UniProt

# Import Pandas, so we can use dataframes
import pandas as pd<jupyter_output><empty_output><jupyter_text>Running a remote `UniProt` query    There are three key steps to running a remote UniProt query with bioservices:1. Make a link to the `UniProt` webservice2. Construct a query string3. Send the query to `UniProt`, and catch the result in a *variable*Once the search result is caught and contained in a *variable*, that *variable* can be processed in any way you like, written to a file, or ignored.  Connecting to `UniProt`To open a connection to `UniProt`, you make an *instance* of the `UniProt()` *class* from `bioservices`. This can be made to be persistent so that, once a single connection to the database is created, you can interact with it over and over again to make multiple queries.     To make a persistent instance, you can assign UniProt() to a variable:```pythonservice = UniProt()  it is good practice to have a meaningful variable name```  Constructing a query    UniProt allows for the construction of complex searches by combining fields. A full discussion is beyond the scope of this lesson, but you will have seen in the preceding notebook that the searches you constructed by pointing and clicking on the UniProt website were converted into text in the search field at the top. To describe the format briefly: there are a set of defined *keywords* (or *keys*) that indicate the specific type of data you want to search in (such as `host`, `annotation`, or sequence `length`), and these are combined with a particular *value* you want to search for (such as `mouse`, or `40674`) in a `key:value` pair, separated by a colon, such as `host:mouse` or `ec:3.2.1.23`.* `UniProt` query fields: [http://www.uniprot.org/help/query-fields](http://www.uniprot.org/help/query-fields)If you provide a string, instead of a `key:value` pair, `UniProt` will search in all *fields* for your search term.Programmatically, we construct the query as a *string*, e.g.```pythonquery = "Q9AJE3"   this query means we want to look in all fields for Q9AJE3```  Perform the queryTo send the query to `UniProt`, you will use the `.search()` *method* of your active *instance* of the `UniProt()` *class*.     If you have assigned your instance to the variable service (as above), then you can run the query string as a remote search with the line:```pythonresult = service.search(query)   Run a query and catch the output in result```In the line above, the output of the search (i.e. your result) is stored in a new variable (created when the search is complete) called `result`. It is good practice to make variable names short and descriptive - this makes your code easier to read.  EXAMPLE: Putting it togetherThe code in the cell below uses the example code above to create an instance of the `UniProt()` class, and uses it to submit a pre-stored query to the `UniProt` service, then catch the result in a variable called `result`. The `print()` statement then shows us what the result returned by the service looks like.<jupyter_code># Make a link to the UniProt webservice (UniProt())
service = UniProt()

# Build a query string ("Q9AJE3")
query = "Q9AJE3"

# Send the query to UniProt, and catch the search result in a variable (service.search())
result = service.search(query)

# Inspect the result
print(result)<jupyter_output><empty_output><jupyter_text>The `UniProt()` instance defined in the cell above is *persistent*, so you can reuse it to make another query, as in the cell below:<jupyter_code># Make a new query string "Q01844", and run a remote search at UniProt
new_query = "Q01844"
new_result = service.search(new_query)

# Inspect the result
print(new_result)<jupyter_output><empty_output><jupyter_text>Advanced queriesThe examples above built queries that were simple strings. They did not exploit the key:value search structure, or combine search terms. In this section, you will explore some queries that use the UniProt query fields, and combine them into powerful, filtering searches.  `key:value` queriesAs noted above (and at [http://www.uniprot.org/help/query-fields](http://www.uniprot.org/help/query-fields)) particular values of specific data can be requested by using `key:value` pairs to restrict searches to named *fields* in the `UniProt` database.As a first example, you will note that the result returned for the query `"Q01844"` has multiple entries. Only one of these is the sequence with `accession` value equal to `"Q01844"`, but the other entries make reference to this sequence somewhere in their database record. If we want to restrict our result only to the particular entry `"Q01844"`, we can specify the field we want to search as `accession`, and build the following query:```pythonquery = "accession:Q01844"   specify a search on the accession field```Note that we can use the same variable name `query` as earlier (this overwrites the previous value in `query`). The code below runs the search and shows the output:<jupyter_code># Make a new query string ("accession:Q01844"), and run a remote search at UniProt
query = "accession:Q01844"
result = service.search(query)

# Inspect the result
print(result)<jupyter_output><empty_output><jupyter_text>By using this and other key:value constructions, we can refine our searches to give us only the entries we're interested in  Exercise 01 (10min)Using `key:value` searches, can you find and download sets of entries for proteins that satisfy the following requirements? (**HINT:** this link to the `UniProt` query fields may be helpful, here): Have publications authored by someone with the surname Broadhurst Have protein length between 9000aa and 9010aa Derive from the taipan snake Have been found in the wing<jupyter_code># SOLUTION - EXERCISE 01
query = "citation:(author:broadhurst)"
result = service.search(query)
print(result)
# SOLUTION - EXERCISE 01
query = "length:[9000 TO 9010]"
result = service.search(query)
print(result)
# SOLUTION - EXERCISE 01
query = "organism:taipan"
result = service.search(query)
print(result)
# SOLUTION - EXERCISE 01
query = 'annotation:(type:"tissue specificity" wing)'
result = service.search(query)
print(result)<jupyter_output><empty_output><jupyter_text>Combining queries    Combining terms in a UniProt query can be as straightforward as putting them in the same string, separated by a space.For example:```pythonquery = "organism:rabbit tissue:eye"```will search for all entries deriving from rabbits that are found in the eye<jupyter_code># Combine queries for rabbit (organism) and eye (tissue), and search
query = "organism:rabbit tissue:eye"
result = service.search(query)
print(result)<jupyter_output><empty_output><jupyter_text>Exercise 02 (10min)Using `key:value` searches, can you find and download sets of entries for proteins that satisfy the following requirements? (**HINT:** this link to the `UniProt` query fields may be helpful, here): Found in sheep spleen Have "rxlr" in their name, have a publication with author name Pritchard, and are between 70aa and 80aa in length Derive from a quokka and have had their annotations manually reviewed Are found in cell membranes of horse heart tissue, and have had their annotations manually reviewed<jupyter_code># SOLUTION - EXERCISE 02
query = "name:rxlr author:pritchard length:[70 TO 80]"
result = service.search(query)
print(result)
# SOLUTION - EXERCISE 02
query = "organism:quokka AND reviewed:yes"
result = service.search(query)
print(result)
# SOLUTION - EXERCISE 02
query = 'organism:horse annotation:(type:"tissue specificity" heart) locations:(location:membrane) reviewed:yes'
result = service.search(query)
print(result)<jupyter_output><empty_output><jupyter_text>Combining terms with Boolean logicBoolean logic allows you to combine search terms with each other in arbitrary ways using three *operators*, specifying whether:* both terms are required (`AND`) **NOTE:** this is implicitly what you have been doing in the examples above* either term is allowed (`OR`)* a term is disallowed (`NOT`)Searches are read from left-to right, but the logic of a search can be controlled by placing the combinations you want to resolve first in parentheses (`()`). Combining these operators can build some extremely powerful searches. For example, to get all proteins from horses and sheep, identified in the ovary, and having length greater than 200aa, you could use the query:```query = "tissue:ovary AND (organism:sheep OR organism:horse) NOT length:[1 TO 200]"```  Processing query resultsSo far you have worked with the default output from `bioservices`, although you know from [the previous notebook](01-uniprot_browser.ipynb) that `UniProt` can provide output in a number of useful formats for searches in the browser.The default output is `tabular`, and gives a good idea of the nature and content of the entries you recover. In this section, you will see some ways to download search results in alternative formats, which can be useful for analysis.All the output format options are controlled in a similar way, using the `frmt=` argument when you conduct your search - with `` being one of the allowed terms (see the [`bioservices` documentation](https://pythonhosted.org/bioservices/references.htmlmodule-bioservices.uniprot) for a full list).  TabularThe default datatype is the most flexible datatype for download: tabular.This can be specified explicitly with the `tab` format:```pythonresult = service.search(query, frmt="tab")```<jupyter_code># Make a query string ("Q01844"), and run a remote search at UniProt,
# getting the result as tabular format (frmt="tab")
query = "Q01844"
result = service.search(query, frmt="tab")

# Inspect the result
print(result)<jupyter_output><empty_output><jupyter_text>By default, the columns that are returned are: `Entry`, `Entry name`, `Status`, `Protein names`, `Gene names`, `Organism`, and `Length`. But these can be modified by passing the `columns=` argument, where the `` is a comma-separated list of column names. For example:```pythoncolumnlist = "id,entry name,length,organism,mass,domains,domain,pathway"result = service.search(query, frmt="tab", columns=columnlist)```The list of allowed column names can be found by inspecting the content of the variable `service._valid_columns`.<jupyter_code># Make a query string ("Q01844")
query = "Q01844"

# Define a list of columns we want to retrive
columnlist = "id,entry name,length,mass,go(cellular component)"

# Run the remote search (columns=columnlist)
result = service.search(query, columns=columnlist)

# View the result
print(result)<jupyter_output><empty_output><jupyter_text>Converting to a dataframe    The pandas module allows us to process tabular data into dataframes, just like in R.To do this, we have to use the `io.StringIO()` class to make it think that our downloaded results are a file```pythondf = pd.read_table(io.StringIO(result))```<jupyter_code># Convert the last search result into a dataframe in Pandas
df = pd.read_table(io.StringIO(result))

# View the dataframe
df<jupyter_output><empty_output><jupyter_text>Doing this will produce a `pandas` dataframe that can be manipulated and analysed just like any other dataframe. We can, for instance, view a histogram of sequence lengths from the table above:<jupyter_code># Plot histogram of protein sequence lengths from dataframe
df.hist();<jupyter_output><empty_output><jupyter_text>Excel    You can download Excel spreadsheets directly from UniProt, just as with the browser interface.```pythonresult = service.search(query, frmt="xls")```You can't use the Excel output directly in your code without some file manipulation, and you'll have to save it to a file, as in the example below. Also, the downloaded format is not guaranteed to be current for your version of Excel, and the application may ask to repair it. But, if you want Excel output to share with/display to others, you can get it programmatically.NOTE: the downloaded format is actually `.xlsx`, rather than `.xls` which is implied by the format<jupyter_code># Make a query string, and run a remote search at UniProt,
# getting the result as an Excel spreadsheer
query = "Q01844"
result = service.search(query, frmt="xls")

# Write the Excel spreadsheet to file
outfile = '../assets/downloads/Q01844.xlsx'
with open(outfile, 'wb') as ofh:
    ofh.write(result)<jupyter_output><empty_output><jupyter_text>FASTA sequenceIf you're interested only in the FASTA format sequence for an entry, you can use the fasta option with frmt to recover the sequences directly, as in the example below:<jupyter_code># Make a query string, and run a remote search at UniProt,
# getting the result as FASTA sequence
query = "go:membrane organism:horse tissue:heart reviewed:yes"
result = service.search(query, frmt="fasta")

# Inspect the result
print(result)<jupyter_output><empty_output><jupyter_text>`pandas` dataframe    In addition to the conversion of tabular output to a pandas dataframe above, you can ask the UniProt() instance to return a pandas dataframe directly, with the .get_df() method.```pythonresult = service.get_df("tissue:venom (organism:viper OR organism:mamba)", limit=None)```However, this is slow compared to the other methods above and can take a long time for queries with thousands of results<jupyter_code># Get a dataframe for all venom proteins from snakes (or mambas, if "snake" is not in the annotation)
df = service.get_df('tissue:venom (organism:viper OR organism:mamba)', limit=None)

# View the dataframe
df.head()<jupyter_output><empty_output><jupyter_text>This dataframe works like any other dataframe. You can get a complete list of returned columns:<jupyter_code>print(list(df.columns))<jupyter_output><empty_output><jupyter_text>Or, for instance, the number of rows and columns in the results:<jupyter_code>print(df.shape)<jupyter_output><empty_output><jupyter_text>and use the convenient features of a dataframe, such as built-in plotting:<jupyter_code># Construct a histogram of returned sequence lengths
df.hist('Length', bins=100);<jupyter_output><empty_output><jupyter_text>and grouping/subsetting:<jupyter_code># Subset out pit vipers
pits = df.loc[df["Organism"].str.contains("pit viper")]
pits.head()

# Plot a strip plot of sequence size by organism in the dataframe
output = sns.stripplot(y="Organism", x="Length",
                       data=pits)  # Render strip plot<jupyter_output><empty_output><jupyter_text>Exercise 03 (10min)Can you use `bioservices`, `UniProt` and `pandas` to:             download a dataframe for all proteins that have "rxlr" in their name         render a violin plot (sns.violinplot()) that shows the distribution of protein lengths grouped according to the evidence for the protein<jupyter_code># SOLUTION - EXERCISE 03

# Get dataframe
df = service.get_df("name:rxlr", limit=None)

# Draw violin plot
output = sns.violinplot(y="Protein existence", x="Length", data=df)

# Profit<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07883418222976797, 'fraction_numerical': 0.010413129598189021, 'mean_word_length': 5.453981008035062, 'pattern_counts': {'":': 0, '<': 77, '<?xml version=': 0, '>': 77, 'https://': 1, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '511116', 'license': 'MIT', 'n_tokens_mistral': 5135, 'n_tokens_neox': 4752, 'n_words': 2307, 'path': 'notebooks/02-uniprot_programming_with_code.ipynb', 'repo_name': 'widdowquinn/2021-01-13_sipbs_workshop', 'size': '17670'}