<jupyter_start><jupyter_text>First Steps with *pyopencga*; the Python client of OpenCGA------This notebook provides guidance for getting started with the *pyopencga* library, which is the Python client of OpenCGA. _pyopencga_ is a **REST client** that fully implements OpenCGA REST API.These notebooks use a demo installation available at the Univeristy of Cambrdige, feel free to change OpenCGA host and credentials to use any other OpenCGA server.We assume that your workstation (Linux, Mac, Windows) is connected to the internet and you have Python 3 and the *pip* package manager installed. We then show you how to:- Install *pyopencga*.- Connect to an OpenCGA instance.- Execute OpenCGA calls and work with responses.- Launch asynchronous jobs and retrieve results.Walk-through guides of some **common use cases** are provided in two further notebooks:- pyopencga_catalog.ipynb- pyopencga_variant_query.ipynb- pyopencga_variant_analysis.ipynb    You can check OpenCGA REST Web Service API with the following public OpenCGA installation:- https://ws.opencb.org/opencga-prod/webservices  Installing and importing the *pyopencga* library------- 1. Install *pyopencga* with *pip*_pyopencga_ is available at PyPI (https://pypi.org/project/pyopencga/) you can easily install it by exeuting:`$ pip install pyopencga`_pyopencga_ uses some other dependencies , make sure you have installed Pandas, IPython or matplotlib.  2. Import *pyopencga* libraryYou can find here the import section required to use _pyopencga_:<jupyter_code>from pyopencga.opencga_config import ClientConfiguration # import configuration module
from pyopencga.opencga_client import OpencgaClient # import client module
from pprint import pprint
from IPython.display import JSON
import matplotlib.pyplot as plt
import seaborn as sns
import datetime<jupyter_output><empty_output><jupyter_text>3. Setup OpenCGA Client You need to provide **at least** a OpenCGA host server URL in the standard configuration format for OpenCGA as a python dictionary or in a json file.Regarding credentials, you can set both user and password as two variables in the script. If you prefer not to show the password, it would be asked interactively without echo.  Set variables for server host, user credentials and project owner<jupyter_code># Server host
host = 'http://bioinfo.hpc.cam.ac.uk/opencga-prod'

# User credentials
user = 'demouser'
passwd = '<PASSWORD>' ## You can skip this, see section 1.3.4 below.<jupyter_output><empty_output><jupyter_text>Creating ConfigClient dictionary for server connection configuration<jupyter_code># Creating ClientConfiguration dict
config_dict = {'rest': {
                       'host': host 
                    }
               }

print('Config information:\n',config_dict)<jupyter_output>Config information:
 {'rest': {'host': 'http://bioinfo.hpc.cam.ac.uk/opencga-prod'}}<jupyter_text>4. Initialize the Client Now we need to pass the *config_dict* dictionary to the **ClientConfiguration** method.Once we have the configuration defined as *config* (see below), we can initiate the client. This is the **most important step**. OpencgaClient: what is and why is so important?The `Opencgaclient` (see *oc* variable below) implements all the methods to query the REST api of OpenCGA. All the webservices available, can be directly accesed through the client.<jupyter_code>## Create the configuration
config = ClientConfiguration(config_dict)

## Define the client
oc = OpencgaClient(config)<jupyter_output><empty_output><jupyter_text>Once we have defined a variable with the client configuration and credentials, we can access to all the methods defined for the client. These methods implement calls to the OpenCGA **[web service endpoints](https://ws.opencb.org/opencga-prod/webservices/)** used by pyopencga.  5. Import the credentials and Login into OpenCGAWe can decide to pass the password as a variable, or just pass the user and be asked for the password interactively<jupyter_code>## Option 1: here we put only the user in order to be asked for the password interactively

# oc.login(user)
## Option 2: you can pass the user and passwd
oc.login(user, passwd)<jupyter_output><empty_output><jupyter_text>✅  Congrats! You are now connected to OpenCGA  Understanding REST Response--------*pyopencga* queries web services that return a RESTResponse object, which might be difficult to interpretate. The RESTResponse type provide the data in a manner that is not as intuitive as a python list or dictionary. Because of this, we have develop a useful functionality that retrieves the data in a simpler format. [OpenCGA Client Libraries](http://docs.opencb.org/display/opencga/Using+OpenCGA), including *pyopencga*, implement a **RESTReponse wrapper** to make even easier to work with REST web services responses. REST responsess include metadata and OpenCGA 2.0.1 has been designed to work in a federation mode (more information about OpenCGA federations can be found **[here](http://docs.opencb.org/display/opencga/Roadmapg)**).All these can make a first-time user to struggle when start working with the responses. Please read this brief documentation about **[OpenCGA RESTful Web Services](http://docs.opencb.org/display/opencga/RESTful+Web+ServicesRESTfulWebServices-OpenCGA2.x)**.Let's see a quick example of how to use RESTResponse wrapper in *pyopencga*. You can get some extra inforamtion [here](http://docs.opencb.org/display/opencga/PythonPython-WorkingwiththeRestResponse). Let's execute a first simple query to fetch all projects for the user **demouser** already logged in **[step 1.3](1.3-Setup-the-client-and-login-in-pyopencga)**.<jupyter_code>## Let's fecth the available projects.
## First let's get the project client and execute search() funciton

project_client = oc.projects
projects = project_client.search()

## Uncomment this line to view the JSON response.
## NOTE: it incudes study information so this can be big
##pprint(projects.get_responses())<jupyter_output><empty_output><jupyter_text>Although you can iterate through all the different projects provided by the response by executing the next chunk of code, this is a **not recommended** way.We can explore this through an example; the next query iterates over all the projects retrieved from `projects.search()`<jupyter_code>## Loop through all diferent projects 

for project in projects.responses[0]['results']:
   print(project['id'], project['name'])<jupyter_output>family Family Studies GRCh37
population Population Studies GRCh38<jupyter_text>RestResponse API Note: Table with API funcitons and the description  1. Using the `get_results()` function Using the functions that *pyopencga* implements for the RestResponse object makes things much easier!  Let's dig into an example using the same query as above:<jupyter_code># Loop through all diferent projects 

for project in projects.get_results():
   print(project['id'], project['name'])<jupyter_output>family Family Studies GRCh37
population Population Studies GRCh38<jupyter_text>2. Using the `result_iterator()` function to iterate over the Rest resultsYou can also iterate results, this is specially interesting when fetching many results from the server:<jupyter_code>## Iterate through all diferent projects 

for project in projects.result_iterator():
   print(project['id'], project['name'])<jupyter_output>family Family Studies GRCh37
population Population Studies GRCh38<jupyter_text>3. Using `print_results()` function to iterate over the Rest results**IMPORTANT**: This function implements a configuration to exclude metadata, change separator or even select the fields! Then it reaches all the user-desired results and prints them directly in the terminal.In this way, the RESTResponse obejct implements a very powerful custom function to print results 😎<jupyter_code>## This function iterates over all the results, it can be configured to exclude metadata, change separator or even select the fields!

projects.print_results()<jupyter_output>#Time: 81
#Num matches: -1
#Num results: 2
#Num inserted: 0
#Num updated: 0
#Num deleted: 0
#id	name	uuid	fqn	creationDate	modificationDate	description	organism	currentRelease	studies	internal	attributes
family	Family Studies GRCh37	eba0e1c7-0172-0001-0001-c7af712652b2	demo@family	20200625131808	20200625131808		{'scientificName': 'Homo sapiens', 'commonName': '', 'assembly': 'GRCh37'}	1	.	{'datastores': {}, 'status': {'name': 'READY', 'date': '20200625131808', 'description': ''}}	{}
population	Population Studies GRCh38	25f2842a-0173-0001-0001-e7bcbedc77ff	demo@population	20200706210517	20200706210517	Some population reference studies for GRCh38	{'scientificName': 'Homo sapiens', 'commonName': '', 'assembly': 'GRCh38'}	1	.	{'datastores': {}, 'status': {'name': 'READY', 'date': '20200706210517', 'description': ''}}	{}<jupyter_text>Now, let's try to costumize the results so we can get printed only the portion of the data that we might be interested in<jupyter_code>## Lets exclude metadata and print only few fields, use dot notation for ensted fields

projects.print_results(fields='id,name,organism.scientificName,organism.assembly',metadata=False)
print()<jupyter_output>#id	name	organism.scientificName	organism.assembly
family	Family Studies GRCh37	Homo sapiens	GRCh37
population	Population Studies GRCh38	Homo sapiens	GRCh38<jupyter_text>A very useful parameter is the *separator*.It allows the user to decide the format in which the data is printed. For example, it's possible to print a CSV-like style:<jupyter_code>## You can change separator

print('Print the projects with a header and a different separator:\n')
projects.print_results(fields='id,name,organism.scientificName,organism.assembly', separator=',', metadata=False)<jupyter_output>Print the projects with a header and a different separator:<jupyter_text>4. Using Pandas DataFrame: `to_data_frame()` Pandas provides a very useful functionality for data science. You can convert RestResponse objects to Pandas DataFram using the following function:`res_response.to_data_frame()`  5. Wait for a job to finish `wait_for_job()`This will stop python execution until the jobs been completed.  Working with Jobs------------------ OpenCGA implemets both a powerful interactive API to query data but also allows users to execute more demanding analysis by executing jobs. There are a number of analysis and operations that are executed as jobs such as Variant Export or GWAS analysis.  1. Job InfoJob data model contain all the information about a single excution: date, id, tool, status, files, ...   You can filter jobs by any of these parameters and even get some stats.  2. Executing JobsJob execution invovle different lifecycle stages: pending, queued, running, done, rejected, ...  OpenCGA takes care of executing and notifying job changes.Executing a job is as simpla s the following code<jupyter_code>## Eexecute GWAS analysis
#rest_response = oc.variant().gwas()

## wait for the job to finish
#oc.wait_for_job(rest_response)

#rest_response.print_results()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07911392405063292, 'fraction_numerical': 0.017540687160940326, 'mean_word_length': 5.615430622009569, 'pattern_counts': {'":': 0, '<': 47, '<?xml version=': 0, '>': 47, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '179443', 'license': 'Apache-2.0', 'n_tokens_mistral': 3216, 'n_tokens_neox': 3005, 'n_words': 1297, 'path': 'opencga-client/src/main/python/notebooks/user-training/pyopencga_first_steps.ipynb', 'repo_name': 'julie-sullivan/opencga', 'size': '11058'}