<jupyter_start><jupyter_text>PIC-SURE API Use-Case: Phenome-Wide Analysis on COPDgene Data This is a tutorial notebook, aimed to be quickly up and running with the Python PIC-SURE API. It covers the main functionalities of the API.  PIC-SURE Python API  What is PIC-SURE?  -->Databases exposed through PIC-SURE API encompasses a wide heterogeneity of architectures and data organizations underneath. PIC-SURE hides this complexity and exposes the different databases in the same format, allowing researchers to focus on the analysis and medical insights, thus easing the process of reproducible sciences. More About PIC-SUREPIC-SURE stands for Patient-centered Information Commons: Standardized Unification of Research Elements. The API is available in two different programming languages, Python and R, allowing investigators to query databases in the same way using any of those languages.PIC-SURE is a large project from which the R/Python PIC-SURE API is only a brick. Among other things, PIC-SURE also offers a graphical user interface, allowing research scientists to get quick knowledge about variables and data available for a specific data source.The Python API is actively developed by the Avillach-Lab at Harvard Medical School.GitHub repo:* https://github.com/hms-dbmi/pic-sure-python-adapter-hpds* https://github.com/hms-dbmi/pic-sure-python-client  -------     Getting Your Own User-Specific Security Token **Before running this notebook, please be sure to review the get_your_token.ipynb notebook. It contains instructions on how to get a security token, mandatory to access the databases.**  Environment Set-Up  Pre-Requisite- Python 3.6 or later (although earlier versions of Python 3 must work too)- PIP: Python package manager, already available in most systems with a Python interpreter installed ([pip installation instructions](https://pip.pypa.io/en/stable/installing/))  IPython Magic CommandThe two lines of code below will load the `autoreload` IPython extension. Although not necessary to execute the rest of the Notebook, it does enable the reload of every dependency each time Python code is executed, thus enabling to take into account changes in external file imported into this Notebook (e.g. user defined function stored in separate file), without having to manually reload libraries. It is very handy when developing interactively. More about [IPython Magic commands](https://ipython.readthedocs.io/en/stable/interactive/magics.html).<jupyter_code>%load_ext autoreload
%autoreload 2<jupyter_output><empty_output><jupyter_text>Installation of Required Python PackagesUsing the PIP package manager, we install the packages listed in the `requirements.txt` file.<jupyter_code>!cat requirements.txt
import sys
!{sys.executable} -m pip install -r requirements.txt
!{sys.executable} -m pip install --upgrade --force-reinstall git+https://github.com/hms-dbmi/pic-sure-python-adapter-hpds.git
!{sys.executable} -m pip install --upgrade --force-reinstall git+https://github.com/hms-dbmi/pic-sure-python-client.git<jupyter_output><empty_output><jupyter_text>Import all the external dependencies, as well as user-defined functions stored in the `python_lib` folder<jupyter_code>import json
from pprint import pprint

import pandas as pd
import numpy as np 
import matplotlib.pyplot as plt
from scipy import stats

import PicSureHpdsLib
import PicSureClient

from python_lib.utils import get_multiIndex_variablesDict, joining_variablesDict_onCol
print("NB: This Jupyter Notebook has been written using PIC-SURE API following versions:\n- PicSureClient: 1.1.0\n- PicSureHpdsLib: 1.1.0\n")
print("The PIC-SURE API libraries versions you've been downloading are: \n- PicSureClient: {0}\n- PicSureHpdsLib: {1}".format(PicSureClient.__version__, PicSureHpdsLib.__version__))<jupyter_output><empty_output><jupyter_text>Set Up the Options for Displaying Tables and Plots in this Notebook<jupyter_code># Pandas DataFrame display options
pd.set_option("max.rows", 100)

# Matplotlib display parameters
plt.rcParams["figure.figsize"] = (14,8)
font = {'weight' : 'bold',
        'size'   : 12}
plt.rc('font', **font)<jupyter_output><empty_output><jupyter_text>Connecting to a PIC-SURE Network The are several pieces of infomration needed before you can get access to data through the PIC-SURE API: a network URL, a resource id, and a user security token which is specific to a given URL + resource.<jupyter_code>PICSURE_network_URL = "https://picsure.biodatacatalyst.nhlbi.nih.gov/picsure"
resource_id = "02e23f52-f354-4e8b-992c-d37c8b9ba140"
token_file = "token.txt"
with open(token_file, "r") as f:
    my_token = f.read()
client = PicSureClient.Client()
connection = client.connect(PICSURE_network_URL, my_token)
adapter = PicSureHpdsLib.Adapter(connection)
resource = adapter.useResource(resource_id)<jupyter_output><empty_output><jupyter_text>Two objects are created here: a `connection` and a `resource` object, using respectively the `picsure` and `hpds` libraries. As we will only be using one single resource, **the `resource` object is actually the only one we will need to proceed with data analysis hereafter** (the `connection` object is useful to get access to different databases stored in different resources). It is connected to the specific data source ID we specified, and enables us to query and retrieve data from this source.  Getting Help with the PIC-SURE Python API Each object exposed by the PicSureHpdsLib library got a `help()` method. Calling it will print out a helper message about it.<jupyter_code>resource.help()<jupyter_output><empty_output><jupyter_text>For instance, this output tells us that this `resource` object got 2 methods, and it gives insights about their function.   Using the Variables Dictionary Once connection to the desired resource has been established, we first need to get a quick grasp of which variables are available in the database. To this end, we will use the `dictionary` method of the `resource` object. A `dictionary` instance offers the possibility to retrieve matching records according to a specific term, or to retrieve information about all available variables, using the `find()` method. For instance, looking for variables containing the term `COPD` is done this way:<jupyter_code>dictionary = resource.dictionary()
dictionary_search = dictionary.find("COPD")<jupyter_output><empty_output><jupyter_text>Subsequently, objects created by the `dictionary.find` exposes the search result using 4 different methods: `.count()`, `.keys()`, `.entries()`, and `.DataFrame()`.<jupyter_code>pprint({"Count": dictionary_search.count(), 
        "Keys": dictionary_search.keys()[0:5],
        "Entries": dictionary_search.entries()[0:5]})
dictionary_search.DataFrame().head()<jupyter_output><empty_output><jupyter_text>**`.DataFrame()` enables us to get the result of the dictionary search in a Pandas DataFrame format. This way, it enables us to:** * Use the various criteria exposed in the dictionary (patientCount, variable type ...) as criteria for variable selection.* Use the row names of the DataFrame to get the actual variables names, to be used in the query, as shown below.Variable names, as currently implemented in the API, aren't handy to use right away.1. Very long2. Presence of backslashes that requires modification right after copy-pasting. However, using the dictionary to select variables can help to deal with this. Hence, one way to proceed is to retrieve the whole dictionary in the form of a Pandas DataFrame, as below:<jupyter_code>plain_variablesDict = resource.dictionary().find().DataFrame()<jupyter_output><empty_output><jupyter_text>Indeed, using the `dictionary.find()` function without arguments returns every entry, as shown in the help documentation.<jupyter_code>resource.dictionary().help()
plain_variablesDict.iloc[10:20,:]<jupyter_output><empty_output><jupyter_text>The dictionary currently returned by the API provides various information about the variables, such as:- observation Count: number of entries with non-null value- categorical: type of the variables, True if categorical, False if continuous/numerical- min/max: only provided for non-categorical variables- HpdsDataType: 'phenotypes' or 'genotypes'. Currently, the API only expsoses'phenotypes' variables  Variable Dictionary + Pandas MultiIndex Though helpful, we can use a simple user-defined function (`get_multiIndex_variablesDict`) to add a little more information and ease dealing with variables names. It takes advantage of Pandas MultiIndex functionality [see Pandas official documentation on this topic](https://pandas.pydata.org/pandas-docs/stable/user_guide/advanced.html).Although not an official feature of the API, such functionality illustrates how to quickly scan an select groups of related variables.Printing the 'MultiIndexed' Variable Dictionary allows us to quickly see the tree-like organization of the variables. Moreover, original and simplified variable names are now stored respectively in the "varName" and "simplified_varName" columns.<jupyter_code>variablesDict = get_multiIndex_variablesDict(plain_variablesDict)
variablesDict
# Now that we have seen how our entire dictionnary looks, we can limit the number of lines to be displayed for the future outputs
pd.set_option("max.rows", 50)<jupyter_output><empty_output><jupyter_text>Below is a simple example to illustrate the ease of use a MultiIndex dictionary. Let's say we are interested in every variables pertaining to the "Medical history" and "Medication history" subcategories.<jupyter_code>mask_medication = variablesDict.index.get_level_values(2) == "Medication History"
mask_medical = variablesDict.index.get_level_values(2) == "Medical History"
medication_history_variables = variablesDict.loc[mask_medical | mask_medication,:]
medication_history_variables<jupyter_output><empty_output><jupyter_text>Although pretty simple, it can be easily combined with other filters to quickly select necessary variables.  Querying and Retrieving Data Aside from the dictionary, the second cornerstone of the API is the `query` object. It is the entering point to retrieve data from the resource.<jupyter_code>my_query = resource.query()<jupyter_output><empty_output><jupyter_text>The query object has several methods that enable us to build a query - The `query.select().add()` method accepts variable names as a string or list of strings as argument, and will allow the query to return all variables included in the list, without any record (ie subjects/rows) subsetting. - The `query.require().add()` method accepts variable names as astring or list of strings as argument, and will allow the query to return all the variables passed, and only records that do not contain any null values for those variables. - The `query.anyof().add()` method accepts variable names as a string or list of strings as argument, and will allow the query to return all variables included in the list, and only records that do contain at least one non-null value for those variables. - The `query.filter().add()` method accept variable names as a string or list of strings as argument, plus additional values to filter on that given variable. The query will return this variable and only the records that do match this filter. All those 4 methods can be combined when building a query. The record eventually returned by the query has to meet all the different specified filters.  Building the Query<jupyter_code>mask = variablesDict["simplified_name"] == "How old were you when you completely stopped smoking? [Years old]"
yo_stop_smoking_varname = variablesDict.loc[mask, "name"] 
mask_cat = variablesDict["categorical"] == True
mask_count = variablesDict["observationCount"].between(100,2000)
varnames = variablesDict.loc[mask_cat & mask_count, "name"]
my_query.filter().add(yo_stop_smoking_varname, min=20, max=70)
my_query.select().add(varnames[:50])<jupyter_output><empty_output><jupyter_text>Retrieving the Data Once our query object is finally built, we use the `query.run` function to retrieve the data corresponding to our query<jupyter_code>query_result = my_query.getResultsDataFrame(low_memory=False)
query_result.shape
query_result.head()
query_result[yo_stop_smoking_varname].plot.hist(legend=None, title= "Age stopping smoking", bins=15)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07439291805408918, 'fraction_numerical': 0.005928693251035491, 'mean_word_length': 5.856347438752784, 'pattern_counts': {'":': 3, '<': 61, '<?xml version=': 0, '>': 62, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '37363', 'license': 'Apache-2.0', 'n_tokens_mistral': 3366, 'n_tokens_neox': 3222, 'n_words': 1408, 'path': 'NHLBI_BioData_Catalyst/python/PICSURE_API_101.ipynb', 'repo_name': 'Pmajette/Access-to-Data-using-PIC-SURE-API', 'size': '12313'}