<jupyter_start><jupyter_text>Programmatically Access TCGA Data using the Seven Bridges Cancer Genomics Cloud using SPARQLTCGA is one of the world’s largest cancer genomics data collections, including more than eleven thousand patients, representing 33 cancers, and over half a million total files. Seven Bridges has created a unified metadata ontology from the diverse cancer studies, made this data available, and provided compute infrastructure to facilitate customized analyses on the Cancer Genomics Cloud (the CGC). The CGC provides powerful methods to query and reproducibly analyze TCGA data - alone or in conjunction with your own data.We continue to develop new methods of interacting with data on the CGC, however, we also appreciate that sometimes it is useful to be able to analyze data locally, or in an AWS environment that you have configured yourself. While the CGC has undergone thorough testing and is certified as a FISMA-moderate system, if you wish to analyze data in alternative locations, you must take the appropriate steps to ensure your computing environment is secure and compliant with current best practices. If you plan to download large numbers of files for local analysis, we recommend using the download utilities available from the Genomic Data Commons which have been specifically optimized for this purpose.Below, we provide a tutorial showing how to find and access TCGA data using the SPARQL console. Alternatively, you can try to query TCGA data [using the Datases API](access_TCGA_on_AWS_via_DatasetsAPI.ipynb). ObjectiveLet's conduct an investigation to look for female Breast Cancer patients who are alive. We want their associated files to be open data, provide gene expression, and be produced by an experimental strategy of RNA-seq. ProcedureIn this tutorial, we will demonstrate how you can use the [SPARQL console](https://opensparql.sbgenomics.com//console) to find all open access gene expression files obtained from RNA-Seq analysis of living female Breast Cancer patients.This method allows you to query the following TCGA entities:* cases* analytes* radiation therapies* drug therapies* follow ups* portions* aliquots* samples* slides* new tumor events* filesA a SPARQL query can return metadata fields, which lets you access and manipulate properties like metadata values. This gives you more flexibility with your query. PrerequisitesThis tutorial is written in **Python 2.7**, for compatibility with Python 3 please swap out the _urllib_ library.Before you begin this tutorial, you should: 1. **Set up your CGC account.** If you haven't already done so, navigate to https://cgc.sbgenomics.com/ and follow these [directions](doc:sign-up-for-the-cgc) to register for the CGC. This tutorial uses Open Data, which is available to all CGC users. The same approach can be used by approved researchers to access Controlled Data. Learn more about TCGA data access here. 2. **Install the Seven Bridges' API Python library.** This tutorial uses the library sevenbridges-python. Learn how to [install it](blog:announcing-the-release-of-seven-bridges-api-clients-in-r-and-python) before continuing. 3. **Obtain your authentication token.** You'll use your authentication token to encode your user credentials when interacting with the CGC programmatically. Learn how to [access your authentication token](http://docs.cancergenomicscloud.org/docs/get-your-authentication-token). It is important to store your authentication token in a safe place as it can be used to access your account. The time and location your token was last used is shown on the developer dashboard. If for any reason you believe your token has been compromised, you can regenerate it at any time.  Query using a SPARQL queryYou can query TCGA data using the query language SPARQL (recursively short for SPARQL Protocol and RDF Query Language). Seven Bridges has made a public SPARQL endpoint to which you can send these queries. In addition to returning entities, the SPARQL query can also return properties such as TCGA metadata fields.Since we'll use a Python script to query TCGA data with a SPARQL query, we'll need to import urllib and SPARQLWrapper for checking the OpenSPARQL endpoint and constructing the SPARQL object.We'll also need to import two Python modules: `json` and `requests`. We'll use these to write a wrapper around the CGC API request in the next step.<jupyter_code># Needed for SPARQL endpoint
import urllib
import SPARQLWrapper as spark<jupyter_output><empty_output><jupyter_text>This tutorial relies on a public endpoint. First, ensure the end point is currently operational. Then, initialize the SPARQL object as below.<jupyter_code># Check SPARQL endpoint
try:
    rc = urllib.urlopen("https://opensparql.sbgenomics.com").getcode()
except Exception:
    rc = 0
if rc != 200:
    print(
        """script relies on sparql endpoint 
        (https://opensparql.sbgenomics.com/) 
        which is currently not responding. 
        Can not continue, exiting.""")
    raise KeyboardInterrupt
else:
    print("Endpoint is operational, we are good to go!")
    
    
# Initialize SPARQL object
sparql_endpoint = "https://opensparql.sbgenomics.com/blazegraph/namespace/tcga_metadata_kb/sparql"
sparql = spark.SPARQLWrapper(sparql_endpoint)<jupyter_output><empty_output><jupyter_text>Now, we can define a query for TCGA data based on its [metadata](doc:tcga-metadata-on-the-cgc).As mentioned in the **Objective** above, we want to search for **female**, **Breast Cancer** patients (**cases**) who are **alive** and the associated **files** which are **open-access**, provide **Gene expression**, and came from an **experimental strategy** of **RNA-seq**. We will assign an exact value to the above properties.However, the query also needs a few non-specific parameters such as  `?file tcga:hasStoragePath ?path`. Non-specific parameters, such as `?path` and `days_to_follow` are returned by the query and can subsequently be analyzed in Python. This contrasts with the specific parameters, such as `?vs rdfs:label 'Alive'`.We include the non-specific parameters in the query without an exact value so that this information is returned by the query. We will need it for accessing the file in the next step. Alternatively, you can return properties and operate on them directly in Python, as in the example below with `?case tcga:hasDaystoLastFollowup ?days_to_follow`.Below, we set the query and execute it. The query results are stored in an object named `results`.<jupyter_code># Create the query above as a block-string
query = """
    prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>
    prefix tcga: <https://www.sbgenomics.com/ontologies/2014/11/tcga#>

    select distinct ?case_id ?file_name ?file ?id ?vital_status ?days_to_follow
    where
    {
      ?case a tcga:Case .
      ?case rdfs:label ?case_id .
       
      ?case tcga:hasDiseaseType ?dt .
      ?dt rdfs:label 'Breast Invasive Carcinoma' .
 
      ?case tcga:hasGender ?gender.
      ?gender rdfs:label 'FEMALE' .
   
      ?case tcga:hasVitalStatus ?vs .
      ?vs rdfs:label 'Alive' .
       
      ?case tcga:hasDaysToLastFollowUp ?days_to_follow .
 
      ?case tcga:hasFile ?file .
     
      ?file rdfs:label ?file_name .
      ?file tcga:hasFileId ?id .
       
      ?file tcga:hasAccessLevel ?ac .
      ?ac rdfs:label 'Open' .
       
      ?file tcga:hasExperimentalStrategy ?es .
      ?es rdfs:label 'RNA-Seq'.
       
      ?file tcga:hasDataType ?dat.
      ?dat rdfs:label 'Gene expression'
    }
"""

sparql.setQuery(query)              # Define query on the wrapper
sparql.setReturnFormat(spark.JSON)  # We want server to return JSON to use
results = sparql.query().convert()  # Convert results to Python object<jupyter_output><empty_output><jupyter_text>Now, we can find properties which are actionable in Python from the `results` object. Below, we can first extract two examples of properties, `UUID` and `Days to last followup`, which we can analyze in Python. Using this option, we can conduct further analysis on the data based on its metadata without downloading the files.Next, we pull out two properties which will be necessary for downloading the data, `Path` and `File name`.Finally, we print out summary stats about the query and list the first 10 results. NOTEThere will very likely be repetitive _UUIDs_ and _days to followup_ in the print-out. This is expected behavior because we have not restricted the query to be exclusive, e.g. by specifying `Sample : hasSampleType = 'Primary Tumor'`. Instead we are seeing multiple files per sample, likely due to multiple samples.<jupyter_code># Information (potentially actionable) about the query results
uuid_list = [result['case_id']['value'] for result in results['results']['bindings']]
day_to_follow_list = \
[result['days_to_follow']['value'] for result in results['results']['bindings']]

# Information for downloading files within the query
file_paths = [result['id']['value'] for result in results['results']['bindings']]
file_names = [result['file_name']['value'] for result in results['results']['bindings']]
file_ids = [result['file']['value'].split('/')[-1] for result in results['results']['bindings']]

# Print some information about the query results
print("Query returned %i results, printing the first 10:" % (len(uuid_list)))
for ii in range(0,min(10, len(uuid_list))):
    print("Case UUID %s had %s days to last followup \n" \
         % (uuid_list[ii], day_to_follow_list[ii]))<jupyter_output><empty_output><jupyter_text>Access TCGA data using the CGC APIIn this section, we will use the CGC API to access TCGA data. Since we are using the CGC API (as opposed to the Datasets API in the previous step), we will use the [sevenbridges-python bindings](http://sevenbridges-python.readthedocs.io/en/latest/). Then, we'll provide our credentials. Finally, we're ready to access TCGA data. Install sevenbridges-pythonTo [install sevenbridges-python](http://sevenbridges-python.readthedocs.io/en/latest/installation/), enter:```bashpip install sevenbridges-python```Once you install sevenbridges-python, you're ready to authenticate with the Seven Bridges API. Set up your authentication tokenSince we're now using the CGC API, we need to provide our credentials. Once you have installed sevenbridges-python, you can authenticate by storing your credentials in a config file, .sbgrc. Enter your credentials in the config file, as shown below, replacing the last line with your authentication token:```bash[cgc]api-url = https://cgc-api.sbgenomics.com/v2auth-token = insert auth token here``` Initialize the sevenbridges-python libraryWe've now installed sevenbridges-python and stored our credentials in a config file. Let's import the `api` class from the official `sevenbridges-python` bindings.<jupyter_code>import sevenbridges as sbg<jupyter_output><empty_output><jupyter_text>Let's initialize the `api` object so the API knows our credentials.<jupyter_code># [USER INPUT] specify platform {cgc, sbg}
prof = 'cgc'


config_file = sbg.Config(profile=prof)
api = sbg.Api(config=config_file)<jupyter_output><empty_output><jupyter_text>Access TCGA dataNow, we loop through the first ten files in the first item of the `files_in_query` list from above using the `id` key.We'll do the following with these ids:1. Create a list of files on the CGC. From this point, it would be possible to take action on the CGC. For instance, you can use a bioinformatics workflow or tool on these files and start an analysis.2. (optional) Generate a list of download links.3. Access each of the ten files in this list. They will be saved to the Downloads folder in your local directory.<jupyter_code># 1) Generate a list a file objects from the file_ids list
file_list = []
for f_id in file_ids[0:10]:
    file_list.append(api.files.get(id = f_id))
    print(file_list[-1].name)    
    
# (BRANCH-POINT) Do something AWESOME with these files on the CGC


# 2) (optional) Generate a list of download links
dl_list = []
for f in file_list:
    dl_list.append(f.download_info())

    
# 3) Download each of the files in the list to a _downloads_ folder in your local directory.
import os

dl_dir = 'downloads'
try:
    os.stat(dl_dir)
except:
    os.mkdir(dl_dir)

for f in file_list:
    f.download(path = ("%s/%s" % (dl_dir, f.name)))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07587044534412955, 'fraction_numerical': 0.003562753036437247, 'mean_word_length': 4.634580291970803, 'pattern_counts': {'":': 0, '<': 31, '<?xml version=': 0, '>': 31, 'https://': 7, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 13, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '318757', 'license': 'CC-BY-4.0', 'n_tokens_mistral': 3455, 'n_tokens_neox': 3289, 'n_words': 1551, 'path': 'Tutorials/CGC/access_TCGA_on_AWS.ipynb', 'repo_name': 'sbg/okAPI', 'size': '12350'}