<jupyter_start><jupyter_text>Driving an Jupyter notebook from GA4GH Search schemaThis notebook explores how the schema available via GA4GH Search can be used to drive a user interface. In this case a Jupyter notebook.>Note: The sequence files used in this notebook are  under controlled access. You will not be able to access those files unless you have been granted access through dbGaP to the phs001554 study. If you have that access you will also have access to the non-scrambled subject and sample data. If that is the case, we would be pleased to hear of your interest, and the possibility of collaborating to ensure the GA4GH tools described here enable your studies. Please see [form to register interest](https://docs.google.com/forms/d/e/1FAIpQLSfmmc3VKd6ANdzaVMyelT3c9gIWuoS4ZwT0vsqD-o2ZRxJf7A/viewform)<jupyter_code>from fasp.search import DiscoverySearchClient
from fasp.loc import crdcDRSClient
from fasp.workflow import sbcgcWESClient
from fasp.runner import FASPRunner

faspRunner = FASPRunner(program='GECCO_Gen3_on_SB')
runNote = 'GECCO files retrieved from Gen3 DRS and computed on SB'

searchClient = DiscoverySearchClient('https://ga4gh-search-adapter-presto-public.prod.dnastack.com/')
query = '''select acc, affection_status, sex, age, race, study, drs_id cram_drs_id, sa.sample_id
from search_cloud.cshcodeathon.gecco_gen3_drs_index i
join dbgap_demo.scr_gecco_susceptibility.sample_multi sa on sa.sample_id = i.sample_id
join dbgap_demo.scr_gecco_susceptibility.subject_phenotypes_multi su on su.dbgap_subject_id = sa.dbgap_subject_id
where file_type = 'cram' '''
df = searchClient.runQuery(query, returnType='dataframe')

df = searchClient.runQuery(query, returnType='dataframe')

def getColValues(info, columns):
    enumVals = {}
    for column in columns:
        var = info['data_model']['properties'][column]
        valueList = []
        for value in var['oneOf']:
            valueList.append(value['const'])
        enumVals[column] = valueList
    return enumVals
    
info = searchClient.listTableInfo('dbgap_demo.scr_gecco_susceptibility.subject_phenotypes_multi')
enumCols = getColValues(info, ['affection_status', 'sex', 'race', 'study'])
print(enumCols)<jupyter_output>_Retrieving the query_
____Page1_______________
____Page2_______________
____Page3_______________
____Page4_______________
____Page5_______________
____Page6_______________
{'affection_status': ['Case', 'Control'], 'sex': ['Female', 'Male'], 'race': ['White'], 'study': ['CPS-II', 'DACHS', 'HPFS', 'NHS', 'PLCO', 'WHI']}<jupyter_text>Build a study specific user interface from the information the GA4GH Search SchemaUse IPython widgets to filter data from the retrived data frame. the controls of the interface are configured using the schema provided by GA4GH Search.<jupyter_code>import ipywidgets as widgets
from ipywidgets import interact, interact_manual, IntRangeSlider
@interact
def filter_gecco(sex=enumCols['sex'], 
                 ageRange=IntRangeSlider(value=[50, 55],min=0,max=80,step=1,description='Age Range:'),
                 race=enumCols['race'],
                 affection_status=enumCols['affection_status'],
                 study=enumCols['study'],

                ):
    
    return df.loc[(df['age'].between(50,55)) & (df['sex'] == sex) & (df['race'] == race) &
                  (df['affection_status'] == affection_status) & (df['study'] == study)]<jupyter_output><empty_output><jupyter_text>To do: Retrieve the filtered data frame and submnit the selected files for compute.<jupyter_code>drsClient = crdcDRSClient('/Users/forei/.keys/crdc_credentials.json', 's3')

settings = faspRunner.settings
wesClient = sbcgcWESClient(settings['SevenBridgesProject'])

faspRunner.configure(searchClient, drsClient, wesClient)

run_ids = faspRunner.runDRSIDs(query, runNote)
import pandas as pd
runs_df = pd.DataFrame(run_ids)
runs_df
merged_df = pd.merge(
    df,
    runs_df,
    how="inner",
    left_on='acc',
    right_on='subject'
)
merged_df.drop('subject', axis=1, inplace=True)
merged_df.drop('drs_id', axis=1, inplace=True)
merged_df.to_csv('~/mywork.csv')
merged_df
faspRunner.getFASPicon()<jupyter_output><empty_output><jupyter_text>Update the dataframe with the status of the runs<jupyter_code>for index, row in merged_df.iterrows():
    status = wesClient.getTaskStatus(row['run_id'])
    merged_df.at[index, "status"] = status
merged_df<jupyter_output><empty_output><jupyter_text>Once the runs have completed the results can be retrieved.First two convenience functions are defined<jupyter_code>import requests
import os
def download(url, file_path):
    with open(os.path.expanduser(file_path), "wb") as file:
        response = requests.get(url)
        file.write(response.content)
        
def getStats(filePath, statsList):
    ''' Extract values from a SAMTools Stats results file'''
    ### 
    
    retDict = {}
    f = open(filePath, "r")
    for x in f:
        if x.startswith('SN'):   
            parts = x.split('\t')
            statName = parts[1].split(':')[0]
            if statName in statsList:
                retDict[statName] = parts[2].rstrip()
    return retDict
        

from fasp.loc import DRSMetaResolver
mr = DRSMetaResolver()
statsList = []
for index, row in merged_df.iterrows():
    log = wesClient.GetRunLog(row['run_id'])
    resultsDRSID = log['outputs']['statistics']['path']
    url = mr.getAccessURL2(resultsDRSID,'s3')
    fileName = log['outputs']['statistics']['name']
    download(url, fileName)
    statsRequired = ['insert size average','insert size standard deviation']
    stats = getStats(fileName, statsRequired)
    stats['acc'] = row['acc']
    statsList.append(stats)
    os.remove(fileName)
stats_df =  pd.DataFrame(statsList)
stats_df
final_df = pd.merge(
    merged_df,
    stats_df,
    how="inner",
    left_on='acc',
    right_on='acc'
)
final_df.drop('status', axis=1, inplace=True)
final_df.drop('run_id', axis=1, inplace=True)

final_df<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11152042866711319, 'fraction_numerical': 0.008707300736771601, 'mean_word_length': 4.782187802516941, 'pattern_counts': {'":': 0, '<': 20, '<?xml version=': 0, '>': 21, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '231660', 'license': 'Apache-2.0', 'n_tokens_mistral': 1976, 'n_tokens_neox': 1912, 'n_words': 476, 'path': 'notebooks/GECCO_Interactive.ipynb', 'repo_name': 'jb-adams/fasp-scripts', 'size': '5972'}