<jupyter_start><jupyter_text>Comparing approaches: Identifying DRS object via query or unpacking DRS BundlesThis notebook explores two approaches to getting to specific objects or files via DRS.* Identifying the specific file of interest from attributes of subjects and spcimens (somnetimes called metadata)* Extracting a specific file of interest from a bundleFor context, another notebook shows how the files identified via the approaches here can be submiited for compute via a  WES service. The data and files used are from the Thousand Genomes project. The following query using GA4GH Search shows how, in a single step, the DRS ids for mapped BAM files for whole exome sequencing for subjects from a particular population.This example was worked through in the January 2021 FASP Hackathon.<jupyter_code>from fasp.search import DiscoverySearchClient

# Step 1 - Discovery
# query for relevant DRS objects
searchClient = DiscoverySearchClient('https://ga4gh-search-adapter-presto-public.prod.dnastack.com/')

query = '''SELECT f.sample_name, drs_id bam_drs_id, acc
FROM thousand_genomes.onek_genomes.ssd_drs s 
join thousand_genomes.onek_genomes.sra_drs_files f on f.sample_name = s.su_submitter_id 
where filetype = 'bam' and mapped = 'mapped' 
and sequencing_type ='exome' and  population = 'JPT' '''

resultRows = searchClient.runQuery(query, returnType='dataframe')
resultRows<jupyter_output>_Retrieving the query_
____Page1_______________
____Page2_______________
____Page3_______________
____Page4_______________
____Page5_______________
____Page6_______________<jupyter_text>The following shows how the SRA DRS server can be used to determine where the files can be obtained from. The following shows this for the first DRS id from the query results.<jupyter_code>from fasp.loc import SRADRSClient

# Set up a client to access NCBI's  DRS Server for the Sequence Read Archive (SRA)
drsClient = SRADRSClient('https://locate.be-md.ncbi.nlm.nih.<EMAIL>', public=True)
# Get the DRS id from the query results above
test_id = resultRows.iloc[0]['bam_drs_id']
print(test_id)
# Use the DRS GetObject function to 
objInfo = drsClient.getObject(test_id)
objInfo<jupyter_output>fb1cfb04d3ef99d07c21f9dbf87ccc68<jupyter_text>A second DRS call can be used to obtain a url to access the file from one of the above locations.<jupyter_code>access_id = objInfo['access_methods'][0]['access_id']
print('access_id:{}'.format(access_id))
url = drsClient.getAccessURL(test_id, access_id=access_id)
print('url:{}'.format(url))<jupyter_output>access_id:b5f46aadbcb48d7141104db0440feb63cd4e61c8
url:https://1000genomes.s3.amazonaws.com/phase3/data/NA18948/exome_alignment/NA18948.mapped.ILLUMINA.bwa.JPT.exome.20121211.bam<jupyter_text>The approach above is that which was used in the full FASP example that runs the compute. Refer to that notebook.  Second approach - unpacking DRS Bundles provided by SRA DRS ServerHere we'll continue with the second approach to working with SRA ids.It addresses one aspect of bundling in DRS - namely when the bundle contains a collection of different files related to the provided DRS id.  First a query against the data we used above for comparison.<jupyter_code>searchClient.query2Frame('''select filetype, mapped, count(*) rowCount from thousand_genomes.onek_genomes.sra_drs_files 
group by filetype, mapped  ''')<jupyter_output>_Retrieving the query_
____Page1_______________
____Page2_______________
____Page3_______________
____Page4_______________
____Page5_______________<jupyter_text>The SRA Identity Exchange and DRS servicesCan we take an SRA accession number from above and see what it looks like through the SRA IDentity eXchange service (IDX), and how that works through in DRS. We'll start with a run accession an SRR?The SRADRSClient has an additional function to access the IDX service with a SRA accession number<jupyter_code>import json
from fasp.loc import SRADRSClient
drsClient = SRADRSClient('https://locate.be-md.ncbi.nlm.nih.gov', public=True)

accession = 'SRR1601121'
idx = drsClient.acc2drs(accession)
print(json.dumps(idx, indent=3))
drsId = idx['response'][accession]['drs']
print (drsId)<jupyter_output>{
   "drs-base": "drs://locate.ncbi.nlm.nih.gov",
   "response": {
      "SRR1601121": {
         "drs": "9466d7c1ec8fde019ce630c9bd88582e",
         "status_code": 200
      }
   }
}
9466d7c1ec8fde019ce630c9bd88582e<jupyter_text>Note: the base URI returned in the result above suggests the DRS service could be accessed at the URL https://locate.ncbi.nlm.nih.gov . At present, for performance purposes the SRA DRS service should be accessed at https://locate.be-md.ncbi.nlm.nih.gov. See the example aboveNow use the DRS service with that id.<jupyter_code>drsClient.getObject(drsId)<jupyter_output><empty_output><jupyter_text>Our intent as in the first approach is to work with the mapped bam file. We can see visually, from the filename, which file which DRS id we need. An issueThis highlights the first issue with this approach. The information we need to identify the file we need is in the file name. That would be fine for low throughput situations carried out by human eye. It does not scale to machine actionable larger use cases. Moving onWe use the manually identified id via DRS to identify how we may get the file of interest. This is identical to how we did this under the first approach.<jupyter_code>drsClient.getObject('fb1cfb04d3ef99d07c21f9dbf87ccc68')<jupyter_output><empty_output><jupyter_text>In the first approach we were able to identify the DRS ids in the same step as querying for the other sample and subject attributes of interest.The first approach is common in many other cases. See fasp-scripts notebooks for a range of examples where it is possible to get DRS ids from a query against some external database.  Bundling at a higher level. An SRP accessionIn another application of bundling, a bundle is simply a list of other DRS idsSRA's data model has a number of levels above the Run. In descending order they are.* SRP - Project, a project in which sequencing has been done* SRS - Sample, a physical sample from the project. What it represents depnds on scientific investigation in the Project.* SRX - Experiment, the application of a particular sequencing technology to some Sample* SRR - Run, the run, on a sequencer, of material from the ExperimentIn the following example, the IDentity eXchange service (IDX) is called to get the DRS id which corresponds to the project. A feature in iPython is used to time how long the response takes.Please be patient for this step to complete, it can take 45-90 seconds.<jupyter_code>%%time
srp = 'SRP048601'
res = drsClient.acc2drs(srp, verbose=True)

#SRS000157
drsId = res['response'][srp]['drs']
print(drsId)<jupyter_output>https://locate.be-md.ncbi.nlm.nih.gov/idx/v1/SRP048601
<Response [200]>
5d8b77dd974e1b7c9de4040cbf9a24c7
CPU times: user 16.6 ms, sys: 4.08 ms, total: 20.7 ms
Wall time: 1min 13s<jupyter_text>The DRS id for the project can now be sent to the DRS server. The response will consists of a bundle of DRS ids for experiments within the project.Note the response time, though it needs to return a list of 5070 ids it does so in less than a second.<jupyter_code>%%time
drsRes = drsClient.getObject(drsId)
print(len(drsRes['contents']))<jupyter_output>5070
CPU times: user 18.2 ms, sys: 4.02 ms, total: 22.3 ms
Wall time: 473 ms<jupyter_text>The full bundle is not printed here. The following is a truncated example.```json{'checksums': [{'checksum': '5d8b77dd974e1b7c9de4040cbf9a24c7',   'type': 'md5'}], 'contents': [{'id': 'f2b7f3f7c123a38eb904c5412ce48757', 'name': 'SRX719457'},  {'id': '<KEY>', 'name': 'SRX719843'},  {'id': '<KEY>', 'name': 'SRX719844'},  {'id': '<KEY>', 'name': 'SRX719845'},  {'id': '<KEY>', 'name': 'SRX719846'},  {'id': '<KEY>', 'name': 'SRX719847'},  {'id': 'b488ab01ce3fa83addea057153ec449c', 'name': 'SRX719848'},  {'id': 'b3dd0d947f7e901bedf9f5789565ed07', 'name': 'SRX719849'},  {'id': 'a3bfebcf770157458454986092aeda62', 'name': 'SRX719850'},  {'id': '8165dc2b262ba94fdfd9a14bc7919fd4', 'name': 'SRX719851'},   ...], 'created_time': '2012-11-15T14:00:55Z', 'id': '<KEY>', 'name': 'SRP048601', 'self_url': 'drs://locate.md-be.ncbi.nlm.nih.gov/5d8b77dd974e1b7c9de4040cbf9a24c7', 'size': 87447929899239}``` The objects returned are still not physical files (i.e. a set of bytes) but an id for another logical concept. The 'experiment'.Calling DRS with the DRS id for the experiment.<jupyter_code>drsClient.getObject('16139c5b6f36034eb09768c17a90fd23')<jupyter_output><empty_output><jupyter_text>We are now down to a level where we get a single drs id, though it is still an id for a logical level entity - the Run. *Strictly what the DRS ids identify is the particular binary content of the file set corresponding to that logical id on a particular date . Essentially it's a version. But unlike GitHub this doesn't have the characteristics of a versioning system. Those characteristics probably aren't needed - the need supported by DRS is "give me the same set of bytes that I, or someone else, got for the same id previously", and that basic need is fulfilled. QED. What it does not support is give me the fileset for this thing that existed on such and such a date. It would not keep the file in sync with the state of the related data at the same point in time. That is an additional requirement not discussed further here, other than to comment that iti could be addressed in a higher level model that deals with versioning as a real world (logical) construct.Returning to the DRS id for the Run. This is familiar from the previous two examples.<jupyter_code>drsClient.getObject('fd074040842ce8c2e114b4eed7accee0')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08012326656394453, 'fraction_numerical': 0.05115562403697997, 'mean_word_length': 5.596205962059621, 'pattern_counts': {'":': 5, '<': 46, '<?xml version=': 0, '>': 46, 'https://': 7, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '67834', 'license': 'Apache-2.0', 'n_tokens_mistral': 3326, 'n_tokens_neox': 3024, 'n_words': 1173, 'path': 'notebooks/drs/SRA_IDs_and_bundling.ipynb', 'repo_name': 'jb-adams/fasp-scripts', 'size': '9893'}