<jupyter_start><jupyter_text>Data that DRS might provide about a fileUse case: Provide ~~meta~~data* about a file identified by a given DRS id.The realistic scenarios for this use case should be identified. Notably, when does this scenario occur, compared with the scenario where file ids are obtained from some other data driven query which reflects user interest?Key points are that* Search and other APIs already provide data relevant to answering that question* Would expect to use the same schemas that those APIs use * Reusing those schema provides rich functionalityi.e. rather than DRS determining what metadata should be provided about a file we should tap into the existing schema that are available.*Note: there is a conscious intent here to avoid the term metadata. For one thing we need to be more specific about which use cases and what data would be useful.  Query to get some DRS ids<jupyter_code>from fasp.search import DataConnectClient
from fasp.loc import crdcDRSClient

import json
# TCGA Query - CRDC
searchClient = DataConnectClient('https://ga4gh-search-adapter-presto-public.prod.dnastack.com/')

query = """
    SELECT  file_id drs_id
    FROM search_cloud.cshcodeathon.gdc_rel24_filedata_active 
    where data_format = 'BAM'
    and project_disease_type = 'Breast Invasive Carcinoma'
    limit 1"""
res = searchClient.runQuery(query)<jupyter_output>_Retrieving the query_
____Page1_______________
____Page2_______________
____Page3_______________
____Page4_______________
____Page5_______________<jupyter_text>An example of basic data about a file.The data returned is the same data used in the original search. e.g. the data_format 'BAM' returned below is exactly that which is in the query.<jupyter_code>drsClient = crdcDRSClient()
for id in res:
    print(json.dumps(drsClient.getFileData(id[0]), indent=3))
    print('_'*80)<jupyter_output>{
   "id": "030e5e74-6461-4f05-a399-de8e470bc056",
   "data_format": "BAM",
   "access": "controlled",
   "file_name": "46db33a7f2003837e88d0a81b8ebec2c_gdc_realn.bam",
   "data_type": "Aligned Reads",
   "data_category": "Sequencing Reads",
   "type": "aligned_reads",
   "experimental_strategy": "WXS",
   "platform": "Illumina",
   "created_datetime": "2016-05-03T00:35:52.946132-05:00",
   "file_size": 23894757370
}
________________________________________________________________________________<jupyter_text>The data can be retrieved through Search API itself<jupyter_code>query = """
    SELECT * 
    FROM search_cloud.cshcodeathon.gdc_rel24_filedata_active 
    where file_id = '030e5e74-6461-4f05-a399-de8e470bc056' """
res2 = searchClient.runQuery(query, returnType='json')
print(json.dumps(res2,indent=3))<jupyter_output>_Retrieving the query_
____Page1_______________
____Page2_______________
____Page3_______________
____Page4_______________
____Page5_______________
[
   {
      "dbname": "active",
      "file_gdc_id": "030e5e74-6461-4f05-a399-de8e470bc056",
      "access": "controlled",
      "acl": "phs000178",
      "analysis_input_file_gdc_ids": null,
      "analysis_workflow_link": "https://github.com/NCI-GDC/cocleaning-cwl",
      "analysis_workflow_type": "BWA with Mark Duplicates and Cocleaning",
      "archive_gdc_id": null,
      "archive_revision": null,
      "archive_state": null,
      "archive_submitter_id": null,
      "associated_entities__case_gdc_id": "1b703058-e596-45bc-80fe-8b98d545c2e2",
      "associated_entities__entity_gdc_id": "7b0b60c7-5fa0-440e-937f-8d82119330d6",
      "associated_entities__entity_submitter_id": "TCGA-AR-A2LK-01A-11D-A17W-09",
      "associated_entities__entity_type": "aliquot",
      "case_gdc_id": "1b703058-e596-45bc-80fe-8b98d545c2e2",
      "project_dbg[...]<jupyter_text>Getting linked objects - Case, Indexes, Downstream Analyses, ...More detailed information is also available.Making use of existing schema means that a DRS id can be a jumping off point into the graph. A user could fulfill any use case that the graph allows. Tapping into the schemas provided by Search provides rich functionality with little additional effort.<jupyter_code>for id in res:
    fileData = drsClient.getFileData(id[0], expanded=True)
    print(json.dumps(fileData, indent=3))
    print('_'*80)<jupyter_output>{
   "id": "030e5e74-6461-4f05-a399-de8e470bc056",
   "data_format": "BAM",
   "access": "controlled",
   "cases": [
      {
         "case_id": "1b703058-e596-45bc-80fe-8b98d545c2e2"
      }
   ],
   "associated_entities": [
      {
         "entity_type": "aliquot",
         "entity_id": "7b0b60c7-5fa0-440e-937f-8d82119330d6"
      }
   ],
   "file_name": "46db33a7f2003837e88d0a81b8ebec2c_gdc_realn.bam",
   "data_category": "Sequencing Reads",
   "downstream_analyses": [
      {
         "analysis_id": "e05199a1-d6ad-4d31-a616-dcc6fb057216"
      },
      {
         "analysis_id": "d0cc7c6d-675c-401e-b751-60c68b4436e5"
      },
      {
         "analysis_id": "90e83350-f2cf-49bc-910e-5331a2cea795"
      },
      {
         "analysis_id": "702c782f-ce48-41c4-b7f2-c97ba26bc8a0"
      }
   ],
   "type": "aligned_reads",
   "analysis": {
      "analysis_id": "35eb6a6c-76d7-4568-ae0a-45734676c43e"
   },
   "platform": "Illumina",
   "created_datetime": "2016-05-03T00:35:52.946132-05:00",
[...]<jupyter_text>Getting attributes of related objects<jupyter_code>for id in res:
    fileData = drsClient.getFileData(id[0], linked=True)
    print(json.dumps(fileData, indent=3))
    print('_'*80)<jupyter_output>{
   "id": "030e5e74-6461-4f05-a399-de8e470bc056",
   "data_format": "BAM",
   "access": "controlled",
   "cases": [
      {
         "case_id": "1b703058-e596-45bc-80fe-8b98d545c2e2",
         "project": {
            "disease_type": "Breast Invasive Carcinoma"
         },
         "diagnoses": [
            {
               "days_to_recurrence": null,
               "morphology": "8520/3",
               "tumor_stage": "stage iii",
               "created_datetime": null,
               "tissue_or_organ_of_origin": "Breast, NOS",
               "primary_diagnosis": "Lobular carcinoma, NOS",
               "age_at_diagnosis": 22800,
               "classification_of_tumor": "not reported",
               "prior_malignancy": "no",
               "site_of_resection_or_biopsy": "Breast, NOS",
               "days_to_last_known_disease_status": null,
               "tumor_grade": "not reported",
               "progression_or_recurrence": "not reported"
            }
         ],
         [...]<jupyter_text>Using the related dataIn this example the file id for the related Index file can be retrieved through DRS.<jupyter_code>drsClient.getObject("18fb79bb-7259-41da-bd76-9dd9f8f84bfc")<jupyter_output><empty_output><jupyter_text>Using table info to find out about Age at Diagnosis<jupyter_code>schema = searchClient.listTableInfo('search_cloud.cshcodeathon.tcga_clinical_gdc_current')
schema.getCol('diag__age_at_diagnosis')<jupyter_output>{
   "format": "bigint",
   "type": "int",
   "$comment": "Age at the time of diagnosis expressed in number of days since birth."
}<jupyter_text>BetterIt would be an improvement if the unit were available in a distinct field as in this case from another source described in Search.<jupyter_code>schema2 = searchClient.listTableInfo('dbgap_demo.scr_gecco_susceptibility.subject_phenotypes_multi')
schema2.getCol('age')<jupyter_output>{
   "type": "number",
   "$comment": "UNIT 'Years'",
   "maximum": 98.0,
   "minimum": 37.0,
   "description": "Participant reference age"
}<jupyter_text>The example above comes from dbGaP which provides machine readable data dictiionaries which are transformed to Search schema.<jupyter_code>{'type': 'number',
    'unit' : 'Days',
    'description': 'Age at the time of diagnosis expressed in number of days since birth.'}<jupyter_output><empty_output><jupyter_text>Best?The definition can also use the Search schema capability to reference a resource of semantic standards.For the age at diagnosis in this example a full definition already exists in a Metadata Repository (MDR), in this case the Cancer Data Standards Repository - caDSR. Note that no new curation is required to do establish this connection. The link to the data element id exists in the source data and just needs to be passed through as part of the schema.A better column definition using the semantic resource would be as follows:<jupyter_code>ageColDefinition = {'type': 'number',
    'unit': 'Days',
    'description': 'Age at the time of diagnosis expressed in number of days since birth.',
        '$ref':'cadsr:2006657'}

print (ageColDefinition['$ref'])<jupyter_output>cadsr:2006657<jupyter_text>The ID above is a CURIE and can be resolved to a machine readable definition as follows:http://identifiers.org/cadsr:2006657 or http://n2t.net/cadsr:2006657As an example of how the linked semantic resource can be used, the following uses an example function to retrieve the details of the Common Data Element that has been used can be retrieved.The values contain linked resources such as the range of values the element may take (Value Domain) or the formal semantic Concept applied during curation of the semantic element.<jupyter_code>cdeDefinition = schema.getcaDSRDefinition(ageColDefinition['$ref'])
print(json.dumps(cdeDefinition, indent=3))<jupyter_output>{
   "dateCreated": "03-03-2003",
   "dateModified": "01-05-2021",
   "longName": "Diagnosis Age",
   "preferredDefinition": "Age at which a condition or disease was first diagnosed.",
   "preferredName": "DX_AGE",
   "publicID": "2006657",
   "registrationStatus": "Qualified",
   "version": "1.0",
   "valueDomain": "https://cadsrapi.nci.nih.gov/cadsrapi41/GetXML?query=ValueDomain&DataElement[@id=B7FF5F20-0AE9-3159-E034-0003BA12F5E7]&roleName=valueDomain",
   "dataElementConcept": "https://cadsrapi.nci.nih.gov/cadsrapi41/GetXML?query=DataElementConcept&DataElement[@id=B7FF5F20-0AE9-3159-E034-0003BA12F5E7]&roleName=dataElementConcept"
}
{'fraction_non_alphanumeric': 0.10630159049728206, 'fraction_numerical': 0.06351922689752365, 'mean_word_length': 4.329935622317596, 'pattern_counts': {'":': 86, '<': 36, '<?xml version=': 0, '>': 36, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '309452', 'license': 'Apache-2.0', 'n_tokens_mistral': 3670, 'n_tokens_neox': 3282, 'n_words': 818, 'path': 'notebooks/drs/DRS File Data.ipynb', 'repo_name': 'ga4gh/fasp-scripts', 'size': '9934'}