<jupyter_start><jupyter_text>Using DOS to download protected data This example shows how DOS can create an interoperability layer to work with data in `indexd`. As we will see, indexd works with fence to provide the credentials to perform URL signing.  Accessing metadata from indexd A lambda has been set up to point at dev.bionimbus.org. Let's get some DataObjects from it.<jupyter_code>from ga4gh.dos.client import Client
client = Client("https://mkc9oddwq0.execute-api.us-west-2.amazonaws.com/api", config={'validate_responses': False})
local_client = client.client
models = client.models<jupyter_output><empty_output><jupyter_text>Now that we've set up the client we can access data using `ListDataObjects`.<jupyter_code>ListDataObjectsRequest = models.get_model('ListDataObjectsRequest')
data_objects = local_client.ListDataObjects(body=ListDataObjectsRequest(page_size=100)).result().data_objects
print("Returned {} data objects.".format(len(data_objects)))<jupyter_output>Returned 3 data objects.<jupyter_text>Downloading data These Data Objects point to s3 addresses.<jupyter_code>data_object = local_client.GetDataObject(data_object_id=data_objects[2].id).result().data_object
print(data_object)<jupyter_output>Data Object: a file, API or other resource(aliases=None, checksums=[Checksum(checksum=u'73d643ec3f4beb9020eef0beed440ad0', type=u'md5')], created=datetime.datetime(2018, 2, 26, 23, 36, 34, 899360), description=None, id=u'd8581e97-bc68-49b0-b2ea-6008950fdb36', mime_type=None, name=u'testdata1', size=9L, updated=datetime.datetime(2018, 2, 26, 23, 36, 34, 899371), urls=[URL(system_metadata=SystemMetadata(baseid=u'c917ab25-b773-4ffc-8b8c-cc1edafca1f7', created_date=u'2018-02-26T23:36:34.899360', did=u'd8581e97-bc68-49b0-b2ea-6008950fdb36', file_name=u'testdata1', form=u'object', hashes={u'md5': u'73d643ec3f4beb9020eef0beed440ad0'}, metadata={u'acls': u'acct'}, rev=u'b21b1b26', size=9, updated_date=u'2018-02-26T23:36:34.899371', urls=[u's3://cdis-presigned-url-test/testdata'], version=None), url=u's3://cdis-presigned-url-test/testdata', user_metadata=UserMetadata(acls=u'acct'))], version=u'b21b1b26')<jupyter_text>Ordinarily these data will only be accessible with a third party client. If the data are in public buckets with requester pays, specially formatted URLs may be available.<jupyter_code>print(data_object.urls[0].url)<jupyter_output>s3://cdis-presigned-url-test/testdata<jupyter_text>Logging in to sign a URL In `fence` my email `<EMAIL>` has been granted access to one of the files for demonstration. To get the signed URL, we need to get a `fence_session` token. Please consider this a preliminary demonstration of crossing auth domains. First, we must access the google login for `bionimbus.org`.Clicking this URL will take us to the bionimbus login process.https://dev.planx-pla.net/On successful authentication we are redirected to bionimbus, where we can generate an API key and download the resulting `credentials.json`.  Load the API Key<jupyter_code>import json

with open("credentials.json","r") as json_file:
     credentials = json.load(json_file)<jupyter_output><empty_output><jupyter_text>Get a token from fence<jupyter_code>import requests
token = requests.post('https://dev.planx-pla.net/user/credentials/cdis/access_token', json=credentials).json()
print(token['access_token'][0:10])<jupyter_output>eyJhbGciOi<jupyter_text>We can now use this token to get signed URLs, in addition to the cloud native URLs.  Using auth with the DOS client<jupyter_code>response = local_client.GetDataObject(data_object_id="d8581e97-bc68-49b0-b2ea-6008950fdb36", _request_options={"headers": {'Authorization': 'bearer {}'.format(token['access_token'])}}).result()
data_object = response.data_object
print(data_object.urls[0].url)
print(data_object.urls[1].url)
print(data_object.id)<jupyter_output>s3://cdis-presigned-url-test/testdata
https://cdis-presigned-url-test.s3.amazonaws.com/testdata?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIEXU2UGJYLJ3ASDA%2F20180228%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20180228T021206Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&user_id=32&username=davidcs%40ucsc.edu&X-Amz-Signature=94dba66fa3bfa4259a176f850e675c707d48fa7a6d25a4c571a9cb11b2d57f2c
d8581e97-bc68-49b0-b2ea-6008950fdb36<jupyter_text>One can then download from this second URL using wget.<jupyter_code>signed_url = data_object.urls[1].url
!wget '$signed_url' -O out
!cat out<jupyter_output>--2018-02-27 18:14:47--  https://cdis-presigned-url-test.s3.amazonaws.com/testdata?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AK<KEY>2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20180228T021206Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&user_id=32&username=<EMAIL>%40<EMAIL>&X-Amz-Signature=94dba66fa3bfa4259a176f850e675c707d48fa7a6d25a4c571a9cb11b2d57f2c
Resolving cdis-presigned-url-test.s3.amazonaws.com (cdis-presigned-url-test.s3.amazonaws.com)... 172.16.58.3
Connecting to cdis-presigned-url-test.s3.amazonaws.com (cdis-presigned-url-test.s3.amazonaws.com)|172.16.58.3|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 40 [binary/octet-stream]
Saving to: ‘out’

out                 100%[===================>]      40  --.-KB/s    in 0s      

2018-02-27 18:14:47 (1.07 MB/s) - ‘out’ saved [40/40]

Hi Zac!
cdis-data-client uploaded this!
{'fraction_non_alphanumeric': 0.12953562699755594, 'fraction_numerical': 0.08779845835683399, 'mean_word_length': 7.0120481927710845, 'pattern_counts': {'":': 1, '<': 31, '<?xml version=': 0, '>': 32, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '162825', 'license': 'MIT', 'n_tokens_mistral': 2242, 'n_tokens_neox': 1993, 'n_words': 365, 'path': 'dos_indexd_signed_url.ipynb', 'repo_name': 'david4096/dos-indexd-lambda', 'size': '5356'}