<jupyter_start><jupyter_text>PDS federated API demo, Osiris-Rex OVIRS data visualization PART 2: FIND DATA YOU ARE INTERESTED INThe purpose of this notebook is to demostrate how the PDS web API can be used to access the PDS data for a scientific use case.The documention of the PDS web API is available on https://nasa-pds.github.io/pds-api/This notebook is available on https://github.com/NASA-PDS/pds-api-notebookIt is the 2nd part of 2 Use cases: - Part1: explore the collection - Part2: find specific the data you are interested in    WARNING: This notebook is a demo and not a real scientific use case. It might contain mistake in the way the data is used or displayed.<jupyter_code>from __future__ import print_function
from pprint import pprint
import time
import matplotlib.pyplot as plt
from astropy.io import fits<jupyter_output><empty_output><jupyter_text>The PDS API is accessed using a python client library documented on https://nasa-pds.github.io/pds-api-client/<jupyter_code>import pds.api_client as pds_api<jupyter_output><empty_output><jupyter_text>Use the PDS demo web API server  Connect to the demo server. See User Interface of the web API: https://pds-gamma.jpl.nasa.gov/api/swagger-ui.htmlNote:  this piece of code will be wrapped into a helper function so that 1 line will be enough to connect to the API using a default host<jupyter_code>configuration = pds_api.Configuration()

# demo server
configuration.host = 'https://pds-gamma.jpl.nasa.gov/api/'

api_client = pds_api.ApiClient(configuration)<jupyter_output><empty_output><jupyter_text>List requestable propertiesAvailable properties can be identified with request<jupyter_code>products_api = pds_api.ProductsApi(api_client)

api_response = products_api.products(only_summary=True)
print(f"Available properties are {', '.join(api_response.summary.properties[1:10])}...")<jupyter_output>Available properties are pds:Modification_Detail.pds:modification_date, ops:Label_File_Info.ops:md5_checksum, pds:Internal_Reference.pds:lidvid_reference, vid, insight:Observation_Information.insight:start_solar_longitude, pds:Observing_System_Component.pds:type, pds:Stream_Text.pds:description, msn:Mission_Information.msn:product_type_name, ops:Data_File_Info.ops:md5_checksum...<jupyter_text>Get observations around specific spot (lat=12, lon=24) with distance to bennu closer than 4 kmGet the result found in part1 by directly posting the search criteria to the APIWe can use `ProductsApi.products` with argument `q=`.The query syntax is described in the PDS API specification. It uses the following operators:- comparison: lt, le, ...- boolean: and, or, not- groups: (, )See https://github.com/NASA-PDS/pds-api/blob/master/docs/spec/pds-api-specification.mdquery-syntax for detailed syntax.We request the FITS data file url from the API in the `field=` argument.<jupyter_code>start_time = time.time() 

criteria = " ( ( ref_lid_target eq urn:nasa:pds:context:target:asteroid.101955_bennu )"
criteria += " and ( ref_lid_instrument eq urn:nasa:pds:context:instrument:ovirs.orex )"
criteria += " and ( orex:spatial.orex:target_range lt 4.0 )"
criteria += " and ( orex:spatial.orex:latitude ge 9.0 ) and ( orex:spatial.orex:latitude le 15.0 )"
criteria += " and ( orex:spatial.orex:longitude ge 21.0 ) and ( orex:spatial.orex:longitude le 27.0 ) )"

properties_of_interest = ['orex:spatial.orex:latitude', 'orex:spatial.orex:longitude', 'ref_lid_instrument', 'orex:spatial.orex:target_range', 'ops:Data_File_Info.ops:file_ref']

closer_products = products_api.products(q=criteria, fields=properties_of_interest)

elapsed = time.time() - start_time
print(f'retrieved {len(closer_products.data)} products in {elapsed} seconds')<jupyter_output>retrieved 4 products in 0.3078958988189697 seconds<jupyter_text>Structure of the reponseThe responses are JSON objects, as follow<jupyter_code>pprint(closer_products.data[0])<jupyter_output>{'description': None,
 'id': 'urn:nasa:pds:orex.ovirs:data_calibrated:20190321t204128s567_ovr_scil2.fits::1.0',
 'investigations': [{'description': None,
                     'href': 'https://pds-gamma.jpl.nasa.gov/products/urn:nasa:pds:context:investigation:mission.orex',
                     'id': 'urn:nasa:pds:context:investigation:mission.orex',
                     'title': None,
                     'type': None}],
 'metadata': {'creation_date_time': '2019-10-23T19:05:39.752Z',
              'label_url': 'https://pds-gamma.jpl.nasa.gov/data/pds4/test-data/registry/orex.ovirs/data_calibrated/bennu_original_calibration/detailed_survey/20190321T204128S567_ovr_scil2.xml',
              'update_date_time': None,
              'version': '1.0'},
 'observing_system_components': [{'description': None,
                                  'href': 'https://pds-gamma.jpl.nasa.gov/products/urn:nasa:pds:context:instrument_host:spacecraft.orex',
                                  'id': 'urn:nasa:p[...]<jupyter_text>Check that we have the right lidvids<jupyter_code>lidvids = [product.id for product in closer_products.data]
print(f'The lidvids of the selected products are {lidvids}')<jupyter_output>The lidvids of the selected products are ['urn:nasa:pds:orex.ovirs:data_calibrated:20190321t204128s567_ovr_scil2.fits::1.0', 'urn:nasa:pds:orex.ovirs:data_calibrated:20190321t204129s468_ovr_scil2.fits::1.0', 'urn:nasa:pds:orex.ovirs:data_calibrated:20190321t204130s368_ovr_scil2.fits::1.0', 'urn:nasa:pds:orex.ovirs:data_calibrated:20190321t204131s368_ovr_scil2.fits::1.0']<jupyter_text>Extract the links to the data files<jupyter_code>data_files = [product.properties['ops:Data_File_Info.ops:file_ref'] for product in closer_products.data]
print(f'product data file url: {data_files}')<jupyter_output>product data file url: ['https://pds-gamma.jpl.nasa.gov/data/pds4/test-data/registry/orex.ovirs/data_calibrated/bennu_original_calibration/detailed_survey/20190321T204128S567_ovr_scil2.fits', 'https://pds-gamma.jpl.nasa.gov/data/pds4/test-data/registry/orex.ovirs/data_calibrated/bennu_original_calibration/detailed_survey/20190321T204129S468_ovr_scil2.fits', 'https://pds-gamma.jpl.nasa.gov/data/pds4/test-data/registry/orex.ovirs/data_calibrated/bennu_original_calibration/detailed_survey/20190321T204130S368_ovr_scil2.fits', 'https://pds-gamma.jpl.nasa.gov/data/pds4/test-data/registry/orex.ovirs/data_calibrated/bennu_original_calibration/detailed_survey/20190321T204131S368_ovr_scil2.fits']<jupyter_text>Plot the data (FITS files)Plot the spectra of the 4 observations on the same figure, one figure per dimension of the instrument.<jupyter_code>hduls = [fits.open(data_file) for data_file in data_files]

for i in range(20): # for each dimension of the instrument
    fig, ax = plt.subplots()
    ax.set_title(f'detector {i}')
    for hdul in hduls: # for each observation
        ax.plot(hdul[2].data[0, i, :], hdul[0].data[0, i, :])
    ax.set_xlabel('wavelength (micrometers)')
    ax.set_ylabel('radiance')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11333237451459802, 'fraction_numerical': 0.039695095642168846, 'mean_word_length': 5.622857142857143, 'pattern_counts': {'":': 0, '<': 32, '<?xml version=': 0, '>': 32, 'https://': 13, 'lorem ipsum': 0, 'www.': 0, 'xml': 1}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '66592', 'license': 'Apache-2.0', 'n_tokens_mistral': 2643, 'n_tokens_neox': 2425, 'n_words': 490, 'path': 'notebooks/ovirs/part2/find-data.ipynb', 'repo_name': 'NASA-PDS/pds-api-notebook', 'size': '6953'}