<jupyter_start><jupyter_text>MAST Query Tutorial for NIRSpec Users  Learning Goals By the end of this tutorial, you will:    1) Be able to perform MAST queries for NIRSpec data     2) Understand MAST search options for NIRSpec MOS, IFU, and Fixed Slit data    3) Filter MAST queries by various parameters    4) Perform potential duplication checks for your targets  Introduction MAST (<NAME>ski Archive for Space Telescopes) provides a means of retrieving data through a variety of methods. The [MAST Portal](https://outerspace.stsci.edu/display/MASTDOCS/Portal+Guide) allows users to interactively search for particular observations and filter by keywords. An in-depth [tutorial](https://outerspace.stsci.edu/display/MASTDOCS/Portal+Guide) is available to aid users in accessing MAST in this manner. In some scenarios, particularly with large queries, it may be more efficient to use [Astroquery](https://astroquery.readthedocs.io/en/latest/mast/mast.html). Some broad capabilities including positional queries, metadata queries, and catalog queries are described in the documentation. General documentation on the [MAST API](https://mast.stsci.edu/api/v0/) is also available.One reason why users may want to search the archive is to find potential duplicatation observations for their proposals. As described in the [JWST Duplicate Observations Policy](https://jwst-docs.stsci.edu/jwst-opportunities-and-policies/jwst-general-science-policies/jwst-duplicate-observations-policy), an observation may be a duplication of another observation if it observes the same astronomical source or field with the same instrument as well as similar sensitivity and spectral range. More instrument-specific details can be found in the documentation. The following topics are covered in this notebook:1) How to submit a NIRSpec MAST Query using python    - How to perform a broad search for all NIRSpec Data    - How to perform a search using one query filter parameter    - How to perform a search using multiple query filter parameters    - Information on available search parameters2) How to perform checks for potential duplication issues with any given targets by comparing with pre-existing MAST Data (NOTE: This may not include scheduled observations, and results should be confirmed in APT)    - Define an input catalog describing a potential observation    - Query MAST within a search radius for observations with overlapping wavelengths and similar exposure times    - Examine potential for duplicate observationsAlthough this notebook is designed for NIRSpec users, other JWST instruments follow similar structures for MAST queries. More information that may be relevant to additional instruments can be found on the [Astroquery](https://astroquery.readthedocs.io/en/latest/mast/mast.html) web page.   Main Content<jupyter_code># Import MAST
from astroquery.mast import Mast<jupyter_output><empty_output><jupyter_text>Perform a search for all NIRSpec Data This first example demonstrates how to use [Mast.service_request_async()](https://astroquery.readthedocs.io/en/latest/api/astroquery.mast.MastClass.htmlastroquery.mast.MastClass.service_request_async), which builds and executes a [Mashup](https://mast.stsci.edu/api/v0/class_mashup_1_1_mashup_request.html) query based upon a service and parameters. More information and description of options can be found in the [documentation](https://mast.stsci.edu/api/v0/class_mashup_1_1_mashup_request.html). The response in this format provides a format which can list filenames.<jupyter_code># Set up the service for whichever instrument you want to query. Format with the first letter capitalized only.
service = "Mast.Jwst.Filtered.Nirspec"

# Enter query parameters, or leave empty (as shown here) to retrieve all data for that instrument.
# Leaving 'columns' as an asterisk includes all data columns. Replacing 'columns' with COUNT_BIG(*) will 
#     return the number of files in the search.
parameters = {"columns": "*",
               "filters": []}

# Retrieve the MAST response (this step may take a few seconds)
response = Mast.service_request_async(service, parameters)
# Gather the results
results = response[0].json()['data']
# Take the filename from each result if you prefer a list of filenames
filenames = []
for result in results:
    filename = result['filename'].split('.')[0]
    filenames.append(filename)
# Print the first ten filenames
print(filenames[:10])<jupyter_output>['jw84600007001_02101_00001_nrs1_rate', 'jw93125002001_02103_00001_nrs2_rate', 'jw84600041001_02101_00001_nrs2_uncal', 'jw95065009001_02104_00001_nrs2_rate', 'jw84600008001_02101_00002_nrs2_rate', 'jw96216006001_02101_00001_nrs2_dark', 'jw84600013001_02102_00001_nrs2_dark', 'jw95065009001_02103_00001_nrs1_rate', 'jw95065006001_02107_00001_nrs2_rate', 'jw95065009001_02101_00001_nrs2_rate']<jupyter_text>Use one query parameter:<jupyter_code># Enter one query parameter:
service = "Mast.Jwst.Filtered.Nirspec"
one_parameter = {"columns": "*",
               "filters": [{"paramName": "exp_type",
                            "values": ['NRS_MSASPEC']
                            }]
             }
response_one_parameter = Mast.service_request_async(service, one_parameter)
results_one_parameter = response_one_parameter[0].json()['data']
# Print the first ten filenames
for result_one_parameter in results_one_parameter[:10]:
    print(result_one_parameter['filename'].split('.')[0])<jupyter_output>jw93125002001_02103_00001_nrs2_rate
jw95065009001_02104_00001_nrs2_rate
jw84600008001_02101_00002_nrs2_rate
jw95065009001_02103_00001_nrs1_rate
jw95065006001_02107_00001_nrs2_rate
jw95065009001_02101_00001_nrs2_rate
jw95065009001_02101_00001_nrs1_rate
jw95065006001_02103_00001_nrs1_rate
jw95065006001_02108_00001_nrs1_rate
jw95065009001_02104_00001_nrs1_rate<jupyter_text>Use multiple query parameters:<jupyter_code># Use multiple query parameters
service = "Mast.Jwst.Filtered.Nirspec"
multiple_parameters = {"columns": "*",
               "filters": [{"paramName": "apername",
                            "values": ['NRS1_FULL', 'NRS_FULL_MSA']
                            },
                            {"paramName": "detector",
                             "values": ['NRS1']
                            },
                            {"paramName": "filter",
                             "values": ['F290LP', 'F170LP']
                            },
                            {"paramName": "exp_type",
                             "values": ['NRS_IFU', 'NRS_MSASPEC', 'NRS_BRIGHTOBJ','NRS_WATA']
                            },
                            {"paramName": "readpatt",
                             "values": ['NRS', 'NRSRAPID']
                            }
                            ]}
response_multiple_parameters = Mast.service_request_async(service, multiple_parameters)
results_multiple_parameters = response_multiple_parameters[0].json()['data']
# Print the first ten filenames
for result_multiple_parameters in results_multiple_parameters[:10]:
    # In addition to the filename, additional parameters can be printed. See below for a list of available options.
    print(result_multiple_parameters['filename'].split('.')[0] + '     ' + result_multiple_parameters['visitype'] )<jupyter_output>jw95065006001_02108_00001_nrs1_rate     PRIME_TARGETED_FIXED
jw95065009001_02104_00001_nrs1_rate     PRIME_TARGETED_FIXED
jw93125002001_02102_00001_nrs1_rate     PRIME_TARGETED_FIXED
jw93125002001_02101_00001_nrs1_rate     PRIME_TARGETED_FIXED
jw95065006001_02107_00001_nrs1_rate     PRIME_TARGETED_FIXED
jw95065009001_02106_00001_nrs1_rate     PRIME_TARGETED_FIXED
jw93125002001_02103_00001_nrs1_rate     PRIME_TARGETED_FIXED
jw93125002001_02105_00001_nrs1_rate     PRIME_TARGETED_FIXED
jw95065006001_02105_00001_nrs1_rate     PRIME_TARGETED_FIXED
jw93125002001_02104_00001_nrs1_rate     PRIME_TARGETED_FIXED<jupyter_text>A list of available filtering parameters in the Archive: The following parameters are available filters using the MAST query service. To use any of these filters, add the filter as the 'paramName' and the various options you would like to filter by as the 'values'.In addition to being used as filters, if you would like to return something other than the filename, any of these keys can also be used to access or return any component of the result. For instance, instead of printing the filenames with "result['filename']" as shown above, one could print the PI name by replacing this with "result['pi_name']".<jupyter_code># Dictionary keys of all of the NIRSpec data (without any filters):
all_keys = ""
for result in results:
    for key in result.keys():
        if key not in all_keys:
            all_keys = all_keys + key + ", "
print(all_keys[:-2]) # print all keys without the last comma<jupyter_output>ArchiveFileID, filename, fileSetName, productLevel, act_id, apername, asnpool, asntable, bartdelt, bendtime, bkgdtarg, bkglevel, bkgsub, bmidtime, bstrtime, category, cont_id, refframe, datamode, dataprob, date, date_mjd, date_end, date_end_mjd, date_obs, date_obs_mjd, detector, drpfrms1, drpfrms3, duration, dva_dec, dva_ra, effexptm, effinttm, eng_qual, eph_time, eph_type, exp_type, expcount, expend, expmid, exposure, expripar, expstart, fastaxis, filetype, filter, frmdivsr, gainfact, gdstarid, groupgap, gs_dec, gs_mag, gs_order, gs_ra, gsendtim, gsendtim_mjd, gsstrttm, gsstrttm_mjd, gs_udec, gs_umag, gs_ura, helidelt, hendtime, hga_move, hmidtime, hstrtime, instrume, intarget, is_psf, jwst_dx, jwst_dy, jwst_dz, jwst_x, jwst_y, jwst_z, lamp, mu_dec, mu_epoch, mu_epoch_mjd, mu_ra, nexposur, nextend, nframes, ngroups, nints, nresets, nrststrt, nsamples, numdthpt, nwfsest, obs_id, observtn, obslabel, origin, pcs_mode, pi_name, pps_aper, prd_ver, program, prop_dec, prop_ra, pwfseet, readp[...]<jupyter_text>Note that some of the above parameters are specific to a particular observation mode. For instance, 'msaconid', 'msametfl', 'msametid', and 'msastate' correspond to MOS observations.<jupyter_code># All options for a given parameter (eg, exp_type):
exp_type_list = []
for x in results:
    exp_type_list.append(x['exp_type']) if x['exp_type'] not in exp_type_list else exp_type_list
for exp_type in exp_type_list:
    print(exp_type)<jupyter_output>NRS_IFU
NRS_MSASPEC
NRS_BRIGHTOBJ
NRS_DARK
NRS_AUTOWAVE
NRS_LAMP
NRS_AUTOFLAT
NRS_IMAGE
NRS_CONFIRM
NRS_FIXEDSLIT
NRS_MIMF
NRS_FOCUS
NRS_TACONFIRM
NRS_WATA
NRS_MSATA<jupyter_text>Potential Duplication Checks Duplication for NIRSpec might occur if there is a target within the duplication search radius that not only uses the same grating and disperser but also has an exposure time within a factor of four of that of the previous observation. In wide field imaging, the field overlap must be greater than 50% to be considered a duplicate. More information on [identifying potential duplicate observations](https://jwst-docs.stsci.edu/jwst-duplication-checking/identifying-potential-duplicate-observations) might be useful for observers.In this demonstration, an input catalog is used to search an area with a radius equal to the longest side of the aperture, query for those observations matching the wavelength and exposure time criteria, and find sources in the archive which might be duplications for any target in the catalog.While the [Mast.service_request_async()](https://astroquery.readthedocs.io/en/latest/api/astroquery.mast.MastClass.htmlastroquery.mast.MastClass.service_request_async) shown above is a useful method for filtering data with metadata parameters, the [Observations.query_criteria()](https://astroquery.readthedocs.io/en/latest/mast/mast.htmlobservation-criteria-queries) is used in this case to search a particular region of the sky with additional parameters. Duplication search radii are dependent on aperture sizes, which are described in the [NIRSpec Fixed Slits JDox page](https://jwst-docs.stsci.edu/near-infrared-spectrograph/nirspec-instrumentation/nirspec-fixed-slits:~:text=NIRSpec%20has%20five%20fixed%20slits,series%20(BOTS)%20observing%20mode). The longest side of the slit is used for the fixed slit duplication search radius. The following cell contains duplication search radii for each NIRSpec aperture in degrees.<jupyter_code># Default duplication search radii
mos_radius = 180/360  # arcseconds/360 to get degrees
mos_slit_radius = 0.46/360 # arcseconds/360 to get degrees
ifu_radius = 3/360  # arcseconds/360 to get degrees
fs_S1600A1_radius = 1.6/360  # arcseconds/360 to get degrees
fs_S200A1_radius = 3.3/360  # arcseconds/360 to get degrees
fs_S200A2_radius = 3.3/360  # arcseconds/360 to get degrees
fs_S400A1_radius = 3.8/360  # arcseconds/360 to get degrees
fs_S200B1_radius = 3.3/360  # arcseconds/360 to get degrees<jupyter_output><empty_output><jupyter_text>The search area to use in the potential duplicate observation search is determined by the longest side of the aperture as defined above.<jupyter_code>search_area = {"MSA": mos_radius, "IFU": ifu_radius, "S1600A1": fs_S1600A1_radius, "S200A1": fs_S200A1_radius,
               "S200A2": fs_S200A2_radius, "S400A1": fs_S400A1_radius, "S200B1": fs_S200B1_radius}<jupyter_output><empty_output><jupyter_text>The value labeled 'instrument_name' in this case includes both the instrument and mode. This dictionary is used to determine the appropriate name to use in the query.<jupyter_code>instrument_name = {"MSA": "NIRSPEC/MSA", "IFU": "NIRSPEC/IFU", "S1600A1": "NIRSPEC/SLIT", "S200A1": "NIRSPEC/SLIT",
                   "S200A2": "NIRSPEC/SLIT", "S400A1": "NIRSPEC/SLIT", "S200B1": "NIRSPEC/SLIT"}<jupyter_output><empty_output><jupyter_text>Setup input catalog containing the target catalog number, aperture, RA, DEC, grating, filter, exposure time, search area, and instrument/mode name for each target. This format was chosen so that additional targets can easily be added to the input catalog list. Although these are the primary considerations for duplication checking, additional criteria can also be added using this dictionary format. It may be useful to check which [observation fields](https://mast.stsci.edu/api/v0/_c_a_o_mfields.html) are available for filtering NIRSpec data using Astroquery.Note that NIRSpec FS or IFU spectroscopic observations can be considered duplications of MOS observations of the same astronomical target with similar wavelength coverage and resolution. Thus, it may be helpful to query for MOS observations as well when searching for FS or IFU observation potential duplicates.<jupyter_code># Input Catalog to use in query

target1_number = "1"
target1_aperture = "MSA"
target1_RA = 53.13
target1_DEC = -27.8
target1_grating = "G395H"
target1_filters = "F290LP"
target1_exposure_time = 950
target1_search_area = search_area[target1_aperture]
target1_instrument_name = instrument_name[target1_aperture]
target1 = {'number': target1_number, 'RA': target1_RA, 'DEC': target1_DEC, 'grating': target1_grating, 
           'filters': target1_filters, 'exposure_time': target1_exposure_time, 'search_area': target1_search_area,
           'instrument_name': target1_instrument_name}

target2_number = "2"
target2_aperture = "IFU"
target2_RA = 68.73091
target2_DEC = 24.48140
target2_grating = "G395H"
target2_filters = "F290LP"
target2_exposure_time = 22.749
target2_search_area = search_area[target2_aperture]
target2_instrument_name = instrument_name[target2_aperture]
target2 = {'number': target2_number, 'RA': target2_RA, 'DEC': target2_DEC, 'grating': target2_grating,
           'filters': target2_filters, 'exposure_time': target2_exposure_time, 'search_area': target2_search_area,
           'instrument_name': target2_instrument_name} 
           
target3_number = "3"
target3_aperture = "S200B1"
target3_RA = 5.130
target3_DEC = -36.895
target3_grating = "G140H"
target3_filters = "F070LP"
target3_exposure_time = 1880.22
target3_search_area = search_area[target3_aperture]
target3_instrument_name = instrument_name[target3_aperture]
target3 = {'number': target3_number, 'RA': target3_RA, 'DEC': target3_DEC, 'grating': target3_grating,
           'filters': target3_filters, 'exposure_time': target3_exposure_time, 'search_area': target3_search_area,
           'instrument_name': target3_instrument_name} 

input_catalog = [target1, target2, target3]<jupyter_output><empty_output><jupyter_text>Import [MAST's Observations package](https://astroquery.readthedocs.io/en/latest/mast/mast.htmlobservation-criteria-queries) in order to query based on particular criteria other than position or target name. This package also allows you to search by 'proposal_pi' or other [observation fields](https://mast.stsci.edu/api/v0/_c_a_o_mfields.html). In the case of duplication checking, this query format is particularly useful because it allows users to search within a minimum and maximum RA and Dec value in addition to other parameters such as filters and gratings.<jupyter_code>from astroquery.mast import Observations
import numpy as np<jupyter_output><empty_output><jupyter_text>Next, a dictionary is generated using the target's catalog number as a key which contains observation tables for each target in the input catalog.<jupyter_code># Generate dictionary of observation tables
table = {}
for target in input_catalog:  # Loop through all targets in catalog
    key=target['number']
    # Perform query using information in the input catalog
    table[key] = Observations.query_criteria(instrument_name = target['instrument_name'],
                     s_ra=[(target['RA']-target['search_area']),(target['RA']+target['search_area'])],
                     s_dec=[target['DEC']-target['search_area'],target['DEC']+target['search_area']],
                     grating=target['grating'],
                     filters=target['filters'],
                     t_exptime=[(target['exposure_time'])/4,(target['exposure_time'])*4])
    # Perform query for MOS data since FS and IFU observations can also be duplicates of MOS observations
    if target['instrument_name'] != "NIRSPEC/MSA":
        table[key+"MOS"] = Observations.query_criteria(instrument_name = "NIRSPEC/MSA",
                         s_ra=[target['RA']-target['search_area'],target['RA']+target['search_area']],
                         s_dec=[target['DEC']-target['search_area'],target['DEC']+target['search_area']],
                         grating=target['grating'],
                         filters=target['filters'],
                         t_exptime=[(target['exposure_time'])/4,(target['exposure_time'])*4]
                         )
    print("""The observation table below contains potential duplicate observations for 
          target {} in the input catalog.""".format(key))
    print(table[key][:5])  # Remove the [:5] indexing to display all results
    print()<jupyter_output>WARNING: InputWarning: Filter grating does not exist. This filter will be skipped. [astroquery.mast.core]<jupyter_text>Because it is important to check these observations in APT to determine more robustly whether or not these potential duplicate observations would indeed be considered duplications, this tool can be used to retrieve the proposal IDs that can then be searched in APT.<jupyter_code># Print a list of the proposal IDs generated in the full table
proposal_ids=[]
for target in input_catalog:  # Go through every target in the input catalog
    for proposal_id in table[target['number']]['proposal_id']:  # Loop through proposal IDs for each observation
        if int(proposal_id) not in proposal_ids:  # Remove duplicates
            proposal_ids.append(int(proposal_id))
print("It may be helpful to check these proposal IDs in APT to compare proposal details:", proposal_ids)
# Get list of Product Group ID (obsid rather than Observation ID / obs_id)
obs_ids=[]
for target in input_catalog:  # Go through every target in the input catalog
    for obs_id in table[target['number']]['obsid']:  # Loop through proposal IDs for each observation
        if obs_id not in obs_ids:  # Remove duplicates
            obs_ids.append(obs_id)
print('Product group IDs:', obs_ids[:10])
# Download data products based on a Product Group ID such as '2003839997'
# Data will download to a folder named "mastDownload" in present working directory unless a download directory is given as a parameter.
# Warning: This cell may take a few minutes to run.
product_group_ID = '2003839997'
manifest = Observations.download_products(product_group_ID, download_dir=None)<jupyter_output>Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:HLA/url/cgi-bin/getdata.cgi?download=1&filename=hst_11635_11_wfc3_uvis_f469n_daophot_trm.cat to ./mastDownload/HLA/hst_11635_11_wfc3_uvis_f469n/hst_11635_11_wfc3_uvis_f469n_daophot_trm.cat ... [Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:HLA/url/cgi-bin/getdata.cgi?download=1&filename=hst_11635_11_wfc3_uvis_f469n_sexphot_trm.cat to ./mastDownload/HLA/hst_11635_11_wfc3_uvis_f469n/hst_11635_11_wfc3_uvis_f469n_sexphot_trm.cat ... [Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:HLA/url/cgi-bin/preview.cgi?dataset=hst_11635_11_wfc3_uvis_f469n to ./mastDownload/HLA/hst_11635_11_wfc3_uvis_f469n/hst_11635_11_wfc3_uvis_f469n_drz.jpg ... [Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:HLA/url/cgi-bin/getdata.cgi?dataset=hst_11635_11_wfc3_uvis_f469n_drz.fits to ./mastDownload/HLA/hst_11635_11_wfc3_uvis_f469n/hst_11635_11_wfc3_uvis_f469[...]
{'fraction_non_alphanumeric': 0.08263422818791946, 'fraction_numerical': 0.05429716629381059, 'mean_word_length': 4.8770199945220485, 'pattern_counts': {'":': 33, '<': 47, '<?xml version=': 0, '>': 47, 'https://': 20, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 20, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '318949', 'license': 'Apache-2.0', 'n_tokens_mistral': 7498, 'n_tokens_neox': 6622, 'n_words': 1938, 'path': '_notebooks/NIRSpec_MAST_Query/NIRSpec_MAST_Query.ipynb', 'repo_name': 'TheRealZoidberg/fastpagesTest', 'size': '21466'}