<jupyter_start><jupyter_text>Subsetting ICESat-2 Data with the NSIDC Subsetter How to Use the NSIDC Subsetter Example NotebookThis notebook illustrates the use of icepyx for subsetting ICESat-2 data ordered through the NSIDC DAAC. We'll show how to find out what subsetting options are available and how to specify the subsetting options for your order.For more information on using icepyx to find, order, and download data, see our complimentary [ICESat-2_DAAC_DataAccess_Example Notebook](https://github.com/icesat2py/icepyx/blob/master/doc/examples/ICESat-2_DAAC_DataAccess_Example.ipynb).Questions? Be sure to check out the FAQs throughout this notebook, indicated as italic headings. Credits* notebook by: <NAME> and <NAME>* some source material: [NSIDC Data Access Notebook](https://github.com/ICESAT-2HackWeek/ICESat2_hackweek_tutorials/tree/master/03_NSIDCDataAccess_Steiker) by <NAME> and <NAME>  _What is SUBSETTING anyway?_Anyone who's worked with geospatial data has probably encountered subsetting. Typically, we search for data wherever it is stored and download the chunks (aka granules, scenes, passes, swaths, etc.) that contain something we are interested in. Then, we have to extract from each chunk the pieces we actually want to analyze. Those pieces might be geospatial (i.e. an area of interest), temporal (i.e. certain months of a time series), and/or certain variables. This process of extracting the data we are going to use is called subsetting.In the case of ICESat-2 data coming from the NSIDC DAAC, we can do this subsetting step on the data prior to download, reducing our number of data processing steps and resulting in smaller, faster downloads and storage.  Import packages, including icepyx<jupyter_code>import icepyx as ipx

import numpy as np
import xarray as xr
import pandas as pd

import h5py
import os,json
from pprint import pprint<jupyter_output><empty_output><jupyter_text>Create a query object and log in to EarthdataFor this example, we'll be working with a sea ice dataset (ATL09) for an area along West Greenland (Disko Bay).<jupyter_code>region_a = ipx.Query('ATL09',[-55, 68, -48, 71],['2019-02-22','2019-02-28'], \
                           start_time='00:00:00', end_time='23:59:59')
region_a.earthdata_login('username','email')<jupyter_output><empty_output><jupyter_text>Discover Subsetting OptionsYou can see what subsetting options are available for a given dataset by calling `show_custom_options()`. The options are presented as a series of headings followed by available values in square brackets. Headings are:* **Subsetting Options**: whether or not temporal and spatial subsetting are available for the dataset* **Data File Formats (Reformatting Options)**: return the data in a format other than the native hdf5 (submitted as a key=value kwarg to `order_granules(format='NetCDF4-CF')`)* **Data File (Reformatting) Options Supporting Reprojection**: return the data in a reprojected reference frame. These will be available for gridded ICESat-2 L3B datasets.* **Data File (Reformatting) Options NOT Supporting Reprojection**: data file formats that cannot be delivered with reprojection* **Data Variables (also Subsettable)**: a dictionary of variable name keys and the paths to those variables available in the dataset<jupyter_code>region_a.show_custom_options(dictview=True)<jupyter_output><empty_output><jupyter_text>By default, spatial and temporal subsetting based on your initial inputs is applied to your order unless you specify `subset=False` to `order_granules()` or `download_granules()`. Additional subsetting options must be specified as keyword arguments to the order/download functions.Although some file format conversions and reprojections are possible using the `format`, `projection`,and `projection_parameters` keywords, the rest of this tutorial will focus on variable subsetting, which is provided with the `Coverage` keyword.  _Why do I have to provide spatial bounds to icepyx even if I don't use them to subset my data order?_Because they're still needed for the granule level search. Spatial inputs are usually required for any data search, on any platform, even if your search parameters cover the entire globe.The spatial information you provide is used to search the data repository and determine which granules might contain data over your area of interest. When you use that spatial information for subsetting, it's actually asking the NSIDC subsetter to extract the appropriate data from each granule. Thus, even if you set `subset=False` and download entire granules, you still need to provide some inputs on what geographic area you'd like data for.  About Data Variables in a query objectThere are two possible variable parameters associated with each ```query``` object.1. `order_vars`, which is for interacting with variables during data querying, ordering, and downloading activities. `order_vars.wanted` holds the user's list to be submitted to the NSIDC subsetter and download a smaller, reproducible dataset.2. `file_vars`, which is for interacting with variables associated with local files [not yet implemented].Each variables parameter (which is actually an associated Variables class object) has methods to:* get available variables, either available from the NSIDC or the file (`avail()` method/attribute).* append new variables to the wanted list (`append()` method).* remove variables from the wanted list (`remove()` method).Each variables instance also has a set of attributes, including `avail` and `wanted` to indicate the list of variables that is available (unmutable, or unchangeable, as it is based on the input dataset specifications or files) and the list of variables that the user would like extracted (updateable with the `append` and `remove` methods), respectively. We'll showcase the use of all of these methods and attributes below.  ICESat-2 data variablesICESat-2 data is natively stored in a nested file format called hdf5. Much like a directory-file system on a computer, each variable (file) has a unique path through the heirarchy (directories) within the file. Thus, some variables (e.g. `'latitude'`, `'longitude'`) have multiple paths (one for each of the six beams in most datasets). To increase readability, some display options (2 and 3, below) show the 200+ variable + path combinations as a dictionary where the keys are variable names and the values are the paths to that variable. Determine what variables are availableThere are multiple ways to get a complete list of available variables.1. `region_a.order_vars.avail`, a list of all valid path+variable strings2. `region_a.show_custom_options(dictview=True)`, all available subsetting options3. `region_a.order_vars.parse_var_list(region_a.order_vars.avail)`, a dictionary of variable:paths key:value pairs<jupyter_code>region_a.order_vars.avail()<jupyter_output><empty_output><jupyter_text>By passing the boolean `options=True` to the `avail` method, you can obtain lists of unique possible variable inputs (var_list inputs) and path subdirectory inputs (keyword_list and beam_list inputs) for your dataset. These can be helpful for building your wanted variable list.<jupyter_code>region_a.order_vars.avail(options=True)<jupyter_output><empty_output><jupyter_text>_Why not just download all the data and subset locally? What if I need more variables/granules?_Taking advantage of the NSIDC subsetter is a great way to reduce your download size and thus your download time and the amount of storage required, especially if you're storing your data locally during analysis. By downloading your data using `icepyx`, it is easy to go back and get additional data with the same, similar, or different parameters (e.g. you can keep the same spatial and temporal bounds but change the variable list). Related tools (e.g. [`captoolkit`](https://github.com/fspaolo/captoolkit)) will let you easily merge files if you're uncomfortable merging them during read-in for processing.  Building your wanted variable listNow that you know which variables and path components are available for your dataset, you need to build a list of the ones you'd like included in your dataset. There are several options for generating your initial list as well as modifying it, giving the user complete control over the list submitted.The options for building your initial list are:1. Use a default list for the dataset (not yet fully implemented across all datasets. Have a default variable list for your field/dataset? Submit a pull request or post it as an issue on GitHub!)2. Provide a list of variable names3. Provide a list of profiles/beams or other path keywords, where "keywords" are simply the unique subdirectory names contained in the full variable paths of the dataset. A full list of available keywords for the dataset is displayed in the error message upon entering `keyword_list=['']` into the `append` function (see below for an example) or by running `region_a.order_vars.avail(options=True)`, as aboveNote: all datasets have a short list of "mandatory" variables/paths (containing spacecraft orientation and time information needed to convert the data's `delta_time` to a readable datetime) that are automatically added to any built list. If you have any recommendations for other variables that should always be included (e.g. uncertainty information), please let us know!Examples of using each method to build and modify your wanted variable list are below.<jupyter_code>region_a.order_vars.wanted
region_a.order_vars.append(defaults=True)
pprint(region_a.order_vars.wanted)<jupyter_output><empty_output><jupyter_text>The keywords available for this dataset are shown in the error message upon entering a blank keyword_list, as seen in the next cell.<jupyter_code>region_a.order_vars.append(keyword_list=[''])<jupyter_output><empty_output><jupyter_text>Modifying your wanted variable listGenerating and modifying your variable request list, which is stored in `region_a.order_vars.wanted`, is controlled by the `append` and `remove` functions that operate on `region_a.order_vars.wanted`. The input options to `append` are as follows (the full documentation for this function can be found by executing `help(region_a.order_vars.append)`).* `defaults` (default False) - include the default variable list for your dataset (not yet fully implemented for all datasets; please submit your default variable list for inclusion!)* `var_list` (default None) - list of variables (entered as strings)* `beam_list` (default None) - list of beams/profiles (entered as strings)* `keyword_list` (default None) - list of keywords (entered as strings); use `keyword_list=['']` to obtain a list of available keywordsSimilarly, the options for `remove` are:* `all` (default False) - reset `region_a.order_vars.wanted` to None* `var_list` (as above)* `beam_list` (as above)* `keyword_list` (as above)<jupyter_code>region_a.order_vars.remove(all=True)
pprint(region_a.order_vars.wanted)<jupyter_output><empty_output><jupyter_text>ExamplesBelow are a series of examples to show how you can use `append` and `remove` to modify your wanted variable list. For clarity, `region_a.order_vars.wanted` is cleared at the start of many examples. However, multiple `append` and `remove` commands can be called in succession to build your wanted variable list (see Examples 3+)  Example 1: choose variablesAdd all `latitude` and `longitude` variables<jupyter_code>region_a.order_vars.append(var_list=['latitude','longitude'])
pprint(region_a.order_vars.wanted)<jupyter_output><empty_output><jupyter_text>Example 2: specify beams/profiles and variableAdd `latitude` for only `profile_1` and `profile_2`<jupyter_code>region_a.order_vars.remove(all=True)
pprint(region_a.order_vars.wanted)
var_dict = region_a.order_vars.append(beam_list=['profile_1','profile_2'], var_list=['latitude'])
pprint(region_a.order_vars.wanted)<jupyter_output><empty_output><jupyter_text>Example 3: add/remove selected beams+variablesAdd `latitude` for `profile_3` and remove it for `profile_2`<jupyter_code>region_a.order_vars.append(beam_list=['profile_3'],var_list=['latitude'])
region_a.order_vars.remove(beam_list=['profile_2'], var_list=['latitude'])
pprint(region_a.order_vars.wanted)<jupyter_output><empty_output><jupyter_text>Example 4: `keyword_list`Add `latitude` for all profiles and with keyword `low_rate`<jupyter_code>region_a.order_vars.append(var_list=['latitude'],keyword_list=['low_rate'])
pprint(region_a.order_vars.wanted)<jupyter_output><empty_output><jupyter_text>Example 5: target a specific variable + pathRemove `'profile_1/high_rate/latitude'` (but keep `'profile_3/high_rate/latitude'`)<jupyter_code>region_a.order_vars.remove(beam_list=['profile_1'], var_list=['latitude'], keyword_list=['high_rate'])
pprint(region_a.order_vars.wanted)<jupyter_output><empty_output><jupyter_text>Example 6: add variables not specific to beams/profilesAdd `rgt` under `orbit_info`.<jupyter_code>region_a.order_vars.append(keyword_list=['orbit_info'],var_list=['rgt'])
pprint(region_a.order_vars.wanted)<jupyter_output><empty_output><jupyter_text>Example 7: add all variables+paths of a groupIn addition to adding specific variables and paths, we can filter all variables with a specific keyword as well. Here, we add all variables under `orbit_info`. Note that paths already in `region_a.order_vars.wanted`, such as `'orbit_info/rgt'`, are not duplicated.<jupyter_code>region_a.order_vars.append(keyword_list=['orbit_info'])
pprint(region_a.order_vars.wanted)<jupyter_output><empty_output><jupyter_text>Example 8: add all possible values for variables+pathsAppend all `longitude` paths and all variables/paths with keyword `high_rate`.Simlarly to what is shown in Example 4, if you submit only one `append` call as `region_a.order_vars.append(var_list=['longitude'], keyword_list=['high_rate'])` rather than the two `append` calls shown below, you will only add the variable `longitude` and only paths containing `high_rate`, not ALL paths for `longitude` and ANY variables with `high_rate` in their path.<jupyter_code>region_a.order_vars.append(var_list=['longitude'])
region_a.order_vars.append(keyword_list=['high_rate'])
pprint(region_a.order_vars.wanted)<jupyter_output><empty_output><jupyter_text>Example 9: remove all variables+paths associated with a beamRemove all paths for `profile_1` and `profile_3`<jupyter_code>region_a.order_vars.remove(beam_list=['profile_1','profile_3'])
pprint(region_a.order_vars.wanted)<jupyter_output><empty_output><jupyter_text>Example 10: generate a default list for the rest of the tutorialGenerate a reasonable variable list prior to download<jupyter_code>region_a.order_vars.remove(all=True)
region_a.order_vars.append(defaults=True)
pprint(region_a.order_vars.wanted)<jupyter_output><empty_output><jupyter_text>Applying variable subsetting to your order and downloadIn order to have your wanted variable list included with your order, you must pass it as a keyword argument to the `subsetparams()` attribute or the `order_granules()` or `download_granules()` (which calls `order_granules` under the hood if you have not already placed your order) functions.<jupyter_code>region_a.subsetparams(Coverage=region_a.order_vars.wanted)<jupyter_output><empty_output><jupyter_text>Or, you can put the `Coverage` parameter directly into `order_granules`:`region_a.order_granules(Coverage=region_a.order_vars.wanted)`However, then you cannot view your subset parameters (`region_a.subsetparams`) prior to submitting your order.<jupyter_code>region_a.order_granules()# <-- you do not need to include the 'Coverage' kwarg to
                             # order if you have already included it in a call to subsetparams
region_a.download_granules('/home/jovyan/icepyx/dev-notebooks/vardata') # <-- you do not need to include the 'Coverage' kwarg to
                             # download if you have already submitted it with your order<jupyter_output><empty_output><jupyter_text>_Why does the subsetter say no matching data was found?_Sometimes, chunks (granules) returned in our initial search end up not containing any data in our specified area of interest. This is because the initial search is completed using summary metadata for a chunk. You've likely encountered this before when viewing available imagery online: your spatial search turns up a bunch of images with only a few border or corner pixels, maybe even in no data regions, in your area of interest. Thus, when you go to extract the data from the area you want (i.e. spatially subset it), you don't get any usable data from that image.  Check the variable list in your downloaded fileCompare the available variables associated with the full dataset relative to those in your downloaded data file.<jupyter_code># put the full filepath to a data file here. You can get this in JupyterHub by navigating to the file,
# right clicking, and selecting copy path. Then you can paste the path in the quotes below.
fn = ''<jupyter_output><empty_output><jupyter_text>Check the downloaded datasetGet all `latitude` variables in your downloaded file:<jupyter_code>varname = 'latitude'

varlist = []
def IS2h5walk(vname, h5node):
    if isinstance(h5node, h5py.Dataset):
        varlist.append(vname)
    return 

with h5py.File(fn,'r') as h5pt:
    h5pt.visititems(IS2h5walk)
    
for tvar in varlist:
    vpath,vn = os.path.split(tvar)
    if vn==varname: print(tvar)<jupyter_output><empty_output><jupyter_text>Compare to the variable paths available in the original data<jupyter_code>region_a.order_vars.parse_var_list(region_a.order_vars.avail)[0][varname]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0750283768444949, 'fraction_numerical': 0.006242905788876277, 'mean_word_length': 6.1600975213327915, 'pattern_counts': {'":': 0, '<': 99, '<?xml version=': 0, '>': 97, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 22, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '23', 'dir': 'jupyter-structured-clean-dedup', 'id': '209488', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 4975, 'n_tokens_neox': 4765, 'n_words': 2032, 'path': 'examples/ICESat-2_DAAC_DataAccess2_Subsetting.ipynb', 'repo_name': 'nabib/icepyx', 'size': '17643'}