<jupyter_start><jupyter_text>Accessing SnowEx Data at the NSIDC DAAC2021 SnowEx HackweekAuthor: <NAME>, NSIDC DAAC, CIRES, University of ColoradoThis tutorial provides a brief overview of the resources provided by the NASA National Snow and Ice Data Center Distributed Active Archive Center, or NSIDC DAAC, and demonstrates how to disscovery and access SnowEx data programmatically.   Learning Objectives:1. Identify resources available at the NSIDC DAAC to help you work with SnowEx data. 2. Search and discover file size and coverage of SnowEx data over a time and geojson region of interest.3. Set up Earthdata Login authentication needed to access NASA Earthdata.4. Download SnowEx data programmatically using the NSIDC DAAC Application Programming Interface (API). ___ Explore snow products and resources NSIDC introduction[The National Snow and Ice Data Center](https://nsidc.org), located in Boulder, Colorado, provides over 1100 data sets covering the Earth's cryosphere and more, all of which are available to the public free of charge. Beyond providing these data, NSIDC creates tools for data access, supports data users, performs scientific research, and educates the public about the cryosphere. The [NASA National Snow and Ice Data Center Distributed Active Archive Center (NSIDC DAAC)](https://nsidc.org/daac) provides access to over 700 data products (over 2 PB of data and growing!) in support of cryospheric research, global change detection, and water resource management. NSIDC is one of 12 DAACs as part of NASA’s EOSDIS, or Earth Observing System Data and Information System, and is also the largest program within the National Snow and Ice Data Center, as part of the University of Colorado’s Cooperative Institute for Research in Environmental Sciences.  Select Data Resources* [NSIDC Data Tutorials](https://github.com/nsidc/NSIDC-Data-Tutorials)    * Jupyter Notebook tutorials providing guidance on working with various NSIDC DAAC data products, including how to access, subset, transform, and visualize data.* [NSIDC Data Search](https://nsidc.org/data/search/keywords=snow)    * Search NSIDC snow data* [NSIDC Data Update Announcements](https://nsidc.org/the-drift/data-update/)     * News and tips for data users* [NASA Earthdata Search](http://search.earthdata.nasa.gov/)    * Search and access data across the NASA Earthdata* [NASA Worldview](https://worldview.earthdata.nasa.gov/)    * Interactive interface for browsing full-resolution, global, daily satellite images     Snow Today[Snow Today](https://nsidc.org/snow-today), a collaboration with the University of Colorado's Institute of Alpine and Arctic Research (INSTAAR), provides near-real-time snow analysis for the western United States and regular reports on conditions during the winter season. Snow Today is funded by NASA Hydrological Sciences Program and utilizes data from the Moderate Resolution Imaging Spectroradiometer (MODIS) instrument and snow station data from the Snow Telemetry (SNOTEL) network by the Natural Resources Conservation Service (NRCS), United States Department of Agriculture (USDA) and the California Department of Water Resources: www.wcc.nrcs.usda.gov/snow. NSIDC's SnowEx pages* [SnowEx](https://nsidc.org/data/snowex)    * We will be using the 'SnowEx17 Ground Penetrating Radar, Version 2' dataset in the following steps: https://doi.org/10.5067/G21LGCNLFSC5 Other relevant snow products:* [ASO Snow Data](https://nsidc.org/data/search/keywords=aso/sortKeys=score,,desc/facetFilters=%257B%2522facet_parameter%2522%253A%255B%2522SNOW%2520DEPTH%2522%252C%2522SNOW%2520WATER%2520EQUIVALENT%2522%255D%257D/pageNumber=1/itemsPerPage=25)* [MODIS Snow Data](https://nsidc.org/data/search/sortKeys=score,,desc/facetFilters=%257B%2522facet_parameter%2522%253A%255B%2522SNOW%2520COVER%2522%252C%2522Snow%2520Cover%2522%252C%2522SNOW%2520WATER%2520EQUIVALENT%2522%252C%2522SNOW%2520DEPTH%2522%252C%2522SNOW%2520MELT%2522%252C%2522SNOW%2520VOLUME%2520BUDGET%2522%252C%2522SNOW%252FICE%2522%255D%252C%2522facet_sensor%2522%253A%255B%2522%2520%257C%2520MODIS%2522%252C%2522Moderate-Resolution%2520Imaging%2520Spectroradiometer%2520%257C%2520MODIS%2522%255D%252C%2522facet_sponsored_program%2522%253A%255B%2522NASA%2520National%2520Snow%2520and%2520Ice%2520Data%2520Center%2520Distributed%2520Active%2520Archive%2520Center%2520%257C%2520NASA%2520NSIDC%2520DAAC%2522%255D%257D/pageNumber=1/itemsPerPage=25)* [VIIRS Snow Data](http://nsidc.org/data/search/sortKeys=score,,desc/facetFilters=%257B%2522facet_sensor%2522%253A%255B%2522Visible-Infrared%2520Imager-Radiometer%2520Suite%2520%257C%2520VIIRS%2522%255D%252C%2522facet_parameter%2522%253A%255B%2522SNOW%2520COVER%2522%252C%2522Snow%2520Cover%2522%255D%257D/pageNumber=1/itemsPerPage=25)* [AMSR-E and AMSR-E/AMSR2 Unified Snow Data](http://nsidc.org/data/search/sortKeys=score,,desc/facetFilters=%257B%2522facet_sensor%2522%253A%255B%2522Advanced%2520Microwave%2520Scanning%2520Radiometer-EOS%2520%257C%2520AMSR-E%2522%252C%2522Advanced%2520Microwave%2520Scanning%2520Radiometer%25202%2520%257C%2520AMSR2%2522%255D%252C%2522facet_parameter%2522%253A%255B%2522SNOW%2520WATER%2520EQUIVALENT%2522%252C%2522Snow%2520Water%2520Equivalent%2522%255D%257D/pageNumber=1/itemsPerPage=25)* [MEaSUREs Snow Data](http://nsidc.org/data/search/keywords=measures/sortKeys=score,,desc/facetFilters=%257B%2522facet_parameter%2522%253A%255B%2522SNOW%2520COVER%2522%255D%252C%2522facet_sponsored_program%2522%253A%255B%2522NASA%2520National%2520Snow%2520and%2520Ice%2520Data%2520Center%2520Distributed%2520Active%2520Archive%2520Center%2520%257C%2520NASA%2520NSIDC%2520DAAC%2522%255D%252C%2522facet_format%2522%253A%255B%2522NetCDF%2522%255D%252C%2522facet_temporal_duration%2522%253A%255B%252210%252B%2520years%2522%255D%257D/pageNumber=1/itemsPerPage=25)  * Near-Real-Time SSM/I-SSMIS EASE-Grid Daily Global Ice Concentration and Snow Extent (NISE), Version 5: https://doi.org/10.5067/3KB2JPLFPK3R ___ Import PackagesGet started by importing packages needed to run the following code blocks, including the `tutorial_helper_functions` module provided within this repository.<jupyter_code>import os
import geopandas as gpd
from shapely.geometry import Polygon, mapping
from shapely.geometry.polygon import orient
import pandas as pd 
import requests
import json
import pprint
import getpass
from netrc import netrc
from platform import system
from getpass import getpass
from urllib import request
from http.cookiejar import CookieJar
from os.path import join, expanduser
import requests
from xml.etree import ElementTree as ET
import time
import zipfile
import io
import shutil<jupyter_output><empty_output><jupyter_text>___ Data DiscoveryStart by identifying your study area and exploring coincident data over the same time and area.   Identify area and time of interestSince our focus is on the Grand Mesa study site of the NASA SnowEx campaign, we'll use that area to search for coincident data across other data products. From the [SnowEx17 Ground Penetrating Radar Version 2](https://doi.org/10.5067/G21LGCNLFSC5) landing page, you can find the rectangular spatial coverage under the Overview tab, or you can draw a polygon over your area of interest in the map under the Download Data tab and export the shape as a geojson file using the Export Polygon icon shown below. An example polygon geojson file is provided in the /Data folder of this repository.     Create polygon coordinate stringRead in the geojson file as a GeoDataFrame object and simplify and reorder using the shapely package. This will be converted back to a dictionary to be applied as our polygon search parameter.<jupyter_code>polygon_filepath = str(os.getcwd() + '/Data/nsidc-polygon.json') # Note: A shapefile or other vector-based spatial data format could be substituted here.

gdf = gpd.read_file(polygon_filepath) #Return a GeoDataFrame object

# Simplify polygon for complex shapes in order to pass a reasonable request length to CMR. The larger the tolerance value, the more simplified the polygon.
# Orient counter-clockwise: CMR polygon points need to be provided in counter-clockwise order. The last point should match the first point to close the polygon.
poly = orient(gdf.simplify(0.05, preserve_topology=False).loc[0],sign=1.0)

#Format dictionary to polygon coordinate pairs for CMR polygon filtering
polygon = ','.join([str(c) for xy in zip(*poly.exterior.coords.xy) for c in xy])
print('Polygon coordinates to be used in search:', polygon)
poly<jupyter_output>Polygon coordinates to be used in search: -108.2352445938561,38.98556907427165,-107.85284607930835,38.978765032966244,-107.85494925720668,39.10596902171742,-108.22772795408136,39.11294532581687,-108.2352445938561,38.98556907427165<jupyter_text>Alternatively: Specify bounding box region of interestInstead of using a vector shape file to specify a region of interest, you can simply use a bounding box. The following cell is commented out below, which can be used instead of the `polygon` search parameter.<jupyter_code># bounds = poly.bounds # Get polygon bounds to be used as bounding box input
# bounding_box = ','.join(map(str, list(bounds))) # Bounding Box spatial parameter in decimal degree 'W,S,E,N' format. 
# print(bounding_box)<jupyter_output><empty_output><jupyter_text>Set time rangeThis is an optional parameter; set this to specify a time range of interest. In this case we'll just select all of 2017 to ensure that we receive all files within this data set campaign.<jupyter_code>temporal = '2017-01-01T00:00:00Z,2017-12-31T23:59:59Z' # Set temporal range<jupyter_output><empty_output><jupyter_text>Create data parameter dictionary Create a dictionary with the data set shortname and version, as well as the temporal range and polygonal area of interest. Data set shortnames, or IDs, as well as version numbers, are located at the top of every NSIDC landing page.<jupyter_code>param_dict = {
    'short_name': 'SNEX17_GPR',
    'version': '2',
    'polygon': polygon,
#     'bounding_box': bounding_box, #optional alternative to polygon search parameter; if using, remove or comment out polygon search parameter
    'temporal':temporal,
}<jupyter_output><empty_output><jupyter_text>Determine how many files exist over this time and area of interest, as well as the average size and total volume of those filesWe will use the `granule_info` function to query metadata about each data set and associated files using the [Common Metadata Repository (CMR)](https://cmr.earthdata.nasa.gov/search/site/docs/search/api.html), which is a high-performance, high-quality, continuously evolving metadata system that catalogs Earth Science data and associated service metadata records. Note that not all NSIDC data can be searched at the file level using CMR, particularly those outside of the NASA DAAC program.<jupyter_code>def search_granules(search_parameters, geojson=None, output_format="json"):
    """
    Performs a granule search with token authentication for restricted results
    
    :search_parameters: dictionary of CMR search parameters
    :token: CMR token needed for restricted search
    :geojson: filepath to GeoJSON file for spatial search
    :output_format: select format for results https://cmr.earthdata.nasa.gov/search/site/docs/search/api.html#supported-result-formats
    
    :returns: if hits is greater than 0, search results are returned in chosen output_format, otherwise returns None.
    """
    search_url = "https://cmr.earthdata.nasa.gov/search/granules"

    
    if geojson:
        files = {"shapefile": (geojson, open(geojson, "r"), "application/geo+json")}
    else:
        files = None
    
    
    parameters = {
        "scroll": "true",
        "page_size": 100,
    }
    
    try:
        response = requests.post(f"{search_url}.{output_format}", params=parameters, data=search_parameters, files=files)
        response.raise_for_status()
    except HTTPError as http_err:
        print(f"HTTP Error: {http_error}")
    except Exception as err:
        print(f"Error: {err}")
    
    hits = int(response.headers['CMR-Hits'])
    if hits > 0:
        print(f"Found {hits} files")
        results = json.loads(response.content)
        granules = []
        granules.extend(results['feed']['entry'])
        granule_sizes = [float(granule['granule_size']) for granule in granules]
        print(f"The total size of all files is {sum(granule_sizes):.2f} MB")
        return response.json()
    else:
        print("Found no hits")
        return

search_granules(param_dict)<jupyter_output>Found 3 files
The total size of all files is 209.20 MB<jupyter_text>___ Data Access Option 1: Python script from the data set landing pageThe [Depths and Snow Pit Data Package Contents](https://snowex-hackweek.github.io/website/tutorials/core-datasets/02_data-package.html) tutorial demonstrates how to access data using the script provided under the "Download Data" tab of each NSIDC DAAC data set landing page.  Earthdata Login authentication setup notes1. All data with the NASA Earthdata system are freely available to the public, but requires an Earthdata Login profile to access. If you have not already done so, visit http://urs.earthdata.nasa.gov to register (it just takes a minute to set up). 2. Create a `.netrc` file within the JupyterHub environment. This is a hidden file typically stored in your home directory that contains your Earthdata login username and password. This is a secure and easy way to ensure that any data download requests are authenticated against your profile. You can set up your `.netrc` within the JupyterHub environment as copied from the preliminary work article [here](https://snowex-hackweek.github.io/website/preliminary/earthdata.htmlconfigure-programmatic-access-to-nasa-servers):Run the following commands on the JupyterHub in a terminal replacing your Earthdata login username and password:```echo "machine urs.earthdata.nasa.gov login EARTHDATA_LOGIN password EARTHDATA_PASSWORD" > ~/.netrcchmod 0600 .netrc```Note that the script below should prompt you with your Earthdata Login username and password if a `.netrc` file does not exist.<jupyter_code>%run './scripts/nsidc-download_SNEX20_SD.001.py' 
print('Grand Mesa 2020 Snow Depth data download complete')<jupyter_output>Querying for data:
	https://cmr.earthdata.nasa.gov/search/granules.json?provider=NSIDC_ECS&sort_key[]=start_date&sort_key[]=producer_granule_id&scroll=true&page_size=2000&short_name=SNEX20_SD&version=001&version=01&version=1&temporal[]=2020-01-28T00:00:00Z,2020-02-12T23:59:59Z

Found 1 matches.
Downloading 2 files...<jupyter_text>Option 2: Additional data access services: API AccessData can be accessed directly from our HTTPS file system as described in the aforementioned tutorial, or through the NSIDC DAAC's Application Programming Interface (API). What is an API? You can think of an API as a middle man between an application or end-use (in this case, us) and a data provider. Here, the data provider is both the Common Metadata Repository (CMR) housing data information, and NSIDC as the data distributor. These APIs are generally structured as a URL with a base plus individual key-value-pairs separated by ‘&’. This option is beneficial for those of you who want to incorporate data access directly into your visualization and analysis coding workflow, without the need to utilize the NSIDC website. This method is also reproducible and documented to ensure data provenance.This API offers you the ability to order data using specific temporal and spatial filters. These options can be requested in a single access command without the need to script against our data directory structure. See the [programmatic access guide](https://nsidc.org/support/how/how-do-i-programmatically-request-data-services) for more information on API options.   Create the data request API endpointProgrammatic API requests are formatted as HTTPS URLs that contain key-value-pairs specifying the service operations that we specified above. The cell below sets up the API request URL using our data search parameters as well as a few other configuration parameters. We will first create a string of key-value-pairs from our data dictionary and we'll feed those into our API endpoint. This API endpoint can be executed via command line, a web browser, or in Python below.<jupyter_code>#Set NSIDC data access base URL
base_url = 'https://n5eil02u.ecs.nsidc.org/egi/request'

#Set the request mode to asynchronous, "no" processing agent (no subsetting or reformatting services available), and optionally removing metadata delivery

param_dict['request_mode'] = 'async'
param_dict['agent'] = 'NO'
param_dict['INCLUDE_META'] ='N' #optional if you do not wish to receive the associated metadata files with each science file. 

param_string = '&'.join("{!s}={!r}".format(k,v) for (k,v) in param_dict.items()) # Convert param_dict to string
param_string = param_string.replace("'","") # Remove quotes

api_list = [f'{base_url}?{param_string}']
api_request = api_list[0]
print(api_request) # Print API base URL + request parameters<jupyter_output>https://n5eil02u.ecs.nsidc.org/egi/request?short_name=SNEX17_GPR&version=2&polygon=-108.2352445938561,38.98556907427165,-107.85284607930835,38.978765032966244,-107.85494925720668,39.10596902171742,-108.22772795408136,39.11294532581687,-108.2352445938561,38.98556907427165&temporal=2017-01-01T00:00:00Z,2017-12-31T23:59:59Z&request_mode=async&agent=NO&INCLUDE_META=N<jupyter_text>Input Earthdata Login credentialsFor our API access option, An Earthdata Login account is required to access data from the NSIDC DAAC. If you do not already have an Earthdata Login account, visit http://urs.earthdata.nasa.gov to register.<jupyter_code># Start authenticated session with Earthdata Login to allow for data downloads:

def setup_earthdata_login_auth(endpoint: str='urs.earthdata.nasa.gov'):
    netrc_name = "_netrc" if system()=="Windows" else ".netrc"
    try:
        username, _, password = netrc(file=join(expanduser('~'), netrc_name)).authenticators(endpoint)
    except (FileNotFoundError, TypeError):
        print('Please provide your Earthdata Login credentials for access.')
        print('Your info will only be passed to %s and will not be exposed in Jupyter.' % (endpoint))
        username = input('Username: ')
        password = getpass('Password: ')
    manager = request.HTTPPasswordMgrWithDefaultRealm()
    manager.add_password(None, endpoint, username, password)
    auth = request.HTTPBasicAuthHandler(manager)
    jar = CookieJar()
    processor = request.HTTPCookieProcessor(jar)
    opener = request.build_opener(auth, processor)
    request.install_opener(opener)

setup_earthdata_login_auth(endpoint="urs.earthdata.nasa.gov")<jupyter_output>Please provide your Earthdata Login credentials for access.
Your info will only be passed to urs.earthdata.nasa.gov and will not be exposed in Jupyter.<jupyter_text>Download dataDownload data using the `requests` library. The data will be downloaded directly to this directory in a new Outputs folder. The progress of each order will be reported.<jupyter_code>def request_nsidc_data(API_request):
    """
    Performs a data customization and access request from NSIDC's API/
    Creates an output folder in the working directory if one does not already exist.
    
    :API_request: NSIDC API endpoint; see https://nsidc.org/support/how/how-do-i-programmatically-request-data-services for more info
    on how to configure the API request.
    
    """

    path = str(os.getcwd() + '/Outputs') # Create an output folder if the folder does not already exist.
    if not os.path.exists(path):
        os.mkdir(path)
        
    base_url = 'https://n5eil02u.ecs.nsidc.org/egi/request'

    
    r = request.urlopen(API_request)
    esir_root = ET.fromstring(r.read())
    orderlist = []   # Look up order ID
    for order in esir_root.findall("./order/"):
        orderlist.append(order.text)
    orderID = orderlist[0]
    statusURL = base_url + '/' + orderID # Create status URL
    print('Order status URL: ', statusURL)
    request_response = request.urlopen(statusURL) # Find order status  
    request_root = ET.fromstring(request_response.read())
    statuslist = []
    for status in request_root.findall("./requestStatus/"):
        statuslist.append(status.text)
    status = statuslist[0]
    while status == 'pending' or status == 'processing': #Continue loop while request is still processing
        print('Job status is ', status,'. Trying again.')
        time.sleep(10)
        loop_response = request.urlopen(statusURL)
        loop_root = ET.fromstring(loop_response.read())
        statuslist = [] #find status
        for status in loop_root.findall("./requestStatus/"):
            statuslist.append(status.text)
        status = statuslist[0]
        if status == 'pending' or status == 'processing':
            continue
    if status == 'complete_with_errors' or status == 'failed': # Provide complete_with_errors error message:
        messagelist = []
        for message in loop_root.findall("./processInfo/"):
            messagelist.append(message.text)
        print('Job status is ', status)
        print('error messages:')
        pprint(messagelist)
    if status == 'complete' or status == 'complete_with_errors':# Download zipped order if status is complete or complete_with_errors
        downloadURL = 'https://n5eil02u.ecs.nsidc.org/esir/' + orderID + '.zip'
        print('Job status is ', status)
        print('Zip download URL: ', downloadURL)
        print('Beginning download of zipped output...')
        zip_response = request.urlopen(downloadURL)
        with zipfile.ZipFile(io.BytesIO(zip_response.read())) as z:
            z.extractall(path)
        print('Download is complete.')
    else: print('Request failed.')
    
    # Clean up Outputs folder by removing individual granule folders 
    for root, dirs, files in os.walk(path, topdown=False):
        for file in files:
            try:
                shutil.move(os.path.join(root, file), path)
            except OSError:
                pass
        for name in dirs:
            os.rmdir(os.path.join(root, name))
    return  

request_nsidc_data(api_request)<jupyter_output>Order status URL:  https://n5eil02u.ecs.nsidc.org/egi/request/5000001156962
Job status is  processing . Trying again.
Job status is  complete
Zip download URL:  https://n5eil02u.ecs.nsidc.org/esir/5000001156962.zip
Beginning download of zipped output...
Download is complete.<jupyter_text>___ Read in SnowEx dataThis SnowEx data set is provided in CSV. A [Pandas DataFrame](https://pandas.pydata.org/pandas-docs/stable/getting_started/overview.html) is used to easily read in data.<jupyter_code>snowex_path = './Outputs/SnowEx17_GPR_Version2_Week1.csv' # Define local filepath
df = pd.read_csv(snowex_path, sep='\t') 
df.head()<jupyter_output><empty_output><jupyter_text>Extract date values The collection date needs to be extracted from the `collection` parameter:<jupyter_code>df['date'] = df.collection.str.rsplit('_').str[-1].astype(str)
df.date = pd.to_datetime(df.date, format="%m%d%y")
df = df.sort_values(['date'])
df.head()<jupyter_output><empty_output><jupyter_text>Convert to Geopandas dataframe to provide point geometryAccording to the SnowEx documentation, the data are available in UTM Zone 12N so we'll set to this projection to allow for geospatial analysis:<jupyter_code>gdf_utm= gpd.GeoDataFrame(df, geometry=gpd.points_from_xy(df.x, df.y), crs='EPSG:32612')
gdf_utm.head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08633124045477686, 'fraction_numerical': 0.05845918886814865, 'mean_word_length': 5.097516813243662, 'pattern_counts': {'":': 3, '<': 48, '<?xml version=': 0, '>': 50, 'https://': 28, 'lorem ipsum': 0, 'www.': 1, 'xml': 1}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 21, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '60668', 'license': 'MIT', 'n_tokens_mistral': 8022, 'n_tokens_neox': 7013, 'n_words': 2286, 'path': 'book/tutorials/nsidc-access/nsidc-data-access.ipynb', 'repo_name': 'julialiu18/website', 'size': '23577'}