<jupyter_start><jupyter_text>THREDDS Data Server: Data access The following information will demonstrate: - Discover some data through NCI's Geonetwork page - Direct download of a file - OPeNDAP services:   - Using OPeNDAP to request/convert subset to an ASCII file    - Using OPeNDAP access and subsetting with Python modules - NetcdfSubset service   - Using THREDDS NetcdfSubset to request/convert subset another netCDF file    ``````  Browse for dataBegin by going to NCI's Geonetwork page: http://geonetwork.nci.org.au/This page contains the metadata records for NCI Data Collections as well as information on where to find the data. In this example, we will search for Landsat data:If we click on the first result, we see a brief overview of the metadata record. **Note:** For the full record, navigate to the upper-right corner of your browser to change to the "Full view" (eyeball icon). One of the options under **Download and links** is the NCI THREDDS Data Server Catalog page: By navigating to this link, the available (public) data subcollections and datasets will be visible:In this example, let's navigate to the ** LANDSAT data: scenes and tiles/ tiles/ EPSG3577/ LS8_OLI_TIRS_NBAR/ ** dataset:   To access the data we have several options. First select a data file:  1. File downloadTo download the full file, click the link titled: **HTTPServer** and the download should commence.   2. THREDDS OPeNDAP On the THREDDS catalog page for the file, click the link titled:  **OPeNDAP**  This will direct to the 'OPeNDAP Dataset Access Form' This page will list information on the file contents, including file metadata and variables. Each variable will also contain a list of variable-level metadata (or attributes).   Request subset in ASCII format: 1. To request a subset with the OPeNDAP Dataset Access Form, begin by selecting the desired subset.  2. To select a variable:    - Tick the box to the left of the variable name    - Provide subset range for each dimension of the variable    3. Navigate to the top of the page and select "Get ASCII" 4. This should return the subset in a new window that can be copied/pasted or saved as a new file.   Use OPeNDAP for remote data access with Python 1. Return to the OPeNDAP Dataset Access Form page 2. Copy the URL in the field titled "Data URL" 3. This URL can be used in the same manner as the path to a local file within Python.The following demonstrates how to access and subset with the common netCDF Python module "netCDF4".<jupyter_code>from netCDF4 import Dataset
data_url = "http://dapds00.nci.org.au/thredds/dodsC/rs0/tiles/EPSG3577/LS8_OLI_TIRS_NBART/LS8_OLI_TIRS_NBART_3577_-10_-28_2013.nc"<jupyter_output><empty_output><jupyter_text>Open file This action opens the file but no data is yet loaded (or decompressed) until requested. Once the file is open, the following actions in Python are identical to those used with local files.<jupyter_code>f = Dataset(data_url, 'r')<jupyter_output><empty_output><jupyter_text>Browse file contents<jupyter_code>## Display file dimensions
for item in f.dimensions:
    print item, f.dimensions[item].size


## Look at variables
vars = f.variables.keys()
for item in vars:
    print 'Variable: \t', item
    print 'Dimensions: \t', f[item].dimensions
    print 'Shape:    \t', f[item].shape, '\n'<jupyter_output>Variable: 	y
Dimensions: 	(u'y',)
Shape:    	(4000,) 

Variable: 	x
Dimensions: 	(u'x',)
Shape:    	(4000,) 

Variable: 	time
Dimensions: 	(u'time',)
Shape:    	(61,) 

Variable: 	crs
Dimensions: 	()
Shape:    	() 

Variable: 	extra_metadata
Dimensions: 	(u'time', u'maxStrlen64')
Shape:    	(61, 64) 

Variable: 	band_6
Dimensions: 	(u'time', u'y', u'x')
Shape:    	(61, 4000, 4000) 

Variable: 	band_7
Dimensions: 	(u'time', u'y', u'x')
Shape:    	(61, 4000, 4000) 

Variable: 	band_4
Dimensions: 	(u'time', u'y', u'x')
Shape:    	(61, 4000, 4000) 

Variable: 	band_5
Dimensions: 	(u'time', u'y', u'x')
Shape:    	(61, 4000, 4000) 

Variable: 	band_2
Dimensions: 	(u'time', u'y', u'x')
Shape:    	(61, 4000, 4000) 

Variable: 	band_3
Dimensions: 	(u'time', u'y', u'x')
Shape:    	(61, 4000, 4000) 

Variable: 	band_1
Dimensions: 	(u'time', u'y', u'x')
Shape:    	(61, 4000, 4000)<jupyter_text>Extract data subset examples (using indices)<jupyter_code># Extract the full spatial (x, y) range for the first time step:
band2_t0 = f.variables['band_2'][0, :, :]

# Extract all of time for a small spatial subset:
band2_tyx_subset = f.variables['band_2'][:, 2000:2500, 1000:1500]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09366026065827258, 'fraction_numerical': 0.033576319858626025, 'mean_word_length': 4.377672209026128, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 16, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '407944', 'license': 'MIT', 'n_tokens_mistral': 1601, 'n_tokens_neox': 1459, 'n_words': 597, 'path': 'Data_Access/Using_Thredds/THREDDS_DataAccess.ipynb', 'repo_name': 'kdruken/nci_notebooks', 'size': '4527'}