<jupyter_start><jupyter_text>PODPAC Data LibraryThe PODPAC data library contains a library of pre-wrapped data sources. These data sources can then be easily accessed programatically. As an example, we have wrapped some of the SMAP products.  SMAP DataSource Example<jupyter_code>import getpass
username = input("Username:");   password = getpass.getpass('Password:')
from podpac import datalib
from podpac.datalib import smap

datalib.smap.SMAP_PRODUCT_MAP.product.data.tolist()
product = datalib.smap.SMAP_PRODUCT_MAP.product[0].item()
product
smap = datalib.smap.SMAP(product=product, interpolation='nearest',
                        username=username, password=password)
# The available coordinates are built from all available SMAP sources for this product
# This requires multiple http get requests from the DAAC OpenDAP server, and may take a while
# Because of this, the results are cached after the first run
smap.native_coordinates<jupyter_output><empty_output><jupyter_text>A data source's `native_coordinates` describes the "coverage" of the data, that is, it gives the lat,lon coordinates and times for which the data is available. Thus, the data is natively available for those coordinates.However, a user may request any coordinates, and PODPAC will do the requested interpolation. This will happen without downloading the entire dataset, or any manual data wrangling<jupyter_code>coordinates = podpac.Coordinate(lat=(-90, 90, 1.), lon=(-180, 180, 1.), time='2017-10-10T12:00:00', 
                               order=['lat', 'lon', 'time'])
output = smap.execute(coordinates)
print (output.shape)
output[..., 0].plot()
figure()
coordinates = podpac.Coordinate(lat=(0., 90, 1.), lon=(-180, 180, 1.), time='2017-10-15T12:00:00', 
                               order=['lat', 'lon', 'time'])
output = smap.execute(coordinates)
print (output.shape)
output[..., 0].plot()
# !!! This takes a while (hours)... patience. 
figure()
coordinates = podpac.Coordinate(lat=39., lon=-77, time=('2017-09-01', '2017-10-31', '1,D'), 
                               order=['lat', 'lon', 'time'])
output = smap.execute(coordinates)
print (output.shape)
output.plot()
# We can also look at what other layers are available for this SMAP product
sm = smap.sources[0].sources[0]
print( str(sm.dataset.keys()).replace(',', '\n'))<jupyter_output>KeysView(<DatasetType with children 'Forecast_Data_sm_surface_forecast'
 'Forecast_Data_tb_v_forecast'
 'Forecast_Data_surface_temp_forecast'
 'Forecast_Data_tb_v_forecast_ensstd'
 'Forecast_Data_soil_temp_layer1_forecast'
 'Forecast_Data_sm_profile_forecast'
 'Forecast_Data_tb_h_forecast'
 'Forecast_Data_sm_rootzone_forecast'
 'Forecast_Data_tb_h_forecast_ensstd'
 'time'
 'Analysis_Data_sm_surface_analysis_ensstd'
 'Analysis_Data_surface_temp_analysis_ensstd'
 'Analysis_Data_sm_rootzone_analysis_ensstd'
 'Analysis_Data_sm_surface_analysis'
 'Analysis_Data_sm_profile_analysis_ensstd'
 'Analysis_Data_sm_rootzone_analysis'
 'Analysis_Data_sm_profile_analysis'
 'Analysis_Data_soil_temp_layer1_analysis'
 'Analysis_Data_soil_temp_layer1_analysis_ensstd'
 'Analysis_Data_surface_temp_analysis'
 'cell_row'
 'cell_column'
 'Observations_Data_tb_v_obs'
 'Observations_Data_tb_v_obs_time_sec'
 'Observations_Data_tb_h_obs'
 'Observations_Data_tb_h_orbit_flag'
 'Observations_Data_tb_h_resolution_fla[...]<jupyter_text>SMAP Implementation DetailsThe SMAP data can be accessed from and OpenDAP server from a NASA DAAC. The data is organized in a folders, where each folder represents a day. Inside each folder, there are multiple files for different times of the day. Or:* SMAP    * YYYY.MM.DD        * YYYY.MM.DD-HH:mm:ss        * YYYY.MM.DD-HH:mm:ss        * ...    * ...    * YYYY.MM.DD        * YYYY.MM.DD-HH:mm:ss        * YYYY.MM.DD-HH:mm:ss        * ...    The SMAP data source organizes the data in the same way. In fact, the SMAP node composites multiple SMAPDateFolder's, which each composites multiple SMAPSource nodes.<jupyter_code>print(smap.sources[:3])
print(smap.source_coordinates.coords['time'][:3])
print(smap.sources[2].sources)
print(smap.sources[2].source_coordinates)
print(smap.sources[2].sources[3])
print(smap.sources[2].sources[3].native_coordinates)<jupyter_output><podpac.datalib.smap.SMAPSource object at 0x7f619dddf668>
Coordinate
	lat: MonotonicCoord: Bounds[-84.65644073486328, 84.65644073486328], N[1624], ctype["segment"]
	lon: MonotonicCoord: Bounds[-179.95474243164062, 179.95474243164062], N[3856], ctype["segment"]
	time: MonotonicCoord: Bounds[2015-04-02T12:00:00, 2015-04-02T12:00:00], N[1], ctype["segment"]<jupyter_text>PODPAC provides an `OrderedCompositor` node to allow this nested structure to be easily implemented.<jupyter_code>smap.__class__.__bases__<jupyter_output><empty_output><jupyter_text>PODPAC provides a PyDAP data type for wrapping OpenDAP data sources:<jupyter_code>smap.sources[0].sources[0].__class__.__bases__<jupyter_output><empty_output><jupyter_text>Standard PODPAC data typesPODPAC provides multiple standard base data types that can be used to wrap new data sources.<jupyter_code>std_types = [t for t in dir(podpac.core.data.type) 
      if '_' not in t and t[0].isupper() and t not in ['BytesIO', 'OrderedDict', 'UnitRegistry']]
std_types.sort()
print(std_types)<jupyter_output>['NumpyArray', 'PyDAP', 'RasterioSource', 'ReprojectedSource', 'S3Source', 'WCS']<jupyter_text>Standard PODPAC algorithmsPODPAC provides a few standard algorithms as well<jupyter_code>black_list = ['Coordinate', 'Node', 'UnitsDataArray', "OrderedDict", 'UnitRegistry', 'COMMON_DOC', 'COMMOM_DOC_NODE', 'COMMON_NODE_DOC']
std_algs = [t for t in np.unique(dir(podpac.core.algorithm.algorithm) + 
                        dir(podpac.core.algorithm.signal) + 
                        dir(podpac.core.algorithm.stats)) if t not in black_list and t[0].isupper()]
std_algs.sort()
print (std_algs)
# Implementing an algorithm is simple
# As an example, let's square some data
class Square(podpac.Algorithm):
    node = podpac.core.node.tl.Instance(podpac.Node)
    
    def algorithm(self):
        return self.outputs['node'] ** 2
    
s = Square(node=smap)
coordinates = podpac.Coordinate(lat=(0., 90, 1.), lon=(-180, 180, 1.), time='2017-10-15T12:00:00', 
                               order=['lat', 'lon', 'time'])
output = s.execute(coordinates)
figure()
output[..., 0].plot()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1167997494911539, 'fraction_numerical': 0.03616721465476749, 'mean_word_length': 5.166023166023166, 'pattern_counts': {'":': 0, '<': 28, '<?xml version=': 0, '>': 27, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '584915', 'license': 'Apache-2.0', 'n_tokens_mistral': 2297, 'n_tokens_neox': 2124, 'n_words': 502, 'path': 'notebooks/scratch/old-examples/PODPAC-demo.ipynb', 'repo_name': 'creare-com/podpac_examples', 'size': '6387'}