<jupyter_start><jupyter_text>Quick introduction<jupyter_code>from erddapy import ERDDAP
import pandas as pd


constraints = {
    'time>=': '2016-07-10T00:00:00Z',
    'time<=': '2017-02-10T00:00:00Z',
    'latitude>=': 38.0,
    'latitude<=': 41.0,
    'longitude>=': -72.0,
    'longitude<=': -69.0,
}

variables = [
 'depth',
 'latitude',
 'longitude',
 'salinity',
 'temperature',
 'time',
]

e = ERDDAP(
    server='https://data.ioos.us/gliders/erddap',
    protocol='tabledap',
    response='csv',
    dataset_id='blue-20160818T1448',
    constraints=constraints,
    variables=variables,
)

url = e.get_download_url()
print(url)
df = e.to_pandas(
    index_col='time',
    parse_dates=True,
    skiprows=(1,)  # units information can be dropped.
).dropna()

df.head()
%matplotlib inline
import matplotlib.pyplot as plt
import matplotlib.dates as mdates

fig, ax = plt.subplots(figsize=(17, 2))
cs = ax.scatter(df.index, df['depth'], s=15, c=df['temperature'], marker='o', edgecolor='none')

ax.invert_yaxis()
ax.set_xlim(df.index[0], df.index[-1])
xfmt = mdates.DateFormatter('%H:%Mh\n%d-%b')
ax.xaxis.set_major_formatter(xfmt)

cbar = fig.colorbar(cs, orientation='vertical', extend='both')
cbar.ax.set_ylabel('Temperature ($^\circ$C)')
ax.set_ylabel('Depth (m)');<jupyter_output><empty_output><jupyter_text>Longer introductionFirst we need to instantiate the `ERDDAP` URL constructor for a server.In this example we will use [https://data.ioos.us/gliders/erddap](https://data.ioos.us/gliders/erddap).<jupyter_code>from erddapy import ERDDAP


e = ERDDAP(server='http://krafla.pmel.noaa.gov:8080/erddap')<jupyter_output><empty_output><jupyter_text>What are the methods/attributes available?<jupyter_code>[method for method in dir(e) if not method.startswith('_')]<jupyter_output><empty_output><jupyter_text>All the `get_` will return a valid `ERDDAP` URL for the requested response and options.`erddapy` will raise an error is if URL HEADER cannot be validated.<jupyter_code>print(e.get_search_url(search_for='all'))<jupyter_output>http://krafla.pmel.noaa.gov:8080/erddap/search/advanced.html?page=1&itemsPerPage=1000&protocol=(ANY)&cdm_data_type=(ANY)&institution=(ANY)&ioos_category=(ANY)&keywords=(ANY)&long_name=(ANY)&standard_name=(ANY)&variableName=(ANY)&minLon=(ANY)&maxLon=(ANY)&minLat=(ANY)&maxLat=(ANY)&minTime=(ANY)&maxTime=(ANY)&searchFor=all<jupyter_text>There are many responses available, see the docs for [griddap](http://coastwatch.pfeg.noaa.gov/erddap/griddap/documentation.html) and[tabledap](http://coastwatch.pfeg.noaa.gov/erddap/tabledap/documentation.html) respectively.The most useful ones for Pythonistas are the `.csv` and `.nc` that can be read with pandas and netCDF4-python respectively.Let's load the `csv` reponse directly with `pandas`.<jupyter_code>import pandas as pd


df = pd.read_csv(e.get_search_url(response='csv', search_for='all'))
'We have {} tabledap, {} griddap, and {} wms endpoints.'.format(
    len(set(df['tabledap'].dropna())),
    len(set(df['griddap'].dropna())),
    len(set(df['wms'].dropna()))
)<jupyter_output><empty_output><jupyter_text>We can refine our search by providing some constraints.<jupyter_code>def show_iframe(src):
    """Helper function to show HTML returns."""
    from IPython.display import HTML
    iframe = '<iframe src="{src}" width="100%" height="950"></iframe>'.format
    return HTML(iframe(src=src))<jupyter_output><empty_output><jupyter_text>Let's narrow the search area, time span, and look for `sea_water_temperature` only.<jupyter_code>kw = {
    'keywords': 'ALAMO',
    'min_lon': -180.0,
    'max_lon': -130.0,
    'min_lat': 65.0,
    'max_lat': 90.0,
    'min_time': '2015-01-10T00:00:00Z',
    'max_time': '2018-02-10T00:00:00Z',
    'cdm_data_type': 'profile'
}

search_url = e.get_search_url(response='html', **kw)

show_iframe(search_url)<jupyter_output><empty_output><jupyter_text>We can see that the search form above was correctly populated with the constraints we provided.Let us change the response from `.html` to `.csv`,so we load it as a `pandas.DataFrame`,and inspect what are the `Dataset ID`s available for download.<jupyter_code>search_url = e.get_search_url(response='csv', **kw)
search = pd.read_csv(search_url)
gliders = search['Dataset ID'].values

print('Found {} Glider Datasets:\n{}'.format(len(gliders), '\n'.join(gliders)))<jupyter_output>Found 7 Glider Datasets:
arctic_heat_alamo_profiles_9058
arctic_heat_alamo_profiles_9076
arctic_heat_alamo_profiles_9085
arctic_heat_alamo_profiles_9115
arctic_heat_alamo_profiles_9116
arctic_heat_alamo_profiles_9117
arctic_heat_alamo_profiles_9119<jupyter_text>Now that we know the `Dataset ID`s we can explore their metadata with the `get_info_url` method.<jupyter_code>info_url = e.get_info_url(dataset_id=gliders[0], response='html')

show_iframe(src=info_url)<jupyter_output><empty_output><jupyter_text>Again, with the `csv` response, we can manipulate the metadata and find the variables that have the `cdm_profile_variables` attribute.<jupyter_code>info_url = e.get_info_url(dataset_id=gliders[0], response='csv')

info = pd.read_csv(info_url)

info.head()
''.join(info.loc[info['Attribute Name'] == 'cdm_profile_variables', 'Value'])<jupyter_output><empty_output><jupyter_text>Selecting variables by theirs attributes is such a common operation that `erddapy` brings its own method to simplify this task.The `get_var_by_attr` method is inspired by `netCDF4-python`'s `get_variables_by_attributes` however, because `erddapy` is operating on remote serves, it will return the variable names instead of the actual variables.Here we check what is/are the variable(s) associated with the `standard_name` used in the search.Note that `get_var_by_attr` caches the last response in case the user needs to make multiple requests,but it will loose its state when a new request is made.(See the execution times below.)<jupyter_code>%%time

# First one, slow.
e.get_var_by_attr(
    dataset_id='blue-20160818T1448',
    standard_name='PRESS'
)
%%time

# Second one on the same glider, fast.
e.get_var_by_attr(
    dataset_id='blue-20160818T1448',
    standard_name='sea_water_practical_salinity'
)
%%time

# New one, slow.
e.get_var_by_attr(
    dataset_id='cp_336-20170116T1254',
    standard_name='sea_water_practical_salinity'
)<jupyter_output>CPU times: user 16.3 ms, sys: 1.95 ms, total: 18.2 ms
Wall time: 326 ms<jupyter_text>With Python it is easy to loop over all the `dataset_id`s for the variables with `standard_names`<jupyter_code>variables = [e.get_var_by_attr(dataset_id=glider, standard_name=lambda v: v is not None) for glider in gliders]<jupyter_output><empty_output><jupyter_text>We can construct a `set` with the common variables in those `dataset_id`s.<jupyter_code>common_variables = set(variables[0]).intersection(*variables[1:])

common_variables<jupyter_output><empty_output><jupyter_text>Last, but not least, the download endpoint!It is important to note that the download constraints are based on the variables names and not the standardized ones for the `get_search_url` method.<jupyter_code>constraints = {
    'longitude>=': kw['min_lon'],
    'longitude<=': kw['max_lon'],
    'latitude>=': kw['min_lat'],
    'latitude<=': kw['max_lat'],
    'time>=': kw['min_time'],
    'time<=': kw['max_time'],
}



download_url = e.get_download_url(
    dataset_id=gliders[0],
    protocol='tabledap',
    variables=common_variables,
    constraints=constraints
)

print(download_url)<jupyter_output>http://krafla.pmel.noaa.gov:8080/erddap/tabledap/arctic_heat_alamo_profiles_9058.html?PRES,longitude,TEMP,latitude,PSAL,time,JULD&longitude>=-180.0&longitude<=-130.0&latitude>=65.0&latitude<=90.0&time>=1420848000.0&time<=1518220800.0<jupyter_text>Putting everything in `DataFrame`s.<jupyter_code>from requests.exceptions import HTTPError


def download_csv(url):
    return pd.read_csv(url, index_col='time', parse_dates=True, skiprows=[1])


dfs = {}
for glider in gliders:
    try:
        download_url = e.get_download_url(
            dataset_id=glider,
            protocol='tabledap',
            variables=common_variables,
            response='csv',
            constraints=constraints
        )
    except HTTPError:
        print('Failed to download {}'.format(glider))
        continue
    dfs.update({glider: download_csv(download_url)})<jupyter_output>Failed to download alldrudnickGliders
https://data.ioos.us/gliders/erddap/tabledap/blue-20160818T1448.csv?time_uv_qc,lon_uv,longitude,conductivity,u,v_qc,depth_qc,v,time_uv,density_qc,temperature_qc,salinity_qc,salinity,conductivity_qc,time,depth,density,time_qc,longitude_qc,lon_uv_qc,pressure_qc,precise_lon,temperature,precise_lat,latitude,u_qc,precise_time_qc,lat_uv,pressure,precise_time,lat_uv_qc,latitude_qc&longitude>=-72.0&longitude<=-69.0&latitude>=38.0&latitude<=41.0&time>=1468108800.0&time<=1486684800.0
https://data.ioos.us/gliders/erddap/tabledap/cp_335-20170116T1459.csv?time_uv_qc,lon_uv,longitude,conductivity,u,v_qc,depth_qc,v,time_uv,density_qc,temperature_qc,salinity_qc,salinity,conductivity_qc,time,depth,density,time_qc,longitude_qc,lon_uv_qc,pressure_qc,precise_lon,temperature,precise_lat,latitude,u_qc,precise_time_qc,lat_uv,pressure,precise_time,lat_uv_qc,latitude_qc&longitude>=-72.0&longitude<=-69.0&latitude>=38.0&latitude<=41.0&time>=1468108800.0&time<=1486684800.0
ht[...]<jupyter_text>It is unclear to me why `ERDDAP` found two datasets with the constraints provided but failed to generate an URL for the download.<jupyter_code>try:
    e.get_download_url(
        dataset_id='silbo-20160413T1534',
        protocol='tabledap',
        variables=common_variables,
        response='csv',
        constraints=constraints,
    )
except Exception as err:
    print(err)<jupyter_output>500 Server Error: Internal Server Error for url: https://data.ioos.us/gliders/erddap/tabledap/silbo-20160413T1534.csv?time_uv_qc,lon_uv,longitude,conductivity,u,v_qc,depth_qc,v,time_uv,density_qc,temperature_qc,salinity_qc,salinity,conductivity_qc,time,depth,density,time_qc,longitude_qc,lon_uv_qc,pressure_qc,precise_lon,temperature,precise_lat,latitude,u_qc,precise_time_qc,lat_uv,pressure,precise_time,lat_uv_qc,latitude_qc&longitude%3E=-72.0&longitude%3C=-69.0&latitude%3E=38.0&latitude%3C=41.0&time%3E=1468108800.0&time%3C=1486684800.0<jupyter_text>The glider datasets should be masked automatically but we found that is not true.The cell below applies the mask as described by the data QC flag.<jupyter_code>import numpy as np


for glider in dfs.keys():
    dfs[glider].loc[dfs[glider]['salinity_qc'] == 9, 'salinity'] = np.NaN
    dfs[glider].loc[dfs[glider]['pressure_qc'] == 9, 'pressure'] = np.NaN
    dfs[glider].loc[dfs[glider]['temperature_qc'] == 9, 'temperature'] = np.NaN
    dfs[glider].loc[dfs[glider]['salinity'] <= 0, 'salinity'] = np.NaN<jupyter_output><empty_output><jupyter_text>Finally let's see some figures!<jupyter_code>%matplotlib inline
import matplotlib.pyplot as plt
import cartopy.crs as ccrs
from cartopy.mpl.ticker import LongitudeFormatter, LatitudeFormatter


def make_map(extent):
    fig, ax = plt.subplots(
        figsize=(9, 9),
        subplot_kw=dict(projection=ccrs.PlateCarree())
    )
    ax.coastlines(resolution='10m')
    ax.set_extent(extent)

    ax.set_xticks([extent[0], extent[1]], crs=ccrs.PlateCarree())
    ax.set_yticks([extent[2], extent[3]], crs=ccrs.PlateCarree())
    lon_formatter = LongitudeFormatter(zero_direction_label=True)
    lat_formatter = LatitudeFormatter()
    ax.xaxis.set_major_formatter(lon_formatter)
    ax.yaxis.set_major_formatter(lat_formatter)

    return fig, ax


dx = dy = 0.5
extent = kw['min_lon']-dx, kw['max_lon']+dx, kw['min_lat']+dy, kw['max_lat']+dy

fig, ax = make_map(extent)
for glider, df in dfs.items():
    ax.plot(df['longitude'], df['latitude'], label=glider)

leg = ax.legend()
def glider_scatter(df, ax, glider):
    ax.scatter(df['temperature'], df['salinity'],
               s=10, alpha=0.5, label=glider)

fig, ax = plt.subplots(figsize=(9, 9))
ax.set_ylabel('salinity')
ax.set_xlabel('temperature')
ax.grid(True)

for glider, df in dfs.items():
    glider_scatter(df, ax, glider)

leg = ax.legend()<jupyter_output><empty_output><jupyter_text>Extras OPeNDAP response<jupyter_code>e.constraints = None
e.protocol = 'tabledap'

opendap_url = e.get_download_url(
    dataset_id='cp_336-20170116T1254',
)

print(opendap_url)
from netCDF4 import Dataset

with Dataset(opendap_url) as nc:
    print(nc.summary)<jupyter_output>The Pioneer Array is located off the coast of New England, south of Martha's Vineyard. The Continental Shelf-Slope area off the New England coast is a highly productive area and one that is located at a dynamic intersection where ocean currents meet in weather-like 'fronts', and where nutrients, pollutants, and other properties are exchanged between the coast and the deep ocean. Data from the inshore, shelf area helps to examine exchanges between the shelf and slope and the shelf ecosystem, as well as provide broader insight into the issues of air-sea gas exchange, including Carbon Dioxide. In order to provide synoptic, multi-scale observations of the outer continental shelf, shelf break region, and continental slope, the moored portion of the Pioneer Array is supplemented by six Coastal Gliders. The role of the Coastal Gliders is to monitor the mesoscale field of the slope sea and outer shelf, resolving rings, eddies and meanders from the Gulf Stream as they impinge on the shelf break[...]<jupyter_text>netCDF "file-like" to xarray`open_dataset` will download a temporary file,so be careful with the constraints to avoid downloading several gigabytes!<jupyter_code>e.dataset_id = 'cp_336-20170116T1254'
e.response = 'nc'
e.variables = common_variables
e.constraints = constraints

download_url = e.get_download_url()
import requests


def humansize(nbytes):
    suffixes = ['B', 'KB', 'MB', 'GB', 'TB', 'PB']
    k = 0
    while nbytes >= 1024 and k < len(suffixes)-1:
        nbytes /= 1024.
        k += 1
    f = ('%.2f' % nbytes).rstrip('0').rstrip('.')
    return '%s %s' % (f, suffixes[k])

r = requests.head(download_url)
nbytes = float(r.headers['Content-Length'])
humansize(nbytes)<jupyter_output><empty_output><jupyter_text>That is the uncompressed size, we are OK because the download will be less than that, `ERDDAP` streams gzip'ed data.<jupyter_code>r.headers['Content-Encoding']
ds = e.to_xarray(decode_times=False)
ds<jupyter_output><empty_output><jupyter_text>```pythonfrom erddapy.utilities import _urlopendef to_xarray(url, **kw):    import xarray as xr    from tempfile import NamedTemporaryFile    data = _urlopen(url).read()    with NamedTemporaryFile(suffix='.nc', prefix='erddapy_') as tmp:        tmp.write(data)        tmp.flush()        return xr.open_dataset(tmp.name, **kw)ds = to_xarray(    download_url,)ds```<jupyter_code>ds['temperature']
data = ds['temperature'].values
depth = ds['depth'].values

mask = ~np.ma.masked_invalid(data).mask
data = data[mask]
depth = depth[mask]
lon = ds['longitude'].values[mask]
lat = ds['latitude'].values[mask]
import warnings


with warnings.catch_warnings():
    warnings.simplefilter('ignore')
    mask = depth <= 5

data = data[mask]
depth = depth[mask]
lon = lon[mask]
lat = lat[mask]
%matplotlib inline
import matplotlib.pyplot as plt
import cartopy.crs as ccrs


dx = dy = 1.5
extent = (
    ds.geospatial_lon_min-dx, ds.geospatial_lon_max+dx,
    ds.geospatial_lat_min-dy, ds.geospatial_lat_max+dy
)
fig, ax = make_map(extent)

cs = ax.scatter(lon, lat, c=data, s=50, alpha=0.5, edgecolor='none')
cbar = fig.colorbar(cs, orientation='vertical',
                    fraction=0.1, shrink=0.9, extend='both')
ax.coastlines('10m');<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13046517539400102, 'fraction_numerical': 0.03126588713777326, 'mean_word_length': 5.988010657193605, 'pattern_counts': {'":': 0, '<': 102, '<?xml version=': 0, '>': 100, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 24, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '22', 'dir': 'jupyter-structured-clean-dedup', 'id': '281433', 'license': 'MIT', 'n_tokens_mistral': 6019, 'n_tokens_neox': 5600, 'n_words': 1153, 'path': '2018/EcoFOCI erddapy examples/erddapy_quick_intro.ipynb', 'repo_name': 'shaunwbell/ipythonnb', 'size': '15736'}