<jupyter_start><jupyter_text>Ocean Observatories Initiative (OOI) Data   Obtaining OOI Data  - Data Portal  - M2M  - [Raw Data Server](https://rawdata.oceanobservatories.org/files/RS03ASHS/PN03B/06-CAMHDA301/) (CamHD Video Data)  - IRIS (seismic data)  - ERDDAP  - [Xarray and Zarr on AWS S3](https://io2data.s3.us-west-2.amazonaws.com/index.htmldata/) (unofficial profiler data demo)  - Others coming soon!  Xarray and Zarr on AWS S3  - Demonstration subset of OOI profiler data    - Stored as Zarr in a cloud-performant object store    - Hopefully most or all of OOI data will be stored like this in the near future    - [OOI Data Team Inspector](https://ooi-visualocean.whoi.edu/regions/view/RS)    - [AWS Store Browser](https://io2data.s3.us-west-2.amazonaws.com/index.htmldata/) (@lsetiawan)   Learning Objectives  - Open zarr groups from Amazon S3  - Use Dask delayed functions to speed up data access  - Resample data using Xarray  - Plot using hvplot  Loading Data  Start Dask Cluster  - Settings > Auto-Start Dask  - Use the new button!<jupyter_code>client<jupyter_output><empty_output><jupyter_text>Build function for Dask delayed<jupyter_code>import s3fs
import xarray as xr
# function to open a zarr group on S3
def open_zarr_group(zarr_group):
    s3 = s3fs.S3FileSystem(anon=True)
    try:
        return xr.open_zarr(store=s3fs.S3Map(zarr_group, s3=s3), consolidated=True)
    except:
        return None<jupyter_output><empty_output><jupyter_text>Create list of Delayed objects<jupyter_code>import datetime
from dask import delayed
# zarr group base and start date
zarr_group_base = 'io2data/data/RS01SBPS-SF01A-2A-CTDPFA102-streamed-ctdpf_sbe43_sample/'
dt_start = datetime.date(year = 2019, month = 1, day = 1)

# loop and create list of Dask delayed objects
dsets_raw = []
num_days = 20
for i in range(num_days):
    dt = dt_start + datetime.timedelta(days=i)
    dt_str = dt.strftime("%Y%m%d")
    zarr_group = zarr_group_base + dt_str
    print(zarr_group)
    dsets_raw.append(delayed(open_zarr_group)(zarr_group))<jupyter_output><empty_output><jupyter_text>Run delayed functions on the Dask cluster<jupyter_code>from dask import compute
%%time
dsets_raw = compute(*dsets_raw)<jupyter_output><empty_output><jupyter_text>Remove "None" Objects, Concatenate, and Clean Up<jupyter_code>dsets = []
for dset_raw in dsets_raw:
    if dset_raw != None:
        dsets.append(dset_raw)
len(dsets)
ds = xr.concat(dsets, dim='time')
ds
ds = ds.reset_coords('seawater_pressure')
ds
print('Dataset size: ~' + str(round(ds.nbytes/1024/1024)) + ' MB\nPoints: ' + str(len(ds.time)))<jupyter_output><empty_output><jupyter_text>Resample Data<jupyter_code>%%time
resampling_period = '4min'
variables = ['time', 'seawater_pressure', 'corrected_dissolved_oxygen']
ds_resampled = ds[variables].resample(time=resampling_period).mean().load()
360/100*8<jupyter_output><empty_output><jupyter_text>Plot Using Hvplot<jupyter_code>import holoviews as hv
hv.extension('bokeh')
import hvplot.pandas
df = ds_resampled.to_dataframe()
df.reset_index(inplace=True)
df.head()
df.hvplot.scatter('time', 'seawater_pressure', c='corrected_dissolved_oxygen',
                  cmap='viridis', size=30, colorbar=True, flip_yaxis=True,
                 ylim=(0, 200))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11168751907232224, 'fraction_numerical': 0.02105584375953616, 'mean_word_length': 5.104283054003725, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 30, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '116352', 'license': 'Apache-2.0', 'n_tokens_mistral': 1263, 'n_tokens_neox': 1196, 'n_words': 262, 'path': '03-OOI-Xarray.ipynb', 'repo_name': 'abarciauskas-bgse/ohw19-tutorial-data-access-viz', 'size': '3277'}