<jupyter_start><jupyter_text>Hurricane Ike Maximum Water LevelsCompute the maximum water level during Hurricane Ike on a 9 million node triangular mesh storm surge model.  Plot the results with Datashader.<jupyter_code>import xarray as xr
import numpy as np
import pandas as pd
import hvplot.xarray
import fsspec
from dask.distributed import Client, progress
#from dask_kubernetes import KubeCluster
#cluster = KubeCluster()
%%time
from dask_cloudprovider import FargateCluster
cluster = FargateCluster(n_workers=1, image='rsignell/pangeo-worker:2020-01-23c')
cluster.dashboard_link<jupyter_output><empty_output><jupyter_text>Start a dask cluster to crunch the data<jupyter_code>cluster.scale(2);
cluster<jupyter_output><empty_output><jupyter_text>For demos, I often click in this cell and do "Cell=>Run All Above", then wait until the workers appear.  This can take several minutes (up to 6!) for instances to spin up and Docker containers to be downloaded. Then I shutdown the notebook and run again from the beginning, and the workers will fire up quickly because the instances have not spun down yet.<jupyter_code>%%time
client = Client(cluster)
client<jupyter_output><empty_output><jupyter_text>Read the data using the cloud-friendly zarr data format<jupyter_code>ds = xr.open_zarr(fsspec.get_mapper('s3://pangeo-data-uswest2/esip/adcirc/ike', anon=False, requester_pays=True))
ds['zeta']<jupyter_output><empty_output><jupyter_text>How many GB of sea surface height data do we have?<jupyter_code>ds['zeta'].nbytes/1.e9<jupyter_output><empty_output><jupyter_text>Take the maximum over the time dimension and persist the data on the workers in case we would like to use it later.  This is the computationally intensive step.<jupyter_code>%%time
max_var = ds['zeta'].max(dim='time').compute()
progress(max_var)<jupyter_output><empty_output><jupyter_text>Visualize data on mesh using HoloViz.org tools<jupyter_code>import numpy as np
import datashader as dshade
import holoviews as hv
import geoviews as gv
import cartopy.crs as ccrs
import hvplot.xarray
import holoviews.operation.datashader as dshade

dshade.datashade.precompute = True
hv.extension('bokeh')
v = np.vstack((ds['x'], ds['y'], max_var)).T
verts = pd.DataFrame(v, columns=['x','y','vmax'])
points = gv.operation.project_points(gv.Points(verts, vdims=['vmax']))
tris = pd.DataFrame(ds['element'].values.astype('int')-1, columns=['v0','v1','v2'])
tiles = gv.tile_sources.OSM
value = 'max water level'
label = '{} (m)'.format(value)
trimesh = gv.TriMesh((tris, points), label=label)
mesh = dshade.rasterize(trimesh).opts(
              cmap='rainbow', colorbar=True, width=600, height=400)
tiles * mesh<jupyter_output><empty_output><jupyter_text>Extract a time series at a specified lon, lat location Because Xarray does not yet understand that `x` and `y` are coordinate variables on this triangular mesh, we create our own simple function to find the closest point. If we had a lot of these, we could use a more fancy tree algorithm.<jupyter_code># find the indices of the points in (x,y) closest to the points in (xi,yi)
def nearxy(x,y,xi,yi):
    ind = np.ones(len(xi),dtype=int)
    for i in range(len(xi)):
        dist = np.sqrt((x-xi[i])**2+(y-yi[i])**2)
        ind[i] = dist.argmin()
    return ind
#just offshore of Galveston
lat = 29.2329856
lon = -95.1535041
ind = nearxy(ds['x'].values,ds['y'].values,[lon], [lat])[0]
ds['zeta'][:,ind].hvplot(grid=True)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11470673215833574, 'fraction_numerical': 0.013579890205143023, 'mean_word_length': 6.152892561983471, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 34, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '5818', 'license': 'Apache-2.0', 'n_tokens_mistral': 1206, 'n_tokens_neox': 1162, 'n_words': 348, 'path': 'hurricane_ike_levels.ipynb', 'repo_name': 'jkmacc-LANL/sagemaker-fargate-test', 'size': '3461'}