<jupyter_start><jupyter_text>lumbric &lt;<EMAIL>&gt;2019-09-25 - PyUGATxarrayhttps://github.com/pydata/xarrayhttps://xarray.pydata.org/      xarray = numpy       xarray = numpy              + named axis      xarray = numpy              + named axis             + coordinates                   xarray = numpy              + named axis             + coordinates             + parallelism      xarray = numpy              + named axis             + coordinates             + parallelism            = pandas              + netcdf              + more dimensions  Numpy broadcasting<jupyter_code>import numpy as np
m = np.arange(12).reshape((3, 4))
m
m + 100
v = np.array([1, 2, 3, 4])
m + v
m.shape
v.shape<jupyter_output><empty_output><jupyter_text>Broadcasting rules    A      (2d array):  5 x 4    B      (1d array):      1    Result (2d array):  5 x 4    A      (2d array):  5 x 4    B      (1d array):      4    Result (2d array):  5 x 4    A      (3d array):  15 x 3 x 5    B      (3d array):  15 x 1 x 5    Result (3d array):  15 x 3 x 5    A      (3d array):  15 x 3 x 5    B      (2d array):       3 x 5    Result (3d array):  15 x 3 x 5    A      (3d array):  15 x 3 x 5    B      (2d array):       3 x 1    Result (3d array):  15 x 3 x 5    Source: https://docs.scipy.org/doc/numpy/user/basics.broadcasting.html Here are examples of shapes that do not broadcast:    A      (1d array):  3    B      (1d array):  4  trailing dimensions do not match    A      (2d array):      2 x 1    B      (3d array):  8 x 4 x 3  second from last dimensions mismatchedSource: https://docs.scipy.org/doc/numpy/user/basics.broadcasting.html<jupyter_code>a = np.arange(5)
a
b = np.ones(5)
b
b
b[np.newaxis, :].shape
a[:, np.newaxis] * b[np.newaxis, :]<jupyter_output><empty_output><jupyter_text>xarray?<jupyter_code>import xarray as xr
a_xr = xr.DataArray(a, dims='a_dim')
b_xr = xr.DataArray(b, dims='b_dim')
a_xr
b_xr
a_xr * b_xr<jupyter_output><empty_output><jupyter_text>Example: wind speed time seriesSee also: https://github.com/lumbric/python_git_programming_course<jupyter_code>%matplotlib inline
import matplotlib
import matplotlib.pyplot as plt

fname = '<snip>'
# matplotlib.rc('figure', figsize=(15, 10))
latitude_resolution = 100
longitude_resolution = 110
time_resolution = 365

# in m/s, postfixing with _ms does not work well, because ms is miliseconds... :-/
wind_speeds = np.random.weibull(10, size=(latitude_resolution, longitude_resolution, time_resolution))
wind_speeds.shape
_ = plt.hist(wind_speeds.flatten(), density=True, bins=50)
# m/s --> kW
wind_to_power = np.random.normal(300, scale=20, size=(latitude_resolution, longitude_resolution))
plt.imshow(wind_to_power)
wind_to_power * wind_speeds
wind_speeds.shape
wind_to_power.shape
wind_speeds = np.rollaxis(wind_speeds, 2)
wind_speeds.shape
wind_to_power.shape
average_power = (wind_to_power * wind_speeds).mean(axis=0)
average_power.shape<jupyter_output><empty_output><jupyter_text>Example: now with xarray<jupyter_code>wind_speeds_raw = np.random.weibull(10, size=(latitude_resolution, longitude_resolution, time_resolution))
wind_speeds = xr.DataArray(wind_speeds_raw, dims=('latitude', 'longitude', 'time'))
wind_speeds
wind_to_power_raw = np.random.normal(300, scale=20, size=(latitude_resolution, longitude_resolution))
wind_to_power = xr.DataArray(wind_to_power_raw, dims=('latitude', 'longitude'))
(wind_to_power * wind_speeds).mean(dim='time')<jupyter_output><empty_output><jupyter_text>xarray = numpy              + named axis             + coordinates             + parallelism            = pandas              + netcdf              + more dimensions  Coordinates<jupyter_code>wind_velocity = xr.load_dataset(fname.format(m=4, y=1994))
wind_velocity
wind_velocity.isel(latitude=103, longitude=411, time=711).u100
wind_velocity.sel(latitude=67.75, longitude=-172.5, time='1994-04-01 11:00' ).u100<jupyter_output><empty_output><jupyter_text>Parallel computing<jupyter_code>fnames = [fname.format(m=m, y=y) for m in range(1,13) for y in range(2008, 2019)]
wind_velocity = xr.open_mfdataset(fnames,
                                  combine='by_coords',
                                  concat_dim='time',
                                  chunks=100)
wind_velocity.nbytes / 1024**3  # in GB
wind_speed = ((wind_velocity.u100**2 + wind_velocity.u100**2)**0.5).sel(time=slice('2014', '2019'))
wind_speed.nbytes / 1024**3  # in GB
wind_speed_avg = wind_speed.mean()
wind_speed_avg.compute()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11623246492985972, 'fraction_numerical': 0.04497884658205299, 'mean_word_length': 3.1554116558741905, 'pattern_counts': {'":': 0, '<': 31, '<?xml version=': 0, '>': 32, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '196868', 'license': 'MIT', 'n_tokens_mistral': 1728, 'n_tokens_neox': 1569, 'n_words': 364, 'path': '2019-09-25_pyugat_xarray_overview/xarray-demo.ipynb', 'repo_name': 'lumbric/presentations', 'size': '4499'}