<jupyter_start><jupyter_text>Xarray Introduction - Unlabeled, N-dimensional arrays of numbers (e.g., NumPy’s ndarray) are the most widely used data structure in scientific computing. However, they lack a meaningful representation of the metadata associated with their data. Implementing such functionality is left to individual users and domain-specific packages. xarry expands on the capabilities of NumPy arrays, providing a lot of streamline data manipulation. - xarray's interface is based largely on the netCDF data model (variables, attributes, and dimensions), but it goes beyond the traditional netCDF interfaces to provide functionality similar to netCDF-java's Common Data Model (CDM). - xarray is a useful tool for parallelizing and working with large datasets in the geosciences.  Data Structures - xarray has 2 fundamental data structures:    - `DataArray`, which holds single multi-dimensional variables and its coordinates    - `Dataset`, which holds multiple variables that potentially share the same coordinates   ![](../assets/images/xarray-data-structures.png)      `DataArray`The DataArray is xarray's implementation of a labeled, multi-dimensional array. It has several key properties:| Attribute 	| Description                                                                                                                              	||-----------	|------------------------------------------------------------------------------------------------------------------------------------------	|| `data`    	| `numpy.ndarray` or `dask.array` holding the array's values.                                                                              	|| `dims`    	| dimension names for each axis. For example:(`x`, `y`, `z`) (`lat`, `lon`, `time`).                                                       	|| `coords`  	| a dict-like container of arrays (coordinates) that label each point (e.g., 1-dimensional arrays of numbers, datetime objects or strings) 	|| `attrs`   	| an `OrderedDict` to hold arbitrary attributes/metadata (such as units)                                                                   	|| `name`    	| an arbitrary name of the array                                                                                                           	|<jupyter_code># Import packages
import numpy as np
import xarray as xr
# Create some sample data
data = 2 + 6 * np.random.exponential(size=(5, 3, 4))
data<jupyter_output><empty_output><jupyter_text>To create a basic `DataArray`, you can pass this numpy array of random data to `xr.DataArray`<jupyter_code>prec = xr.DataArray(data)
prec<jupyter_output><empty_output><jupyter_text>**Note:**     Xarray automatically generates some basic dimension names for us. You can also pass in your own dimension names and coordinate values:<jupyter_code># Use pandas to create an array of datetimes
import pandas as pd
times = pd.date_range('2019-04-01', periods=5)
times
# Use numpy to create array of longitude and latitude values
lons = np.linspace(-150, -60, 4)
lats = np.linspace(10, 80, 3)
lons, lats
coords = {'time': times, 'lat': lats, 'lon': lons}
dims = ['time', 'lat', 'lon']
# Add name, coords, dims to our data
prec = xr.DataArray(data, dims=dims, coords=coords, name='prec')
prec<jupyter_output><empty_output><jupyter_text>This is already improved upon from the original numpy array, because we have names for each of the dimensions (or axis in NumPy universe).  We can also add attributes to an existing `DataArray`:<jupyter_code>prec.attrs['units'] = 'mm'
prec.attrs['standard_name'] = 'precipitation'
prec
prec.data<jupyter_output><empty_output><jupyter_text>`Dataset`- Xarray's `Dataset` is a dict-like container of labeled arrays (`DataArrays`) with aligned dimensions. - It is designed as an in-memory representation of a netCDF dataset. - In addition to the dict-like interface of the dataset itself, which can be used to access any `DataArray` in a `Dataset`. Datasets have the following key properties:| Attribute   	| Description                                                                                                                              	||-------------	|------------------------------------------------------------------------------------------------------------------------------------------	|| `data_vars` 	| OrderedDict of `DataArray` objects corresponding to data variables.                                                                      	|| `dims`      	| dictionary mapping from dimension names to the fixed length of each dimension  (e.g., {`lat`: 6, `lon`: 6, `time`: 8}).                  	|| `coords`    	| a dict-like container of arrays (coordinates) that label each point (e.g., 1-dimensional arrays of numbers, datetime objects or strings) 	|| `attrs`     	| OrderedDict to hold arbitrary metadata pertaining to the dataset.                                                                        	|| `name`      	| an arbitrary name of the dataset                                                                                                         	| - DataArray objects inside a Dataset may have any number of dimensions but are presumed to share a common coordinate system. - Coordinates can also have any number of dimensions but denote constant/independent quantities, unlike the varying/dependent quantities that belong in data. To create a `Dataset` from scratch, we need to supply dictionaries for any variables (`data_vars`), coordinates (`coords`) and attributes (`attrs`):<jupyter_code>dset = xr.Dataset({'precipitation' : prec})
dset<jupyter_output><empty_output><jupyter_text>Let's add some fake `temperature` data array to this existing dataset:<jupyter_code>temp_data = 283 + 5 * np.random.randn(5, 3, 4)
temp = xr.DataArray(data=temp_data, dims=['time', 'lat', 'lon'], 
                    coords={'time': times, 'lat': lats, 'lon': lons},
                    name='temp',
                    attrs={'standard_name': 'air_temperature', 'units': 'kelvin'})
temp
# Now add this data array to our existing dataset
dset['temperature'] = temp
dset.attrs['history'] = 'Created for the xarray tutorial'
dset.attrs['author'] = 'foo and bar'
dset<jupyter_output><empty_output><jupyter_text>**Going Further:**     Xarray Documentation on Data Structures: http://xarray.pydata.org/en/latest/data-structures.html  Core xarray Features  Reading and Writing Files Xarray supports direct serialization and I/O to several file formats including pickle, netCDF, OPeNDAP (read-only), GRIB1/2 (read-only), and HDF by integrating with third-party libraries. Additional serialization formats for 1-dimensional data are available through pandas.File types- Pickle- NetCDF 3/4- RasterIO- Zarr- PyNioInteroperability- Pandas- Iris- CDMS- dask DataFrame  Opening xarray datasetsXarray's `open_dataset` and `open_mfdataset` are the primary functions for opening local or remote datasets such as netCDF, GRIB, OpenDap, and HDF. These operations are all supported by third party libraries (engines) for which xarray provides a common interface.<jupyter_code>!ncdump -h ./data/rasm.nc
ds = xr.open_dataset('./data/rasm.nc', engine='netcdf4')
ds<jupyter_output><empty_output><jupyter_text>Saving xarray datasets as netcdf filesXarray provides a high-level method for writing netCDF files directly from Xarray Datasets/DataArrays.<jupyter_code>ds.to_netcdf('./data/rasm_test.nc')<jupyter_output><empty_output><jupyter_text>Multifile datasetsXarray can read/write multifile datasets using the `open_mfdataset` and `save_mfdataset` functions.<jupyter_code>years, datasets = zip(*ds.groupby('time.year'))
paths = ['./data/%s.nc' % y for y in years]
print(paths)
len(datasets)
# write the 4 netcdf files
xr.save_mfdataset(datasets, paths)<jupyter_output><empty_output><jupyter_text>- Open a group of files and concatenate them into a single xarray.Dataset<jupyter_code>ds2 = xr.open_mfdataset('./data/19*nc')
ds2<jupyter_output><empty_output><jupyter_text>ZarrZarr is a Python package providing an implementation of chunked, compressed, N-dimensional arrays. Zarr has the ability to store arrays in a range of ways, including in memory, in files, and in cloud-based object storage such as Amazon S3 and Google Cloud Storage. Xarray’s Zarr backend allows xarray to leverage these capabilities.<jupyter_code># Zarr
ds.to_zarr('./data/rasm.zarr', mode='w')
!ls data/*zarr
!du -h data/*zarr
import zarr
compressor = zarr.Blosc(clevel=2, shuffle=-1)
ds.to_zarr('./data/rasm_compressed.zarr', mode='w', encoding={var: {'compressor': compressor} 
                                                              for var in ds.variables})
!ls data/*zarr
!du -h data/*zarr<jupyter_output>data/rasm.zarr:
[34mTair[m[m [34mtime[m[m [34mxc[m[m   [34myc[m[m

data/rasm_compressed.zarr:
[34mTair[m[m [34mtime[m[m [34mxc[m[m   [34myc[m[m
348K	data/rasm.zarr/yc
 12K	data/rasm.zarr/time
7.6M	data/rasm.zarr/Tair
332K	data/rasm.zarr/xc
8.3M	data/rasm.zarr
356K	data/rasm_compressed.zarr/yc
 12K	data/rasm_compressed.zarr/time
7.8M	data/rasm_compressed.zarr/Tair
340K	data/rasm_compressed.zarr/xc
8.5M	data/rasm_compressed.zarr<jupyter_text>**Going Further:**     Xarray I/O Documentation: http://xarray.pydata.org/en/latest/io.html  Label-based indexing Scientific data is inherently labeled. For example, time series data includes timestamps that label individual periods or points in time, spatial data has coordinates (e.g. longitude, latitude, elevation), and model or laboratory experiments are often identified by unique identifiers.<jupyter_code>ds = xr.open_dataset('./data/air_temperature.nc')
ds<jupyter_output><empty_output><jupyter_text>NumPy Positional IndexingWhen working with numpy, indexing is done by position (slices/ranges/scalars).<jupyter_code>t = ds['air'].data # numpy array 
t
t.shape
# extract a time-series for one spatial location
t[:, 20, 40]<jupyter_output><empty_output><jupyter_text>**but wait, what labels go with 10 and 20? Was that lat/lon or lon/lat? Where are the timestamps that go along with this time-series?**  Indexing with xarray xarray offers extremely flexible indexing routines that combine the best features of NumPy and pandas for data selection.<jupyter_code>da = ds['air'] # Extract data array
da<jupyter_output><empty_output><jupyter_text>- **NumPy style indexing still works (but preserves the labels/metadata)**<jupyter_code>da[:, 20, 40]<jupyter_output><empty_output><jupyter_text>- **Positional indexing using dimension names**<jupyter_code>da.isel(lat=20, lon=40)<jupyter_output><empty_output><jupyter_text>- **Label-based indexing**<jupyter_code>da.sel(lat=50., lon=200.)<jupyter_output><empty_output><jupyter_text>- **Nearest Neighbor Lookups**<jupyter_code>da.sel(lat=52.25, lon=251.8998, method='nearest')<jupyter_output><empty_output><jupyter_text>- **All of these indexing methods work on the dataset too:**<jupyter_code>ds.sel(lat=52.25, lon=251.8998, method='nearest')<jupyter_output><empty_output><jupyter_text>Vectorized IndexingLike numpy and pandas, xarray supports indexing many array elements at once in a vectorized manner:<jupyter_code># generate a coordinates for a transect of points
lat_points = xr.DataArray([52, 52.5, 53], dims='points')
lon_points = xr.DataArray([250, 250, 250], dims='points')
lat_points
# nearest neighbor selection along the transect
da.sel(lat=lat_points, lon=lon_points, method='nearest')<jupyter_output><empty_output><jupyter_text>Aggregation Xarray supports many of the aggregations methods that numpy has. A partial list includes: all, any, argmax, argmin, max, mean, median, min, prod, sum, std, var.Whereas the numpy syntax would require scalar axes, xarray can use dimension names:<jupyter_code>ds = xr.open_dataset("./data/air_temperature.nc")
da = ds['air']
da
da.mean(dim=['lat'])<jupyter_output><empty_output><jupyter_text>Arithmetic Arithmetic operations with a single DataArray automatically vectorize (like numpy) over all array values:<jupyter_code>da - 273.15
da_mean = da.mean(dim='time')
da_mean
da - da_mean<jupyter_output><empty_output><jupyter_text>**Note:**     Notice that this required broadcasting along the time dimension. NumPy broadcasting is covered in great detail in [NumPy Guide](../numpy/01-numpy-guide.ipynb).  Alignment xarray enforces alignment between index Coordinates (that is, coordinates with the same name as a dimension, marked by `*`) on objects used in binary operations.<jupyter_code>da
arr = da.isel(time=0, lat=slice(5, 10), lon=slice(7, 11))
arr
part = arr[:-1]
part<jupyter_output><empty_output><jupyter_text>- **Default behavior is an `inner join`**<jupyter_code>(arr + part) / 2<jupyter_output><empty_output><jupyter_text>- **We can also use an `outer join`**<jupyter_code>with xr.set_options(arithmetic_join="outer"):
    print((arr + part) / 2)<jupyter_output><xarray.DataArray 'air' (lat: 5, lon: 4)>
array([[   nan,    nan,    nan,    nan],
       [279.5 , 279.9 , 280.5 , 279.9 ],
       [279.4 , 278.9 , 278.4 , 276.6 ],
       [272.9 , 272.  , 271.79, 271.1 ],
       [265.1 , 264.9 , 265.5 , 266.5 ]], dtype=float32)
Coordinates:
  * lat      (lat) float64 52.5 55.0 57.5 60.0 62.5
  * lon      (lon) float32 217.5 220.0 222.5 225.0
    time     datetime64[ns] 2013-01-01<jupyter_text>**Note:**     Notice that missing values (`nan`) were inserted where it is appropriate.   GroupBy Operations xarray supports “group by” operations with the same API as pandas to implement the split-apply-combine strategy:- Split your data into multiple independent groups.- Apply some function to each group.- Combine your groups back into a single data object.Group by operations work on both Dataset and DataArray objects. Most of the examples focus on grouping by a single one-dimensional variable, although support for grouping over a multi-dimensional variable is also supported: - **Using groupby to calculate a monthly climatology:**<jupyter_code>da_climatology = da.groupby('time.month').mean('time')

da_climatology<jupyter_output><empty_output><jupyter_text>In this case, we provide what we refer to as a virtual variable (`time.month`). Other virtual variables include: `year`, `month`, `day`, `hour`, `minute`, `second`, `dayofyear`, `week`, `dayofweek`, `weekday` and `quarter`. It is also possible to use another DataArray or pandas object as the grouper.  Resampling OperationsIn order to resample time-series data, xarray provides a `resample` convenience method for frequency conversion and resampling of time series.<jupyter_code>da<jupyter_output><empty_output><jupyter_text>- **Downsample our 6 hourly time-series data to seasonal data:**<jupyter_code>da.resample(time="QS-DEC").mean(dim='time')<jupyter_output><empty_output><jupyter_text>- **Upsample our 6 hourly time-series data to 1 hourly data:**<jupyter_code>da.resample(time='1H').interpolate('linear')<jupyter_output><empty_output><jupyter_text>Rolling Window OperationsXarray objects include a rolling method to support rolling window aggregations:<jupyter_code>roller = da.rolling(time=3)
roller
roller.mean()<jupyter_output><empty_output><jupyter_text>- **We can also provide a custom function**<jupyter_code>def sum_minus_2(da, axis):
    return da.sum(axis=axis) - 2

roller.reduce(sum_minus_2)<jupyter_output><empty_output><jupyter_text>MaskingIndexing methods on xarray objects generally return a subset of the original data. However, it is sometimes useful to select an object with the same shape as the original data, but with some elements masked. To do this type of selection in xarray, use `where()`:<jupyter_code>da.where(da < 273)
xr.where(da < 273, 0, 1)<jupyter_output><empty_output><jupyter_text>Plotting Labeled data enables expressive computations. These same labels can also be used to easily create informative plots.xarray plotting functionality is a thin wrapper around the popular matplotlib library. Matplotlib syntax and function names were copied as much as possible, which makes for an easy transition between the two.  Matplotlib Integration Xarray has built-in plotting via `matplotlib` for DataAr`rays:<jupyter_code>da<jupyter_output><empty_output><jupyter_text>Plotting >2d Data<jupyter_code>%matplotlib inline
import matplotlib.pyplot as plt
da.plot()<jupyter_output><empty_output><jupyter_text>For high-dimensional data, xarray plots `histograms` by default.   Plotting 1D Data<jupyter_code>da_point_resample = da.isel(lat=20, lon=40).resample(time='1D')
t_max = da_point_resample.max('time')
t_min = da_point_resample.min('time')
t_max
t_min
t_max.plot(label='t_max')
t_min.plot(label='t_min')
plt.legend()<jupyter_output>/Users/abanihi/opt/miniconda3/envs/dev/lib/python3.6/site-packages/pandas/plotting/_converter.py:129: FutureWarning: Using an implicitly registered datetime converter for a matplotlib plotting method. The converter was registered by pandas on import. Future versions of pandas will require you to explicitly register matplotlib converters.

To register the converters:
	>>> from pandas.plotting import register_matplotlib_converters
	>>> register_matplotlib_converters()
  warnings.warn(msg, FutureWarning)<jupyter_text>Plotting 2D Data For 2-dimensional data, the xarray `plot()` method automatically does a QuadMesh contour plot informed by the metadata:<jupyter_code>t_mean = da.mean('time')
t_mean
t_mean.plot()<jupyter_output><empty_output><jupyter_text>FacetGrid Plots - **Calculate some seasonal anomalies and plot them:**<jupyter_code>da_month = da.resample(time='QS-Dec').mean('time')

climatology = da_month.groupby('time.season').mean('time')
anomalies = da_month.groupby('time.season') - climatology
anomalies
anomalies.plot(col='time', col_wrap=4)<jupyter_output><empty_output><jupyter_text>**Going Further:**     - [Advanced Plotting Notebook](02-xarray-advanced-plotting.ipynb)- Xarray's Documentation on Plotting: http://xarray.pydata.org/en/latest/plotting.htm  Reference - [Pangeo Tutorial for 2018 UCAR SEA Conference](https://github.com/pangeo-data/pangeo-tutorial-sea-2018)<jupyter_code>%load_ext watermark
%watermark --iversion -g -m -v -u -d<jupyter_output>xarray     0.11.1+69.gc770eec3
pandas     0.24.1
zarr       2.2.0
numpy      1.16.1
matplotlib 3.0.2
last updated: 2019-04-02 

CPython 3.6.7
IPython 7.1.1

compiler   : GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)
system     : Darwin
release    : 17.7.0
machine    : x86_64
processor  : i386
CPU cores  : 8
interpreter: 64bit
Git hash   : 9a29dfaac3c5e3bf36f9d351581a72f9542ad7ed
{'fraction_non_alphanumeric': 0.1177807848443843, 'fraction_numerical': 0.021975642760487146, 'mean_word_length': 3.848071372343217, 'pattern_counts': {'":': 0, '<': 152, '<?xml version=': 0, '>': 157, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 37, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '38', 'dir': 'jupyter-structured-clean-dedup', 'id': '334745', 'license': 'Apache-2.0', 'n_tokens_mistral': 5885, 'n_tokens_neox': 5474, 'n_words': 1747, 'path': 'notebooks/xarray/01-xarray-introduction.ipynb', 'repo_name': 'roberttomas/NCAR-pangeo-tutorial', 'size': '18475'}