<jupyter_start><jupyter_text>Xarray-I: Data Structure   BackgroundIn the previous notebook, we experienced that the data we wanna access are loaded in a form called **`xarray.dataset`**. This is the form in which earth observation data are usually stored in a datacube.**`xarray`** is an open source project and Python package which offers a toolkit for working with ***multi-dimensional arrays*** of data. **`xarray.dataset`** is an in-memory representation of a netCDF (network Common Data Form) file. Understanding the structure of a **`xarray.dataset`** is the key to enable us work with these data. Thus, in this notebook, we are mainly dedicated to helping users of our datacube understand its data structure.Firstly let's come to the end stage of the previous notebook, where we have loaded a data product. The data product "s2_l2a_bavaria" is used as example in this notebook.  DescriptionThe following topics are convered in this notebook:* **What is inside a `xrray.dataset` (the structure)?*** **(Basic) Subset Dataset / DataArray*** **Reshape a Dataset**<jupyter_code>import datacube
import pandas as pd
from odc.ui import DcViewer 
from odc.ui import with_ui_cbk
import xarray as xr
import matplotlib.pyplot as plt

# Set config for displaying tables nicely
# !! USEFUL !! otherwise parts of longer infos won't be displayed in tables
pd.set_option("display.max_colwidth", 200)
pd.set_option("display.max_rows", None)

# Connect to DataCube
# argument "app" --- user defined name for a session (e.g. choose one matching the purpose of this notebook)
dc = datacube.Datacube(app = "nb_understand_ndArrays")
# Load Data Product
ds = dc.load(product= "s2_l2a",
             x = (-2.44, -2.42),
             y = (13.58, 13.6),
             output_crs = "EPSG:32734",
             time = ("2020-06-01", "2020-06-30"),
             measurements= ["blue", "green", "red","nir"],
             resolution = (-10,10),
             group_by = "solar_day",
             progress_cbk=with_ui_cbk())

ds
da = ds.to_array().rename({"variable":"band"})
print(da)
ds2 = da.to_dataset(dim="time")
ds2<jupyter_output><empty_output><jupyter_text>**What is inside a `xarray.dataset`?**The figure below is a diagramm depicting the structure of the **`xarray.dataset`** we've just loaded. Combined with the diagramm, we hope you may better interpret the texts below explaining the data strucutre of a **`xarray.dataset`**.![xarray data structure](https://live.staticflickr.com/65535/51083605166_70dd29baa8_k.jpg) As read from the output block, this dataset has three ***Data Variables*** , "blue", "green" and "red" (shown with colors in the diagramm), referring to individual spectral band.Each data variable can be regarded as a **multi-dimensional *Data Array*** of same structure; in this case, it is a **three-dimensional array** (shown as 3D Cube in the diagramm) where `time`, `x` and `y` are its ***Dimensions*** (shown as axis along each cube in the diagramm).In this dataset, there are 35 ***Coordinates*** under `time` dimension, which means there are 35 time steps along the `time` axis. There are 164 coordinates under `x` dimension and 82 coordinates under `y` dimension, indicating that there are 164 pixels along `x` axis and 82 pixels along `y` axis.As for the term ***Dataset***, it is like a *Container* holding all the multi-dimensional arrays of same structure (shown as the red-lined box holding all 3D Cubes in the diagramm).So this instance dataset has a spatial extent of 164 by 82 pixels at given lon/lat locations, spans over 35 time stamps and 3 spectral band.**In summary, *`xarray.dataset`* is substantially a container for high-dimensional *`DataArray`* with common attributes (e.g. crs) attached**, :* **Data Variables (`values`)**: **it's generally the first/highest dimension to subset from a high dimensional array.** Each `data variable` contains a multi-dimensional array of all other dimensions.* **Dimensions (`dims`)**: other dimensions arranged in hierachical order *(e.g. 'time', 'y', 'x')*.* **Coordinates (`coords`)**: Coordinates along each `Dimension` *(e.g. timesteps along 'time' dimension, latitudes along 'y' dimension, longitudes along 'x' dimension)** **Attributes (`attrs`)**: A dictionary(`dict`) containing Metadata. Now let's deconstruct the dataset we have just loaded a bit further to have things more clarified!:D * **To check existing dimensions of a dataset**<jupyter_code>ds.dims<jupyter_output><empty_output><jupyter_text>* **To check the coordinates of a dataset**<jupyter_code>ds.coords#['time']<jupyter_output><empty_output><jupyter_text>* **To check all coordinates along a specific dimension**<jupyter_code>ds.time
# OR
#ds.coords['time']<jupyter_output><empty_output><jupyter_text>* **To check attributes of the dataset**<jupyter_code>ds.attrs<jupyter_output><empty_output><jupyter_text>**Subset Dataset / DataArray*** **To select all data of "blue" band**<jupyter_code>ds.blue
# OR
#ds['blue']
# Only print pixel values
ds.blue.values<jupyter_output><empty_output><jupyter_text>* **To select blue band data at the first time stamp**<jupyter_code>ds.blue[0]<jupyter_output><empty_output><jupyter_text>* **To select blue band data at the first time stamp while the latitude is the largest in the defined spatial extent**<jupyter_code>ds.blue[0][0]<jupyter_output><empty_output><jupyter_text>* **To select the upper-left corner pixel**<jupyter_code>ds.blue[0][0][0]<jupyter_output><empty_output><jupyter_text>**subset dataset with `isel` vs. `sel`*** Use `isel` when subsetting with **index*** Use `sel` when subsetting with **labels** * **To select data of all spectral bands at the first time stamp**<jupyter_code>ds.isel(time=[0])<jupyter_output><empty_output><jupyter_text>* **To select data of all spectral bands of year 2020**<jupyter_code>ds.sel(time='2020-06')
#print(ds.sel(time='2019'))<jupyter_output><empty_output><jupyter_text>***Tip: More about indexing and sebsetting Dataset or DataArray is presented in the [Notebook_05](https://github.com/eo2cube/eo2cube_notebooks/blob/main/get_started/intro_to_eo2cube/05_xarrayII.ipynb).***  **Reshape Dataset*** **Convert the Dataset (subset to 2019) to a *4-dimension* DataArray**<jupyter_code>da = ds.sel(time='2020-06').to_array().rename({"variable":"band"})
da<jupyter_output><empty_output><jupyter_text>* **Convert the *4-dimension* DataArray back to a Dataset by setting the "time" as DataVariable (reshaped)**![ds_reshaped](https://live.staticflickr.com/65535/51151694092_ca550152d6_o.png)<jupyter_code>ds_reshp = da.to_dataset(dim="time")
print(ds_reshp)<jupyter_output><xarray.Dataset>
Dimensions:              (band: 4, y: 263, x: 259)
Coordinates:
  * y                    (y) float64 1.163e+07 1.163e+07 ... 1.163e+07 1.163e+07
  * x                    (x) float64 -2.101e+06 -2.101e+06 ... -2.099e+06
    spatial_ref          int32 32734
  * band                 (band) <U5 'blue' 'green' 'red' 'nir'
Data variables:
    2020-06-03 10:47:23  (band, y, x) uint16 1680 1686 1686 ... 2968 2942 2988
    2020-06-08 10:47:19  (band, y, x) uint16 957 955 955 961 ... 2540 2446 2640
    2020-06-13 10:47:23  (band, y, x) uint16 971 932 932 919 ... 2718 2686 2810
    2020-06-18 10:47:19  (band, y, x) uint16 9856 9864 9864 ... 10440 10472
    2020-06-23 10:47:22  (band, y, x) uint16 779 716 716 753 ... 2436 2484 2562
    2020-06-28 10:47:19  (band, y, x) uint16 1015 993 993 ... 2840 2840 3002
Attributes:
    crs:           EPSG:32734
    grid_mapping:  spatial_ref
{'fraction_non_alphanumeric': 0.12306253340459647, 'fraction_numerical': 0.061063602351683594, 'mean_word_length': 4.762124711316397, 'pattern_counts': {'":': 2, '<': 54, '<?xml version=': 0, '>': 53, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '496454', 'license': 'MIT', 'n_tokens_mistral': 2707, 'n_tokens_neox': 2377, 'n_words': 840, 'path': '3_Data_Cube_Basics/notebooks/.ipynb_checkpoints/03_xarrayI_data_structure-checkpoint.ipynb', 'repo_name': 'eo2cube/datacube_eagles', 'size': '7484'}