<jupyter_start><jupyter_text>Data Lookup and Loading* [**Sign up to the JupyterHub**](https://www.phenocube.org/) to run this notebook interactively from your browser* **Compatibility:** Notebook currently compatible with the Open Data Cube environments of the University of Wuerzburg* **Products used**: `Sentinel-2`* **Prerequisites**:  Users of this notebook should have a basic understanding of:    * How to run a [Jupyter notebook](01_jupyter_introduction.ipynb)    * The basic structure of the eo2cube [satellite datasets](02_eo2cube.ipynb)  BackgroundA "datacube" is a digital information architecture that specialises in hosting and cataloguing spatial information. Eo2cube is based on the [Open Data Cube](https://www.opendatacube.org/) infrastructure and specialises in storing remotely sensed data, particularly from Earth Observation satellites such as Landsat and Sentinel-2. The eo2cube contains multiple analyse ready satellite data "products". These data products are often composed of a range of "measurements" such as the suite of remote sensing band values. This notebook will focus on several straightforward ways to inspect the product,  measurement of a datacube, as well as to run data query and loading the data using a wide range of options, including resampling and interpolation. DescriptionThis notebook demonstrates how to **browse the available products/ measurements** stored within, and how to **load data** from the eo2cube datacube by using the `dc.load()` function. Topics covered include:* How to connect to the datacube* How to list all the products/ measurements* Loading data with the `dc.load()`function* Customising the `dc.load()`query* Tips and tricks to simplify the data loading process***  Load packagesThe `datacube` package is required to query the eo2cube datacube database and load the requested data. The pandas package is required to format tables. The `with_ui_cbk` function from `odc.ui` enables a progress bar when loading large amounts of data<jupyter_code>import datacube
import pandas as pd
from odc.ui import with_ui_cbk

# Set some configurations for displaying tables nicely
pd.set_option('display.max_colwidth', 200)
pd.set_option('display.max_rows', None)<jupyter_output><empty_output><jupyter_text>Datacube connectionAfter importing the datacube package, you are ready to connect to the eo2cube datacube. Users need to specify a name for their session, known as the app name. This name is generated by the user and is used to track down issues with database queries. It does not have any effect on the analysis. Use a short name that is consistent with the purpose of your notebook such as the way 03_Products_and_measurements has been used as the app name in this notebook.The resulting dc object provides access to all the data contained within the eo2cube datacube.<jupyter_code>dc = datacube.Datacube(app = '03_Products_and_measurement') #datacube connection<jupyter_output><empty_output><jupyter_text>List productsOnce a datacube instance has been created, users can explore the products and measurements stored within.The following cell lists all products that are currently available in the eo2cube datacube by using the dc.list_products() function.<jupyter_code>dc.list_products()<jupyter_output><empty_output><jupyter_text>List measurementsEvery product is associated with a range of available measurements. For the most time, these are the individual satellite bands.The dc.list_measurements() function can be used to interrogate the measurements associated with a given product (specified by the name column from the table above). For example, s2_l2a_bavaria refers to the Sentinel 2 ARD L2A scenes product for Bavaria.<jupyter_code>product = "s2_l2a"
measurements = dc.list_measurements()
measurements.loc[product]<jupyter_output><empty_output><jupyter_text>Explore Data availabilty In order to check which scenes are available, we can use the function DcViewer in order to interactivly explore the data<jupyter_code>from odc.ui import DcViewer

dcv = DcViewer(dc, time='2019-04-24', center=(53,13), zoom=8)
dcv<jupyter_output><empty_output><jupyter_text>Display MapBefore loading the data, it is a good idea to first check out the area of interest in a map. We can use display_map function for that.**Don't forget to import function `display_map` from `DEAPlotting` when you load the libraries.**<jupyter_code>from dea_tools.plotting import display_map
display_map([12.94 ,13.05],[53.88,53.94], crs = 'EPSG:4326')<jupyter_output><empty_output><jupyter_text>Loading data with the `dc.load()`functionLoading data from the datacube uses the `dc.load()` function.The function requires the following minimum arguments:* product: The data product to load (to see all available eo2cube products, see the [Products and measurements notebook](03_products_and_measurements.ipynb)).* x: The spatial region in the x dimension. By default, the x and y arguments accept queries in a geographical co-ordinate system WGS84, identified by the EPSG code 4326.* y: The spatial region in the y dimension. The dimensions longitude/latitude and x/y can be used interchangeably. It is also possible to use the extent of an imported shapefile as x/y (see the [notebook for working with vectordata](XX_vectordata.ipynb)).* time: The temporal extent. The time dimension can be specified using a tuple of datetime objects or strings in the “YYYY”, “YYYY-MM” or “YYYY-MM-DD” format.* output_crs: Reproject the data from the default or stored CRS into a specified CRS* resolution: Defines the output resolutionThe used product is defined by its name. The spatial extent is defined by lat/lon coordinates. The time intervall ist defined by the format "YYYY-MM-DD". The argument `progess_cbk` with the imported `with_ui_cbk()`function enables loading bar.<jupyter_code>data = dc.load(product= "s2_l2a",
               x= [12.94 ,13.05],
               y= [53.88,53.94],
               time= ("2021-08-01", "2021-08-15"),
               output_crs = "EPSG:32632",
               resolution = (-10, 10),
               #sat_orbit_state = 'descending',
               progress_cbk=with_ui_cbk())
    
data
data.B05.isel(time=[1,2,3]).plot(col="time")<jupyter_output><empty_output><jupyter_text>Customization of the `dc.load()` functionThe query created above only used the basic arguments of the `dc.load()` function. However, it is possible to customize your loading query by using multiple other arguments. A detailed description of every useable argument can be found in the [original documentation of the `dc.load()` function](https://datacube-core.readthedocs.io/en/latest/dev/api/generate/datacube.Datacube.load.html) of the Opendatacube initiative. In this notebook we will present the most used arguments that allow easy and effective querying of eo2cube products.  Specifying MeasurementsThe `measurements` argument provides the option to filter the desired product by its bands. This argument takes a list of the names of the desired measurement, as listed in the `dc.list_measurements()` function (see [notebook 03](03_products_and_measurements.ipynb)). If not provided, all stored measurements of the product will be loaded.In the following code only the `blue`, `green` and `red` bands of the specified product are added to the query. The resulting dataset will only include the defined measurements.<jupyter_code>data = dc.load(product= "s2_l2a",
               x= [12.94 ,13.05],
               y= [53.82,54.02],
               time= ("2020-01-01", "2020-01-15"),
               output_crs = "EPSG:32632",
               resolution = (-10, 10),
               measurements= ["blue", "green", "red"],
               progress_cbk=with_ui_cbk())

data<jupyter_output><empty_output><jupyter_text>Specifying CRS and Reprojecting To reproject the data from the default or stored CRS into a specified CRS, it is neccessary to define the `output_crs` and a new `resolution` for your data. The unit of the new `output_crs` and `resolution`must identical.<jupyter_code>data = dc.load(product= "s2_l2a",
               x= [12.94 ,13.05],
               y= [53.88,53.94],
               measurements= ["blue", "green", "red"],
               time= ("2020-01-01", "2020-01-15"),
               output_crs = "EPSG:4326",
               resolution = (-0.0002734477121776, 0.0002734477121776),
               progress_cbk=with_ui_cbk())

data
data.red.isel(time=1).plot()<jupyter_output><empty_output><jupyter_text>Spatial resamplingTo resample the data to another resolution, multiple resampling methods are available. You can define the `resampling` argument with one method from the following:`"nearest", "cubic", "bilinear", "cubic_spline", "lanczos", "average", "mode", "gauss", "max", "min", "med", "q1", "q3"`.By default, the `dc.load()` function is using "nearest neighbour" resampling, which allocates each new pixel with the value of the closest input pixel. However, for some data (eg. continuous data) this may not be the most appropiate choice.The example code below, resamples the specified product to a greater resolution of (-30, 30) meters using the bilinear interpolation method (`bilinear`). Note that the absolute number of pixels has been halved as the dataset has been resampled to twice the pixel size ([compare with the first query](first_query))<jupyter_code>data_10 = dc.load(product= "s2_l2a",
               x= [12.94 ,13.05],
               y= [53.88,53.94],
            time= ("2020-01-01", "2020-01-15"),
               measurements= ["blue", "green", "red"],
               output_crs = "EPSG:32632",
               resolution = (-10,10),
               progress_cbk=with_ui_cbk())
    
data_30 = dc.load(product= "s2_l2a",
              x= [12.94 ,13.05],
               y= [53.88,53.94],
               time= ("2020-01-01", "2020-01-15"),
               measurements= ["blue", "green", "red"],
               output_crs = "EPSG:32632",
               resolution = (-60,60),
               resampling = "bilinear",
               progress_cbk=with_ui_cbk())
import matplotlib.pyplot as plt

fig, axes = plt.subplots(ncols=2, figsize=(13,5))


data_10.red.isel(time=1).plot(ax=axes[0])
data_30.red.isel(time=1).plot(ax=axes[1])
plt.tight_layout()
plt.show()<jupyter_output><empty_output><jupyter_text>It is also possible to define different resampling methods for different measurements. This is useful if a dataset stores continuous data and categorical data. For example, the Sentinel-2 dataset has a scene classification layer (`scl`) which contains categorical values about the quality of a single pixel. If you want to resample your continuous data to another resolution with the `bilinear` method but also include the categorical `scl` band (categorical) it is best to use an interpolation method that does not modify the data directly for the resampling of the categorical `scl` band. This can be done by using Python Dictionaries as value for the `resampling` argument. The following code resample continuous bands with `bilinear` interpolation but also uses the `nearest` (which does not modify the data values) for the categorical `scl` band.<jupyter_code>data = dc.load(product= "s2_l2a",
               x= [12.94 ,13.05],
               y= [53.88,53.94],
               time= ("2020-04-01", "2020-04-15"),
               measurements= ["blue", "green", "red"],
               output_crs = "EPSG:32632",
               resolution = (-20, 20),
               resampling = {"*": "bilinear", "scl": "nearest"},
               progress_cbk=with_ui_cbk())


data<jupyter_output><empty_output><jupyter_text>Group by argumentThe `group_by` argument is a useful tool to combine/mosaic multiple scenes of one date (e.g. if the extent is visible in two sequential scenes of the satellite). Satellite datasets can have multiple observations per day with slightly different time stamps as the satellite collects data along its path. These observations can be combined by reducing the time dimension to the day level using `group_by = "solar_day"`. By default the scenes are simply copied together (e.g. the NA values of the first scene are filled with the values of the second scene). With the argument "fuse_func" the mosaicing can be controlled by the user (e.g. specify an interpolation method).This example loads data for the first week in April 2020. Since the extent of the AOI is to big to be covered by just one Sentinel 2 scene there are multiple scenes (from contiguous paths) available for each date. By using the `group_by = "solar_day"` argument the `dc.load()` function automatically mosaics all scenes of a single date together. In this example no special interpolation method was defined.<jupyter_code>data = dc.load(product= "s2_l2a",
               x= [12.94 ,13.05],
               y= [53.88,53.94],
               time= ("2020-04-01", "2020-04-15"),
               measurements= ["blue", "green", "red"],
               output_crs = "EPSG:32632",
               resolution = (-20, 20),
               group_by = "solar_day",
               progress_cbk=with_ui_cbk())

data
data.red.isel(time=[0,1,2]).plot(col="time")<jupyter_output><empty_output><jupyter_text>Define area interactively<jupyter_code>from odc.ui import select_on_a_map

geopolygon = select_on_a_map(height='500px')


data = dc.load(product= "s2_l2a",
               geopolygon = geopolygon,
               time= ("2020-04-01", "2020-04-15"),
               measurements= ["blue", "green", "red"],
               output_crs = "EPSG:32632",
               resolution = (-20, 20),
               group_by = "solar_day",
               progress_cbk=with_ui_cbk())

data
data.red.isel(time=2).plot()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0903244665302543, 'fraction_numerical': 0.03456591639871383, 'mean_word_length': 3.9619289340101522, 'pattern_counts': {'":': 2, '<': 53, '<?xml version=': 0, '>': 53, 'https://': 3, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '291649', 'license': 'MIT', 'n_tokens_mistral': 4209, 'n_tokens_neox': 3857, 'n_words': 1546, 'path': '3_Data_Cube_Basics/notebooks/.ipynb_checkpoints/01_data_lookup_and_loading-checkpoint.ipynb', 'repo_name': 'eo2cube/datacube_eagles', 'size': '13684'}