<jupyter_start><jupyter_text>Overview This overview demonstrates the basic capabilities of `eodag` in a simple worflow, whose steps are introduced in more details in the following pages.The workflow consists of the following steps:1. [Configure](Configure): `eodag` is configured to use the provider *PEPS* (registering to *PEPS* is required to download the products, see [how to register to a provider](../../getting_started_guide/register.rst))2. [Search](Search): *Sentinel 2 Level-1C* products (i.e. images) are searched for over an area in France in March 2021.3. [Crunch](Crunch): A decicated filter - that we call *cruncher* - provided by `eodag`  is used to select products with a cloud cover less than 10%.4. [Serialize](Serialize): Save the filtered products as a GeoJSON file.4. [Download](Download): The products are downloaded.5. [Post-process](Post-process): [eodag-cube](https://github.com/CS-SI/eodag-cube) is an external package that is used to access a product's data, it is going to be used to calculate the NDVI of a product.  Configure Let us first create a workspace directory to save products downloaded during this workflow.<jupyter_code>workspace = 'eodag_workspace_overview'
if not os.path.isdir(workspace):
    os.mkdir(workspace)<jupyter_output><empty_output><jupyter_text>Now we will configure `eodag` to be able to download using *PEPS*. For that we need to fill our credentials:* in the user configuration file `~/.config/eodag.eodag.yml`:```yamlpeps:    priority:  Lower value means lower priority (Default: 1)    search:   Search parameters configuration    download:        extract:   whether to extract the downloaded products (true or false).        outputs_prefix:  where to store downloaded products.        dl_url_params:   additional parameters to pass over to the download url as an url parameter        delete_archive:  whether to delete the downloaded archives (true or false, Default: true).    auth:        credentials:            username: PLEASE_CHANGE_ME            password: <PASSWORD>```* or using environment variables: (we also set `outputs_prefix`, the directory where to store downloaded products)<jupyter_code>import os
os.environ["EODAG__PEPS__AUTH__CREDENTIALS__USERNAME"] = "PLEASE_CHANGE_ME"
os.environ["EODAG__PEPS__AUTH__CREDENTIALS__PASSWORD"] = "<PASSWORD>"
os.environ["EODAG__PEPS__DOWNLOAD__OUTPUTS_PREFIX"] = os.path.abspath(workspace)<jupyter_output><empty_output><jupyter_text>Logging is then activated with the [setup_logging()](../../api_reference/utils.rsteodag.utils.logging.setup_logging) method. It's a useful way to see what `eodag` does under the hood, e.g. requesting the provider, adapting the response. It's also useful to detect when things go wrong, and, if relevant, create an [issue on GitHub](https://github.com/CS-SI/eodag/issues) with the log messages.<jupyter_code>from eodag import setup_logging
setup_logging(2)  # 3 for even more information<jupyter_output><empty_output><jupyter_text>The next object to import, and this is certainly one of the most important objects provided by `eodag`, is the [EODataAccessGateway](../../api_reference/core.rsteodag.api.core.EODataAccessGateway) class. The creation of a single instance of this class is enough in a workflow, it is going to take care of configuring the providers, exposing the products configured off-the-shelf by `eodag`, and many more things.<jupyter_code>from eodag import EODataAccessGateway
dag = EODataAccessGateway()<jupyter_output>2021-04-20 18:02:39,915-15s eodag.config                     [INFO    ] Loading user configuration from: /home/maxime/.config/eodag/eodag.yml
2021-04-20 18:02:39,953-15s eodag.core                       [INFO    ] Locations configuration loaded from /home/maxime/.config/eodag/locations.yml<jupyter_text>`eodag` stores an internal catalog of products it makes easily accessible. *Sentinel 2 Level-1C* products are designated with the identifier *S2_MSI_L1C*. It's possible to check that it's made available by *PEPS* with the method [list_product_types()](../../api_reference/core.rsteodag.api.core.EODataAccessGateway.list_product_types) which returns a list of dictionnaries, each one of them containing general metadata (`eodag`'s product type ID, platform, sensor type, etc.).<jupyter_code>[product_type["ID"] for product_type in dag.list_product_types("peps")]<jupyter_output><empty_output><jupyter_text>The method [available_providers()](../../api_reference/core.rsteodag.api.core.EODataAccessGateway.available_providers) allows to get the list of providers that make available a given product.<jupyter_code>dag.available_providers("S2_MSI_L1C")<jupyter_output><empty_output><jupyter_text>Finally *PEPS* is declared as the preferred provider, which means that `eodag` will look for products with this provider (all the providers are pre-configured).<jupyter_code>dag.set_preferred_provider("peps")<jupyter_output><empty_output><jupyter_text>Search The [EODataAccessGateway](../../api_reference/core.rsteodag.api.core.EODataAccessGateway) class provides three search methods which have a similar signature but behave in different ways:[search()](../../api_reference/core.rsteodag.api.core.EODataAccessGateway.search) returns a tuple with:* a [SearchResult](../../api_reference/searchresult.rsteodag.api.search_result.SearchResult) that stores the products obtained from a given **page** (default: `page=1`) and a given maximum **number of items per page** (default: `items_per_page=20`)* an integer that is the **estimated** total number of products matching the search criteriaNote[search()](../../api_reference/core.rsteodag.api.core.EODataAccessGateway.search) returns a *paginated result*. Pagination is the act of serving the result of a search page by page, each page contaning a maximum number of items. This is ubiquitous among search engines, e.g. a Google search displays in the browser 10 results per page. Pagination is built deep into the implementation of `eodag` since most providers return a paginated result.NoteA search made with the [search()](../../api_reference/core.rsteodag.api.core.EODataAccessGateway.search) method with its default parameter often ends up with a [SearchResult](../../api_reference/searchresult.rsteodag.api.search_result.SearchResult) containing 20 products and a significantly greater estimated total number of products available. The parameter `items_per_page` can be set to a higher value, or, the [search_all()](../../api_reference/core.rsteodag.api.core.EODataAccessGateway.search_all) method can be used instead.WarningThe second element returned by [search()](../../api_reference/core.rsteodag.api.core.EODataAccessGateway.search) is the **estimated** total number of products matching the search criteria, since, unfortunately, all the providers do not return the **exact** total number. [search_all()](../../api_reference/core.rsteodag.api.core.EODataAccessGateway.search_all) returns a [SearchResult](../../api_reference/searchresult.rsteodag.api.search_result.SearchResult) that contains **all** the products matching the search criteria. It takes the pain away from thinking about pagination.[search_iter_page()](../../api_reference/core.rsteodag.api.core.EODataAccessGateway.search_iter_page) is a generator that returns a [SearchResult](../../api_reference/searchresult.rsteodag.api.search_result.SearchResult) page per page. To get the *S2_MSI_L1C* products looked for, [search()](../../api_reference/core.rsteodag.api.core.EODataAccessGateway.search) is used first as it's handy to obtain some products and inspect them. Then [search_all()](../../api_reference/core.rsteodag.api.core.EODataAccessGateway.search_all) is used to make all the products offered by *PEPS* are collected.<jupyter_code>search_criteria = {
    "productType": "S2_MSI_L1C",
    "start": "2021-03-01",
    "end": "2021-03-31",
    "geom": {"lonmin": 1, "latmin": 43, "lonmax": 2, "latmax": 44}
}<jupyter_output><empty_output><jupyter_text>The search criteria in this example consists of:* `productType`: `eodag`'s product type* `start` and `end`: the start and end sensing datetimes (UTC)* `geom`: the region of interest, which can be defined in different ways (`list`, `dict`, WKT string, `shapely.geometry`).<jupyter_code>products_first_page, estimated_total_number = dag.search(**search_criteria)
print(f"Got {len(products_first_page)} products and an estimated total number of {estimated_total_number} products.")<jupyter_output>Got 20 products and an estimated total number of 48 products.<jupyter_text>A [SearchResult](../../api_reference/searchresult.rsteodag.api.search_result.SearchResult) contains a number of [EOProduct](../../api_reference/eoproduct.rsteodag.api.product._product.EOProduct). Each one of them has a `properties` attribute that is a dictionnary storing the product's metadata.<jupyter_code>products_first_page
one_product = products_first_page[0]
one_product.properties.keys()<jupyter_output><empty_output><jupyter_text>A product also has a `location` attribute. After being searched, it is equal to the *remote location* of the product, that will be used later by `eodag` to download it.<jupyter_code>one_product.location<jupyter_output><empty_output><jupyter_text>All the products are now obtained from *PEPS*.<jupyter_code>all_products = dag.search_all(**search_criteria)
print(f"Got a total number of {len(all_products)} products.")<jupyter_output>Got a total number of 48 products.<jupyter_text>Crunch Crunching as defined in `eodag` is a way to filter the products contained in a [SearchResult](../../api_reference/searchresult.rsteodag.api.search_result.SearchResult) object. Several crunchers/filters are available (e.g. [FilterProperty](../../plugins_reference/generated/eodag.plugins.crunch.filter_property.FilterProperty.rsteodag.plugins.crunch.filter_property.FilterProperty) to filter products products according to one of their properties). The following example show how to filter products to keep only those twhose cloud cover is less than 10%.<jupyter_code>from eodag.plugins.crunch.filter_property import FilterProperty
filtered_products = all_products.crunch(FilterProperty({"cloudCover": 10, "operator": "lt"}))
print(f"Got now {len(filtered_products)} products after filtering.")<jupyter_output>Got now 10 products after filtering.<jupyter_text>Serialize The method [serialize()](../../api_reference/core.rsteodag.api.core.EODataAccessGateway.serialize) allows to save a [SearchResult](../../api_reference/searchresult.rsteodag.api.search_result.SearchResult) as a *GeoJSON* file.<jupyter_code>filtered_prods_filepath = dag.serialize(filtered_products, filename=os.path.join(workspace, "filtered_products.geojson"))<jupyter_output><empty_output><jupyter_text>This method can come in handy to save the state of a search and restore it later with [deserialize_and_register()](../../api_reference/core.rsteodag.api.core.EODataAccessGateway.deserialize_and_register).<jupyter_code># restored_filtered_prods = dag.deserialize_and_register(filtered_prods_filepath)<jupyter_output><empty_output><jupyter_text>Download Before downloading any product, it can be useful to have a quick look at them. EO products usually offer a *quicklook* image, a low resolution by-product of the original data. An `EOProduct` has a [get_quicklook](../../api_reference/eoproduct.rsteodag.api.product._product.EOProduct.get_quicklook) method that takes care of downloading the quicklook image and returns its path. `matplotlib` can be used here to display the images collected.<jupyter_code>import matplotlib.pyplot as plt
import matplotlib.image as mpimg

quicklooks_dir = os.path.join(workspace, "quicklooks")
if not os.path.isdir(quicklooks_dir):
    os.mkdir(quicklooks_dir)

fig = plt.figure(figsize=(10, 8))
for i, product in enumerate(filtered_products, start=1):

    # This line takes care of downloading the quicklook
    quicklook_path = product.get_quicklook(base_dir=quicklooks_dir)
    
    img = mpimg.imread(quicklook_path)
    ax = fig.add_subplot(3, 4, i)
    ax.set_title(i)
    plt.imshow(img)
plt.tight_layout()<jupyter_output><empty_output><jupyter_text>[EOProduct](../../api_reference/eoproduct.rsteodag.api.product._product.EOProduct)s can either be downloaded individually with [download()](../../api_reference/core.rsteodag.api.core.EODataAccessGateway.download) or together with [download_all()](../../api_reference/core.rsteodag.api.core.EODataAccessGateway.download_all) from a [SearchResult](../../api_reference/searchresult.rsteodag.api.search_result.SearchResult). The last image is going to be downloaded, it is cloud-free and has no no-data pixel.<jupyter_code>product_to_download = filtered_products[-1]
product_path = dag.download(product_to_download)
product_path<jupyter_output>2021-04-20 18:02:48,884-15s eodag.plugins.download.base      [INFO    ] Download url: https://peps.cnes.fr/resto/collections/S2ST/f913081f-eead-5709-ab65-cbaa1884150c/download<jupyter_text>The `location` property of this product now points to a local path.<jupyter_code>product_to_download.location<jupyter_output><empty_output><jupyter_text>Post-process Now the product is downloaded, it can be post-processed with softwares such as [Sen2Cor](http://step.esa.int/main/snap-supported-plugins/sen2cor/) or [SNAP](http://step.esa.int/main/download/snap-download/).At some point `eodag` had some capabilities to directly post-process a product, i.e. to access its data. These capabilities, which relied on [rasterio](https://rasterio.readthedocs.io/en/latest/), have been ported to the Python package [eodag-cube](https://github.com/CS-SI/eodag-cube) to avoid the heavy dependencies associated with  *GDAL* in particular. Installing this package is enough to benefit from its capabilities, it is going to extend [EOProduct](../../api_reference/eoproduct.rsteodag.api.product._product.EOProduct) with a `get_data()` method which returns a product's image band as a `xarray.DataArray`.The capabilities of [eodag-cube](https://github.com/CS-SI/eodag-cube) are used hereafter to compute the NDVI of the downloaded product over a sub-extent of the original product (this is actually Toulouse, France). Warning[eodag-cube](https://github.com/CS-SI/eodag-cube) needs to be installed to run correcly the following code.<jupyter_code>crs = "epsg:4326"
resolution = 0.0001
sub_extent = (1.435905, 43.586857, 1.458907, 43.603827)
VIR = product_to_download.get_data(band="B04", extent=sub_extent, crs=crs, resolution=resolution)
NIR = product_to_download.get_data(band="B08", extent=sub_extent, crs=crs, resolution=resolution)
NDVI = (NIR - VIR * 1.) / (NIR + VIR)<jupyter_output><empty_output><jupyter_text>The NDVI can quickly be plotted directly with `xarray` which uses `matplotlib` under the hood.<jupyter_code>NDVI.plot(cmap="RdYlGn", center=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10576015108593012, 'fraction_numerical': 0.013422366113584244, 'mean_word_length': 5.736483416628805, 'pattern_counts': {'":': 10, '<': 78, '<?xml version=': 0, '>': 78, 'https://': 7, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 36, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '534539', 'license': 'Apache-2.0', 'n_tokens_mistral': 4626, 'n_tokens_neox': 4449, 'n_words': 1327, 'path': 'docs/notebooks/api_user_guide/1_overview.ipynb', 'repo_name': 'CS-SI/eodag', 'size': '14838'}