<jupyter_start><jupyter_text>Crunch Crunching as defined in `eodag` is a way to filter the EO products contained in a [SearchResult](../../api_reference/searchresult.rsteodag.api.search_result.SearchResult) object. Several filters are available and further described in this document. A [SearchResult](../../api_reference/searchresult.rsteodag.api.search_result.SearchResult) has a [crunch()](../../api_reference/searchresult.rsteodag.api.search_result.SearchResult.crunch) method that requires a filter instance as an argument, itself initialized with a dictionary that contains the required parameters. According to the filter used, some more kwargs may need to be passed to [crunch()](../../api_reference/searchresult.rsteodag.api.search_result.SearchResult.crunch). The filters return a `list` of [EOProduct](../../api_reference/eoproduct.rsteodag.api.product._product.EOProduct)s.  Setup Results obtained from a search of *Sentinel 2 Level-1C* products over France in March 2021 are loaded in a [SearchResult](../../api_reference/searchresult.rsteodag.api.search_result.SearchResult).<jupyter_code>from eodag import EODataAccessGateway
dag = EODataAccessGateway()
search_results = dag.deserialize("data/crunch_search_results.geojson")
print(f"This SearchResult stores {len(search_results)} products.")
# This code cell has a special metadata entry: "nbsphinx": "hidden"
# That hides it when the documentation is built with nbsphinx/sphinx.

# Uncomment these lines to regenerate the GeoJSON file used in this notebook.

#search_results, _ = dag.search(
#    productType="S2_MSI_L1C",
#    start="2021-03-01",
#    end="2021-03-31",
#    geom={"lonmin": 1, "latmin": 45, "lonmax": 5, "latmax": 47},
#    items_per_page=50
#)
#dag.serialize(search_results, "data/crunch_search_results.geojson")<jupyter_output><empty_output><jupyter_text>The original search geometry is used throughout the notebook as long as with its representation as a a `shapely` object which is easier to map with `folium`.<jupyter_code>original_search_geometry = {"lonmin": 1, "latmin": 45, "lonmax": 5, "latmax": 47}
import shapely

search_geometry = shapely.geometry.box(
    original_search_geometry["lonmin"],
    original_search_geometry["latmin"],
    original_search_geometry["lonmax"],
    original_search_geometry["latmax"],
)
# To create interactive maps
import folium

def create_search_result_map(search_results, extent):
    """Small utility to create an interactive map with folium
    that displays an extent in red and EO Producs in blue"""
    fmap = folium.Map([46, 3], zoom_start=6)
    folium.GeoJson(
        extent,
        style_function=lambda x: dict(color="red")
    ).add_to(fmap)
    folium.GeoJson(
        search_results
    ).add_to(fmap)
    return fmap<jupyter_output><empty_output><jupyter_text>Filter by start and end date [FilterDate](../../plugins_reference/generated/eodag.plugins.crunch.filter_date.FilterDate.rsteodag.plugins.crunch.filter_date.FilterDate) allows to filter out products that are older than a start date (optional) or more recent than an end date (optional).<jupyter_code>from eodag.plugins.crunch.filter_date import FilterDate
filtered_products = search_results.crunch(
    FilterDate(dict(start="2021-03-25", end="2021-03-29"))
)
print(f"{len(search_results) - len(filtered_products)} products were filtered out by the date filter.")<jupyter_output><empty_output><jupyter_text>Filter by geometry [FilterOverlap](../../plugins_reference/generated/eodag.plugins.crunch.filter_overlap.FilterOverlap.rsteodag.plugins.crunch.filter_overlap.FilterOverlap) allows to filter out products that:* whose overlap area with a geometry is less than a percentage of their area* are not *within* a geometry* do not *contain* a geometry* do not *intersect* with a geometryTo execute a [FilterOverlap](../../plugins_reference/generated/eodag.plugins.crunch.filter_overlap.FilterOverlap.rsteodag.plugins.crunch.filter_overlap.FilterOverlap), its instance must be created by passing a dictionary with either:* `minimum_overlap` set to a number between 0 and 100. `within`, `contains` and `intersects` cannot be used in that case.* **One** of `within`, `contains` and `intersects` (they are mutually exclusive) set to True. `minimum_overlap` cannot be used in that case.Additionally, a geometry (shapely geometry, bounding box as a dictionary or a list) must be passed through the `geometry` parameter.The examples below show how [FilterOverlap](../../plugins_reference/generated/eodag.plugins.crunch.filter_overlap.FilterOverlap.rsteodag.plugins.crunch.filter_overlap.FilterOverlap) filter out products. The original products will be displayed in blue and the filtered products in green.<jupyter_code>from eodag.plugins.crunch.filter_overlap import FilterOverlap<jupyter_output><empty_output><jupyter_text>All the products are displayed on the next map. As it can be observed, they all intersect with the search geometry.<jupyter_code>create_search_result_map(search_results, search_geometry)<jupyter_output><empty_output><jupyter_text>The next two examples show how `minimum_overlap` affects the filter, with its value (i.e. percentage) set to 10 and 50%.<jupyter_code>filtered_products = search_results.crunch(
    FilterOverlap(dict(minimum_overlap=10)),
    geometry=search_geometry
)
print(f"{len(search_results) - len(filtered_products)} products were filtered out by the geometry filter.")
fmap = create_search_result_map(search_results, search_geometry)
folium.GeoJson(
    filtered_products,
    style_function=lambda x: dict(color="green")
).add_to(fmap)
fmap
filtered_products = search_results.crunch(
    FilterOverlap(dict(minimum_overlap=50)),
    geometry=search_geometry
)
print(f"{len(search_results) - len(filtered_products)} products were filtered out by the geometry filter.")
fmap = create_search_result_map(search_results, search_geometry)
folium.GeoJson(
    filtered_products,
    style_function=lambda x: dict(color="green")
).add_to(fmap)
fmap<jupyter_output><empty_output><jupyter_text>More and more products are filtered out when `minimum_overlap` increases. The next parameter given as an example is `within`, it is actually equivalent to setting `minimum_overlap` to 100.<jupyter_code>filtered_products = search_results.crunch(
    FilterOverlap(dict(within=True)),
    geometry=search_geometry
)
print(f"{len(search_results) - len(filtered_products)} products were filtered out by the geometry filter.")
fmap = create_search_result_map(search_results, search_geometry)
# Create a layer that represents the search area in red
folium.GeoJson(
    filtered_products,
    style_function=lambda x: dict(color="green")
).add_to(fmap)
fmap<jupyter_output><empty_output><jupyter_text>All the products not withing the read area are correctly filtered out by [FilterOverlap](../../plugins_reference/generated/eodag.plugins.crunch.filter_overlap.FilterOverlap.rsteodag.plugins.crunch.filter_overlap.FilterOverlap). A new geometry is created in order to test the next parameter `intersects`.<jupyter_code>from shapely.geometry import Polygon
shifted_geom = Polygon([[4, 44], [9, 44], [9, 48], [4, 48], [4, 48]])
filtered_products = search_results.crunch(
    FilterOverlap(dict(intersects=True)),
    geometry=shifted_geom
)
print(f"{len(search_results) - len(filtered_products)} products were filtered out by the geometry filter.")
fmap = create_search_result_map(search_results, shifted_geom)
# Create a layer that represents a smaller area in red
folium.GeoJson(
    filtered_products,
    style_function=lambda x: dict(color="green")
).add_to(fmap)
fmap<jupyter_output><empty_output><jupyter_text>The products that do not intersect with the red area are correctly filtered out. Finally another new geometry is created to test the parameter `contains`.<jupyter_code>small_geom = Polygon([[3.2, 44.4], [3.7, 44.4], [3.7, 44.9], [3.2, 44.9], [3.2, 44.4]])
filtered_products = search_results.crunch(
    FilterOverlap(dict(contains=True)),
    geometry=small_geom
)
print(f"{len(search_results) - len(filtered_products)} products were filtered out by the geometry filter.")
fmap = create_search_result_map(search_results, small_geom)
# Create a layer that represents a smaller area in red
folium.GeoJson(
    filtered_products,
    style_function=lambda x: dict(color="green")
).add_to(fmap)
fmap<jupyter_output><empty_output><jupyter_text>The only product preserved is the one that contains the red area.  Filter by property [FilterProperty](../../plugins_reference/generated/eodag.plugins.crunch.filter_property.FilterProperty.rsteodag.plugins.crunch.filter_property.FilterProperty) evaluates a single property of all the products against a value (e.g. cloud cover less than 10). The dictionary it requires should contain:* A single property name from [EOProduct](../../api_reference/eoproduct.rsteodag.api.product._product.EOProduct)`.properties` and its tested value, e.g. `dict(cloudCover=10)` or `dict(storageStatus="ONLINE")`* One (optional) operator among `lt` (=), `gt` (>). `eq` by default.<jupyter_code>from eodag.plugins.crunch.filter_property import FilterProperty
filtered_products = search_results.crunch(
    FilterProperty(dict(cloudCover=1, operator="lt"))
)
print(f"{len(search_results) - len(filtered_products)} products were filtered out by the property filter.")<jupyter_output><empty_output><jupyter_text>List comprehensions over a collection of EO products are useful to quickly extract their properties, and here to check that the filter correctly filtered the products.<jupyter_code>all([p.properties["cloudCover"] < 1 for p in filtered_products])<jupyter_output><empty_output><jupyter_text>Filter the latest products intersecting a geometry [FilterLatestIntersect](../../plugins_reference/generated/eodag.plugins.crunch.filter_latest_intersect.FilterLatestIntersect.rsteodag.plugins.crunch.filter_latest_intersect.FilterLatestIntersect) does the following:1. it sorts the products by date, from the newest to the oldest2. it filters out products that do not intersect with a requested geometry (a dictionary bounding box)3. it stops early if the requested geometry is 100% covered by the products, if not, it returns the result of 2.This results in getting the most recent products that intersect (or completely cover) a given geometry.<jupyter_code>from eodag.plugins.crunch.filter_latest_intersect import FilterLatestIntersect
filtered_products = search_results.crunch(
    FilterLatestIntersect({}),
    geometry=original_search_geometry
)
print(f"{len(search_results) - len(filtered_products)} products were filtered out by the property filter.")
from shapely import geometry

fmap = create_search_result_map(search_results, search_geometry)
# Create a layer that represents a smaller area in red
folium.GeoJson(
    filtered_products,
    style_function=lambda x: dict(color="green")
).add_to(fmap)
fmap<jupyter_output><empty_output><jupyter_text>The map shows that the area is fully covered by products. The filtered products are indeed the most recent ones.<jupyter_code>[p.properties["startTimeFromAscendingNode"] for p in filtered_products][::10]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09923732615522655, 'fraction_numerical': 0.01148497083894123, 'mean_word_length': 6.223590408295528, 'pattern_counts': {'":': 9, '<': 54, '<?xml version=': 0, '>': 54, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '32948', 'license': 'Apache-2.0', 'n_tokens_mistral': 3415, 'n_tokens_neox': 3292, 'n_words': 959, 'path': 'docs/notebooks/api_user_guide/6_crunch.ipynb', 'repo_name': 'drnextgis/eodag', 'size': '11145'}