<jupyter_start><jupyter_text>05 - Google Earth Engine>>  Amazon Web Services - Open Data Registry  About [The Registry of Open Data on AWS]("https://registry.opendata.aws/?search=tags:gis,earth%20observation,events,mapping,meteorological,environmental,transportation") is a registry of open geospatial datasets on Amazon Web Services.Large volumes of geospatial data are made available as S3 storage buckets and are available for download or to directly load it in AWS processing services.Check out the program [Earth on AWS](https://aws.amazon.com/earth/) as well.  Data There is a variety of different `geospatial` data available. So far, the following Copernicus data are available:* [Sentinel-1](https://registry.opendata.aws/sentinel-1/)* [Sentinel-2](https://registry.opendata.aws/sentinel-2/)* [Sentinel-3](https://registry.opendata.aws/sentinel-3/)* [Sentinel-5P Level 2](https://registry.opendata.aws/sentinel5p/)* [ECMWF ERA-5 climate reanalysis](https://registry.opendata.aws/ecmwf-era5/)  How to retrieve data? [boto3](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html) is the Amazon Web Services (AWS) SDK for Python in order e.g. to access data on AWS S3 storage buckets.Follow the steps below for an example on how to use boto3 to retrieve data.   Load required libraries<jupyter_code>import boto3
import botocore<jupyter_output><empty_output><jupyter_text>Example to retrieve data with `boto3`  1. Initiate the a `boto3 client` with `boto3.client()` and define the bucket of your interest. Initiate the a `s3` client and define `meeo-s5p` as your bucket of interest.<jupyter_code>client = boto3.client('s3', config=botocore.client.Config(signature_version=botocore.UNSIGNED))
s5p_bucket = 'meeo-s5p'<jupyter_output><empty_output><jupyter_text>2. Create a `paginator` and iterate of the results from the API request `Paginators` are a feature of boto3 that act as an abstraction over the process of iterating over an entire result set of a truncated API operation. You must call the `paginate()` method of a Paginator in order to iterate over the pages of API operation results.<jupyter_code>paginator = client.get_paginator('list_objects_v2')
results = paginator.paginate(Bucket=s5p_bucket, Delimiter='/')
results<jupyter_output><empty_output><jupyter_text>But the `paginator` object is just a `PageIterator`. In order to see the structure in which the data is organised on the S3 bucket, you have to iterate over the object:<jupyter_code>for prefix in results.search('CommonPrefixes'):
    print(prefix.get('Prefix'))<jupyter_output>COGT/
NRTI/
OFFL/
RPRO/<jupyter_text>3. Paginate over the PageIteror with a `Prefix` parameter Instead of the `Delimiter` keyword, the `paginate` function accepts also a `Prefix` parameter, which can be used to filter the paginated results by prefix on the server-side before sending them to the client:<jupyter_code>prefix=('COGT/OFFL/L3__NO2__')
pages = paginator.paginate(Bucket=s5p_bucket, Prefix=prefix)
pages<jupyter_output><empty_output><jupyter_text>The next step is now to iterate over the `Contents` of each page of the PageIterator `pages`. Since the resulting object are dictionaries, we get a much cleaner output by just selecting the `dictionary key`. You see that the data are daily aggregates and for each day, there are three differenct COG files with the following endings:- `_4326`- `_mask50_4326`- `_mask75_4326`<jupyter_code>for page in pages:
    for obj in page['Contents']:
#        print(obj)
        print(obj.get('Key'))<jupyter_output>COGT/OFFL/L3__NO2___/2020/02/S5P_OFFL_L3__NO2____20200225_PRODUCT_daily_nitrogendioxide_tropospheric_column_4326.tif
COGT/OFFL/L3__NO2___/2020/02/S5P_OFFL_L3__NO2____20200225_PRODUCT_daily_nitrogendioxide_tropospheric_column_mask50_4326.tif
COGT/OFFL/L3__NO2___/2020/02/S5P_OFFL_L3__NO2____20200225_PRODUCT_daily_nitrogendioxide_tropospheric_column_mask75_4326.tif
COGT/OFFL/L3__NO2___/2020/02/S5P_OFFL_L3__NO2____20200226_PRODUCT_daily_nitrogendioxide_tropospheric_column_4326.json
COGT/OFFL/L3__NO2___/2020/02/S5P_OFFL_L3__NO2____20200226_PRODUCT_daily_nitrogendioxide_tropospheric_column_4326.tif
COGT/OFFL/L3__NO2___/2020/02/S5P_OFFL_L3__NO2____20200226_PRODUCT_daily_nitrogendioxide_tropospheric_column_mask50_4326.tif
COGT/OFFL/L3__NO2___/2020/02/S5P_OFFL_L3__NO2____20200226_PRODUCT_daily_nitrogendioxide_tropospheric_column_mask75_4326.tif
COGT/OFFL/L3__NO2___/2020/02/S5P_OFFL_L3__NO2____20200227_PRODUCT_daily_nitrogendioxide_tropospheric_column_4326.json
COGT/OFFL/L3__NO2___/2020/02/S5P_O[...]<jupyter_text>4. Store the `dictionary_keys` in a Python list Let us store the dictionary keys in a list with the name `key_list`.<jupyter_code>i=0
key_list=[]
for page in pages:
    for obj in page['Contents']:
        print(i)
        print(obj.get('Key'))
        key_list.append(obj.get('Key'))
        i=i+1<jupyter_output>0
COGT/OFFL/L3__NO2___/2020/02/S5P_OFFL_L3__NO2____20200225_PRODUCT_daily_nitrogendioxide_tropospheric_column_4326.tif
1
COGT/OFFL/L3__NO2___/2020/02/S5P_OFFL_L3__NO2____20200225_PRODUCT_daily_nitrogendioxide_tropospheric_column_mask50_4326.tif
2
COGT/OFFL/L3__NO2___/2020/02/S5P_OFFL_L3__NO2____20200225_PRODUCT_daily_nitrogendioxide_tropospheric_column_mask75_4326.tif
3
COGT/OFFL/L3__NO2___/2020/02/S5P_OFFL_L3__NO2____20200226_PRODUCT_daily_nitrogendioxide_tropospheric_column_4326.json
4
COGT/OFFL/L3__NO2___/2020/02/S5P_OFFL_L3__NO2____20200226_PRODUCT_daily_nitrogendioxide_tropospheric_column_4326.tif
5
COGT/OFFL/L3__NO2___/2020/02/S5P_OFFL_L3__NO2____20200226_PRODUCT_daily_nitrogendioxide_tropospheric_column_mask50_4326.tif
6
COGT/OFFL/L3__NO2___/2020/02/S5P_OFFL_L3__NO2____20200226_PRODUCT_daily_nitrogendioxide_tropospheric_column_mask75_4326.tif
7
COGT/OFFL/L3__NO2___/2020/02/S5P_OFFL_L3__NO2____20200227_PRODUCT_daily_nitrogendioxide_tropospheric_column_4326.json
8
COGT/OFFL/L3__NO[...]<jupyter_text>5. Filter the relevant files With Sentinel-5P data, you are interested in a conservative cloud factor of at least 75%. Let us selected only the files with the ending `_mask75_4326.tif`. We can do this by filtering the list `key_list`.<jupyter_code>selected_files = list(filter(lambda x: x.endswith('mask75_4326.tif'), key_list))
selected_files<jupyter_output><empty_output><jupyter_text>6. Download the files of interest with `client.download_file()` Now you are ready to go to download the specific files. The client function `client.download_file` takes the bucket name, key name and output folder to download the data.For the example, we only select one image to download<jupyter_code>selected_files[161]
for i in selected_files[161:162]:
    print(i)
    tmp = i.split('/')
    print(tmp[5])
    client.download_file(s5p_bucket, i, './'+tmp[5])<jupyter_output>COGT/OFFL/L3__NO2___/2020/08/S5P_OFFL_L3__NO2____20200806_PRODUCT_daily_nitrogendioxide_tropospheric_column_mask75_4326.tif
S5P_OFFL_L3__NO2____20200806_PRODUCT_daily_nitrogendioxide_tropospheric_column_mask75_4326.tif
{'fraction_non_alphanumeric': 0.08256748994830557, 'fraction_numerical': 0.07668006892590465, 'mean_word_length': 6.941847206385405, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 35, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '417009', 'license': 'CC-BY-4.0', 'n_tokens_mistral': 2889, 'n_tokens_neox': 2563, 'n_words': 542, 'path': '04_aws_open_data_registry.ipynb', 'repo_name': 'vtquang194/Big-Earth-data', 'size': '6964'}