<jupyter_start><jupyter_text>211 - AC SAF GOME-2 Level 2 Load and Browse>>  1.1 WEkEO Harmonized Data Access (HDA) API [WEkEO](https://www.wekeo.eu/) is the EU Copernicus DIAS (Data and Information Access Service) reference service for environmental data, virtual processing environments and skilled user support.WEkEO offers access to a variety of data, including different parameters sensored from Sentinel-1, Sentinel-2 and Sentinel-3. It further offers access to climate reanalysis and seasonal forecast data.The [Harmonised Data Access (HDA) API](https://www.wekeo.eu/documentation/using_jupyter_notebooks), a REST interface, allows users to subset and download datasets from WEkEO.This notebook is a step-by-step guide on how to search for and download data from WEkEO using the `HDA API`.Follow the following steps: - [1. Search for datasets on WEkEO](wekeo_search) - [2. Get the dataset's Collection ID](wekeo_collection_id) - [3. Get your WEkEO API key](wekeo_api_key) - [4. Initialise the WEkEO Harmonised Data Access request](wekeo_hda_request) - [5. Load data descriptor file and request data](wekeo_json) - [6. Download requested data](wekeo_download) - [7. Unzip downloaded .zip archives](wekeo_unzip)  Load required libraries<jupyter_code>import os
import sys
import json
import xarray as xr
import numpy as np
from zipfile import ZipFile
import matplotlib.pyplot as plt

import glob
import matplotlib
from matplotlib import gridspec
import cartopy.crs as ccrs
import cartopy.feature as cfeature
import matplotlib.ticker as mticker
from IPython.core.display import display, HTML
import warnings
warnings.filterwarnings('ignore')<jupyter_output><empty_output><jupyter_text>Load helper functions<jupyter_code>from ipynb.fs.full.ltpy_hda_api_functions import init, get_access_token, accept_TandC, launch_query, check_job_status, get_results_list, get_download_links, download_data<jupyter_output><empty_output><jupyter_text>1. Search for datasets on WEkEO Under [WEkEO DATA](https://www.wekeo.eu/dataset-navigator/start), you can search all datasets available on WEkEO. As an exmaple: 29 results are returned as a list or the search term `sentinel 3 level 1`. You can click on the dataset you are interested in and you will be guided to a product description.    WEkEO interface to search for datasets  2. Get the Collection ID  At the end of the data description, you get additional information to the dataset, including the `Collection ID`. You need the `Collection ID` to request data from the Harmonised Data Access API. For `OLCI Level 1B Full Resolution - Sentinel-3` data for example, the collection ID is `EO:EUM:DAT:SENTINEL-3:OL_1_EFR___`.    Additional dataset information on WEkEOLet's store the Collection ID as a variable to be used later.<jupyter_code>collection_id = "EO:EUM:DAT:SENTINEL-3:OL_1_EFR___"<jupyter_output><empty_output><jupyter_text>3. Get the WEkEO API key In order to interact with WEkEO's Harmonised Data Access API, each user gets assigned an `API key` and `API token`. You will need the API key in order to download data in a programmatic way.Under [https://www.wekeo.eu/api-keys](https://www.wekeo.eu/api-keys), you can retrieve your WEkEO user credentials. At the bottom of the interface, you can click on `Show hidden keys` and your `API key` is shown.  NOTE: you have to be logged in with your user account.    WEkEO API key interfaceYou can copy paste the `API key` in the first grey box after `-H "Authorization: Basic ...` and store it as a variable called `api_key`.<jupyter_code>api_key = 'REPLACE_WITH_YOUR_API_KEY'<jupyter_output><empty_output><jupyter_text>In addition, you can also store the path to the folder, where you want to save the files after download.<jupyter_code>download_dir_path = './eodata/sentinel3/olci/'
verbose = False<jupyter_output><empty_output><jupyter_text>4. Initialise the Harmonised Data Access API request In order to initialise an API request, you have to initialise a dictionary that contains information on `dataset_id`, `api_key`, `download_directory_path` and whether you wish the request to be `verbose`.The notebook [ltpy_hda_api_functions.ipynb](./ltpy_hda_api_functions.ipynb) offers a set of helper functions.Once initialised, you can request an access token with the function `get_access_token`. You might need to accept the Terms&Conditions, which you can do with the function `accept_TandC`.<jupyter_code>HAPI_dict = init(collection_id, api_key, download_dir_path, verbose=verbose)

HAPI_dict = get_access_token(HAPI_dict)
HAPI_dict = accept_TandC(HAPI_dict)<jupyter_output><empty_output><jupyter_text>5. Load data descriptor file and request data The Harmonised Data Access API can read your data request from a `JSON` file. In this JSON-based file, you can describe the dataset you are interested in downloading. The file is in principle a dictionary. Following keys can be defined:- `datasetID`: the dataset's collection ID- `stringChoiceValues`: type of dataset, e.g. 'Non Time Critical'- `dataRangeSelectValues`: time period you would like to retrieve data- `boundingBoxValues`: optional to define a subset of a global fieldSee an example of a `data descriptor` file [here](./olci_data_descriptor.json)You can load the `JSON` file with `json.load()`. The function `launch_query` will launch your data request.<jupyter_code># load the query
with open('./olci_data_descriptor.json', 'r') as f:
    query = json.load(f)

# launch job
HAPI_dict = launch_query(HAPI_dict, query)<jupyter_output><empty_output><jupyter_text>6. Download requested data The functions `check_job_status`, `get_results_list` and `get_download_links` help you to monitor the retrieval request.As a final step, you can use the function `download_data` to initialize the data download.<jupyter_code># wait for jobs to complete
check_job_status(HAPI_dict)

# check results
HAPI_dict = get_results_list(HAPI_dict)
HAPI_dict = get_download_links(HAPI_dict)

# download data
HAPI_dict = download_data(HAPI_dict, skip_existing=True)<jupyter_output><empty_output><jupyter_text>7. Unzip the downloaded data archives Sentinel-3 data are disseminated in `.zip archives`. Thus, a final step is to unzip the downloaded .zip archives. The result is a folder that ends with .SEN3 and which contains the Sentinel3 data files in `NetCDF` format.<jupyter_code>def unzip_downloads():
    file_list = glob.glob('*.zip')
    print(file_list)
    for filename in file_list:
        print(filename)
        print('Unzipping file')
        with ZipFile(filename, 'r') as zipObj:
            zipObj.extractall(os.path.dirname(filename))
unzip_downloads()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0753693096171239, 'fraction_numerical': 0.005728067530901417, 'mean_word_length': 5.562809099901088, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 39, 'https://': 5, 'lorem ipsum': 0, 'www.': 5, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 24, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '476595', 'license': 'MIT', 'n_tokens_mistral': 2019, 'n_tokens_neox': 1987, 'n_words': 770, 'path': '12_ltpy_WEkEO_harmonized_data_access_api.ipynb', 'repo_name': 'federicofierli1/LTPy_Notebooks_satellite_copernicus', 'size': '6634'}