<jupyter_start><jupyter_text>12 - AC-SAF GOME-2 AAH >>  1.1 AC SAF - GOME-2 - Level 3 - Absorbing Aerosol Index (AAI) data   Example Siberian fires in July 2020 The data of the GOME-2 `Absorbing Aerosol Index (AAI)` are provided by KNMI in the framework of the [EUMETSAT Satellite Application Facility on Atmospheric Composition Monitoring (AC SAF)](https://acsaf.org/index.html).The Absorbing Aerosol Index (AAI) indicates the presence of elevated absorbing aerosols in the Earth's atmosphere. The aerosol types that are mostly seen in the AAI are desert dust and biomass burning aerosols. The web-based service for atmospheric satellite data products [TEMIS](http://www.temis.nl/airpollution/absaai/), maintained by KNMI, provides daily and monthly aggregated Level 3 (gridded) data products for the three satellites Metop-A, -B, and -C.Find more information on the GOME-2 Level 3 AAI data product processed by KNMI [here](http://www.temis.nl/airpollution/absaai/gome2/GOME2_aai.README).  This module has the following outline:* [1 - Load and browse GOME-2 Level 3 AAI data](load_browse)* [2 - Concatenate the data from the three satellites Metop-A, -B and -C](concat)* [3 - Retrieve  time coordinate information and assign time coordinates for the time dimension](assign_time_coord)* [4 - Combine AAI data from the three satellites Metop-A, -B, and -C onto one single grid](combine)* [5 - Visualize AAI data with data from the three satellites Metop-A, -B, and C combined on one single grid](visualize)* [6 - Generate and visualize a geographical subset](subset_aai)  Load required libraries<jupyter_code>import os
from netCDF4 import Dataset
import xarray as xr
import numpy as np
import pandas as pd
from datetime import datetime

from IPython.display import HTML
# Python libraries for visualisation
from matplotlib import pyplot as plt
from matplotlib import animation


import cartopy.crs as ccrs
from cartopy.mpl.gridliner import LONGITUDE_FORMATTER, LATITUDE_FORMATTER
%run ./functions.ipynb<jupyter_output><empty_output><jupyter_text>Load and browse GOME-2 Level 3 AAI data The GOME-3 Level 3 AAI data files can be downloaded from the TEMIS website in `NetCDF` data format. TEMIS offers the data of all three satellites Metop-A, -B and -C, which, combined, provide daily measurements for the entire globe.The following example uses daily gridded AAI data for the three satellites Metop-A, -B, and -C for 3 consecutive days between `25 July 2020 and 27 July 2020`. The example shows the dispersion of aerosols during the Siberian fires in July 2020.The daily gridded data is available for each satellite. Thus, a first step is to inspect one file to get a better understanding of the general data structure. Followed by loading the data files for the entire time period into one xarray `DataArray` and to repeat this for each of the three satellites Metop-A, -B and -C.  Inspect the structure of one daily gridded AAI data file The data is in the folder `/eodata/gome2/aai/`. Since the data is distributed in the `NetCDF` format, you can use the xarray function `open_dataset()` to load one single file to better understand the data structure.<jupyter_code>file = './eodata/gome2/aai/2020/07/25/ESACCI-AEROSOL-L3-AAI-GOME2A-1D-20200725-fv1.8.nc'
aai_gome2a = xr.open_dataset(file)
aai_gome2a<jupyter_output><empty_output><jupyter_text>The output of the xarray `Dataset` above shows that one file contains the data of three variables: * `absorbing_aerosol_index`,* `number_of_observations`, and* `solar_zenith_angle`. The variable of interest is `absorbing aerosol_index`. By adding the variable of interest into square brackets `[]`, you can select the variable's `DataArray`. You can see that the daily gridded data are on a 1 deg x 1 deg data grid, with 180 latitude values and 360 longitude values.<jupyter_code>aai = aai_gome2a['absorbing_aerosol_index']
aai<jupyter_output><empty_output><jupyter_text>Load a time-series of daily GOME-2 Level 3 AAI data into one xarray `Dataset` The xarray `open_mfdataset()` function allows the openening of multiple files at once. You have to specify the dimension the files shall be concatenated by. It can be an existing dimension within the data file or a new dimension, which is newly specified.Let's open the daily gridded AAI data from Metop-A for the 3 days from 25 July 2020 to 27 July 2020 and we specify `time` as a new dimension the data files shall be concatenated by.After you loaded the multiple files in a `Dataset` with the function `open_mfdataset()`, you have to select again `absorbing_aerosol_index` as the variable of interest.The resulting `DataArray` has three dimensions (`time`, `latitude` and `longitude`).<jupyter_code>ds_a = xr.open_mfdataset('./eodata/gome2/aai/2020/07/ESACCI-AEROSOL-L3-AAI-GOME2A*.nc', concat_dim='time', combine='nested')
aai_a=ds_a['absorbing_aerosol_index']
aai_a<jupyter_output><empty_output><jupyter_text>The same process has to be repeated for the daily gridded AAI data from the satellites Metop-B and Metop-C respectively.Below, we load the data from the Metop-B satellite.<jupyter_code>ds_b = xr.open_mfdataset('./eodata/gome2/aai/2020/07/ESACCI-AEROSOL-L3-AAI-GOME2B*.nc', concat_dim='time', combine='nested')
aai_b =ds_b['absorbing_aerosol_index']
aai_b<jupyter_output><empty_output><jupyter_text>And here, we load the daily gridded AAI data files from the Metop-C satellite.<jupyter_code>ds_c = xr.open_mfdataset('./eodata/gome2/aai/2020/07/ESACCI-AEROSOL-L3-AAI-GOME2C*.nc', concat_dim='time', combine='nested')
aai_c=ds_c['absorbing_aerosol_index']
aai_c<jupyter_output><empty_output><jupyter_text>Concatenate the data from the three satellites Metop-A, -B and -C The overall goal is to bring the AAI data from all three satellites together. Thus, the next step is to concatenate the `DataArrays` from the three satellites Metop-A, -B and -C and we call the dimension `satellite`. You can use the `concat()` function from the xarray library.The result is a four-dimensional `DataArray`, with the dimensions `satellite`, `time`, `latitude` and `longitude`.You can see that the resulting `DataArray` holds coordinate information for the two spatial dimensions `longitude` and `latitude`, but not for `time` and `satellite`.However, especially the coordinates for `time` will be important for plotting, to know for which day the data is valid. Thus, a next step is to assign coordinates to the `time` dimension.<jupyter_code>aai_concat = xr.concat([aai_a,aai_b,aai_c], dim='satellite')
aai_concat<jupyter_output><empty_output><jupyter_text>Retrieve  time coordinate information and assign time coordinates for the time dimension By inspecting the metadata of the single data file `aai_gome2a` we loaded at the beginning, you can see that the only metadata attribute that contains the valid time step is the `description` attribute.The first step is to retrieve the metadata attribute `description` and split the resulting string object at the positions with a space. The day string is the fourth position of the resulting string.The `description` attribute can be accessed directly from the `aai_gome2a` `Dataset` object.<jupyter_code>start_day = aai_gome2a.description.split()[4]
start_day<jupyter_output><empty_output><jupyter_text>With the help of the Python library `pandas`, you can build a `DateTime` time series for 3 consecutive days, starting from the `start_day` variable that was defined above.You can use the `date_range` function from pandas, using the length of the time dimension of the `aai_concat` DataArray and `'d'` (for day) as freqency argument.The result is a time-series with `DataTime` information from 25 July 2020 to 27 July 2020.<jupyter_code>time_coords = pd.date_range(start_day,periods=len(aai_concat.time), freq='d').strftime("%Y-%m-%d").astype('datetime64[ns]')
time_coords<jupyter_output><empty_output><jupyter_text>The final step is to assign the pandas time series object `time_coords` to the `aai_concat` DataArray object. You can use the `assign_coords()` function from xarray.The result is that now, only the `satellite` dimension has not defined any coordinates.<jupyter_code>aai_concat = aai_concat.assign_coords(time=time_coords)
aai_concat<jupyter_output><empty_output><jupyter_text>Combine AAI data from the three satellites Metop-A, -B, and -C onto one single grid Since the final aim is to combine the data from the three satellites Metop-A, -B and -C onto one single grid, the next step is to reduce the `satellite` dimension. You can do this by applying the reduced function `mean` to the `aai_concat` Data Array. The dimension (`dim`) to be reduced shall be the `satellite` dimension.This function builds the average of all data points within a grid cell. The resulting `DataArray` has three dimensions `time`, `latitude` and `longitude`.<jupyter_code>aai_combined = aai_concat.mean(dim='satellite')
aai_combined<jupyter_output><empty_output><jupyter_text>Visualize AAI data with data from the three satellites Metop-A, -B, and C combined on one single grid The final step is to visualize the Absorbing Aerosol Index data for one time step. You can use the function `visualize_pcolormesh` for it.You can use `afmhot_r` as color map, `ccrs.PlateCarree()` as projection and by applying `dt.strftime('%Y-%m-%d').data` to the time coordinate variable, you can add the valid time step to the title of the plot.The resulting plot shows elevated AAI levels on the east coast of Australia on 25 July 2020.<jupyter_code>visualize_pcolormesh(data_array = aai_combined[1,:,:],
                     longitude = aai_combined.longitude, 
                     latitude = aai_combined.latitude,
                     projection = ccrs.PlateCarree(), 
                     color_scale = 'afmhot_r', 
                     unit = ' ',
                     long_name = aai_a.long_name + ' ' + str(aai_combined.time[0].dt.strftime('%Y-%m-%d').data), 
                     vmin = 0, 
                     vmax = 10)<jupyter_output><empty_output><jupyter_text>6. Generate a geographical subset and visualize the subset Define the bounding box information for the Artic area in Siberia where the fires occured.<jupyter_code>latmin = 50.
latmax = 90.
lonmin = 120.
lonmax = 180.<jupyter_output><empty_output><jupyter_text>Based on the bounding box information, you can now generate a geographical subset with the help of the function `generate_geographical_subset()`. The number of latitude and longitudde pixels have decreased to 40 latitude and 60 longitude, respectively.<jupyter_code>aai_combined_subset = generate_geographical_subset(aai_combined,
                                           latmin = latmin,
                                           latmax = latmax,
                                           lonmin = lonmin,
                                           lonmax = lonmax)
aai_combined_subset<jupyter_output><empty_output><jupyter_text>In a last step, you can visualize the geographical subset for Siberia with the help of the `visualize_pcolormesh()` function. This time, you have to set the `set_global` kwarg to `False` and define the latitude and longitude minimum and maximum values as geographical bounds.<jupyter_code>visualize_pcolormesh(data_array = aai_combined_subset[1,:,:],
                     longitude = aai_combined_subset.longitude, 
                     latitude = aai_combined_subset.latitude,
                     projection = ccrs.PlateCarree(), 
                     color_scale = 'afmhot_r', 
                     unit = ' ',
                     long_name = aai_a.long_name + ' ' + str(aai_combined_subset.time[1].dt.strftime('%Y-%m-%d').data), 
                     vmin = 0, 
                     vmax = 10,
                     latmin = latmin,
                    latmax = latmax,
                    lonmin = lonmin,
                    lonmax = lonmax,
                    set_global=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0764577381952613, 'fraction_numerical': 0.014535372206351874, 'mean_word_length': 4.14390665514261, 'pattern_counts': {'":': 0, '<': 61, '<?xml version=': 0, '>': 63, 'https://': 1, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '204265', 'license': 'MIT', 'n_tokens_mistral': 3706, 'n_tokens_neox': 3441, 'n_words': 1365, 'path': '90_workshops/202105_fire_workshop/11_AC-SAF_Metop-AB_GOME-2_AAI.ipynb', 'repo_name': 'trivedi-c/atm_Practical3', 'size': '11902'}