<jupyter_start><jupyter_text>Time Series Analytics for AERONET sitesThe goal of this notebook is to provide an example of how AERONET sites (https://aeronet.gsfc.nasa.gov/) can be combined with the existing Time Series Service (https://proba-v-mep.esa.int/proba-v-mep-toolset/time-series-viewer).Within this example we will use the location of the AERONET sites for calculating time series. To start, the location of all AERONET sites will be retrieved and displayed on the map. In the next step, the user has the possibility to select a AERONET site and calculate the time series for this specific location based on one of the available datasets and compare it with the data of the site.**This notebook requires the use of Python 3.6**  Script imports<jupyter_code>#Imports of general requests library
import requests

#Imports for leaflet
from ipyleaflet import Map, Marker, MarkerCluster, LayerGroup, GeoJSON

#Import of widgets used in the notebook
import ipywidgets as widgets<jupyter_output><empty_output><jupyter_text>Collecting AERONET sitesAERONET exposes the location of the stations through an online TXT file (https://aeronet.gsfc.nasa.gov/aeronet_locations.txt). By parsing this file we can extract the geographical information for each station. The resulting list of AERONET sites will be displayed on a map for further discovery.<jupyter_code>def fetchAERONETSites():
    sites = list()
    page = requests.get('https://aeronet.gsfc.nasa.gov/aeronet_locations.txt', stream=True).text
    pageLines = list(page.splitlines())
    for location in pageLines[2:]:
        (name, lon, lat, elev) = str(location).replace('\'', '').split(',')
        site = { 
            'name': name, 
            'lon': float(lon),
            'lat': float(lat),
            'elevation': float(elev),
        }
        sites.append(site)
    return sites

sites = fetchAERONETSites()<jupyter_output><empty_output><jupyter_text>Overview of AERONET sitesAfter fetching the location of each AERONET site, we show them on the map. For this we are using the *ipyleaflet* library (https://ipyleaflet.readthedocs.io/en/latest/index.html) to create a map and a clustered view of all sites. A tooltip is added to each marker, showing the name of the site.<jupyter_code>markers = list()
for site in sites:
    marker = Marker(location=(site['lat'], site['lon']), title=site['name'], draggable=False)
    markers.append(marker)
marker_cluster = MarkerCluster(markers=markers)

overview = Map(center=(45, 0), zoom=2, scroll_wheel_zoom=True)
overview.add_layer(marker_cluster)

display(overview)<jupyter_output><empty_output><jupyter_text>Collect time series data setsIn this step we will explore the layers that are available within the existing time series service.<jupyter_code>def fetchDatasets():
    data = requests.get('https://proba-v-mep.esa.int/api/timeseries/v1.0/ts').json()
    return [dataset['name'] for dataset in data['layers']]
    
    
datasets = fetchDatasets()
datasets.sort()
print('Available datasets: %d' % len(datasets))
datasets<jupyter_output>Available datasets: 83<jupyter_text>Calculating time seriesIn this step you can select the AERONET site of interest, together with a dataset available in the time series service. It is also possible to define an offset which represents the distance in metres that will be taken as an area around the site. AREONET Time SeriesThe AERONET data is downloaded from the AERONET general website (https://aeronet.gsfc.nasa.gov/). The temporal range of the data is limited, starting from 2017. Dataset Time SeriesThe MEP time series service (https://proba-v-mep.esa.int/proba-v-mep-toolset/time-series-viewer) is used to calculate the time series for an AEROSET site based on a selected dataset.   Helper functionsBelow are the definitions of the helper functions that are used in parsing the form input and calculating the time series. For the time series calculation we use the MEP dataclient library. It is available at https://bitbucket.org/vitotap/mep-dataclient.  TimeSeries functions<jupyter_code>aeronet_data = None
site_data = None

def calculateTimeSeries():
    
    '''
         Form callback - Calculate the timeseries when the form is submitted.
    '''
    global site_data, aeronet_data

    site_selection_value = [site for site in sites if site['name'] == site_selection.value]
    site_data = getSiteSeries(site_selection_value, dataset_selection.value, offset_selection.value)
    
    
    if len(site_data) > 0: 
        date_min = site_data.index.values[0]
        date_max = site_data.index.values[-1]
        aeronet_data = getAERONETSeries(site_selection_value, date_min, date_max)
        showAREONETparams(aeronet_data)
import dataclient

def getSiteSeries(sites, dataset, offset):
    '''
         Retrieval of timeseries based on the selected site, offset and dataset
    '''
    for site in sites:
        try:
            print('Fetching %s site information for %s with an offset of %s m' % (site['name'], dataset, offset))
            data = dataclient.get_timeseries(createGeometry(site, offset), dataset).dropna()
            print('Retrieved information successfully')
            if len(data) > 0:
                data.name=site['name']
                return data
            else: 
                print('There is no data available for %s' % site['name'])
                return []
        except:
            print('')
            return []<jupyter_output><empty_output><jupyter_text>AERONET data functions<jupyter_code>from calendar import monthrange
import zipfile
import os

def downloadAERONETdata(site, year_start, month_start, year_end, month_end):
    '''
         Download the data from the AERONET site (https://aeronet.gsfc.nasa.gov/)
    '''
    
    # Calculate the last day of the ending month
    last = monthrange(year_end, month_end)[1]

    # Create request for calculating data 
    form_data = dict()
    form_data['site'] = site['name']
    form_data['day'] = 1
    form_data['month'] = month_start
    form_data['year'] = year_start - 1900
    form_data['day2'] = last
    form_data['month2'] = month_end
    form_data['year2'] = year_end - 1900
    form_data['AOD15'] = 1
    form_data['AVG'] = 20
    
    r = requests.post('https://aeronet.gsfc.nasa.gov/cgi-bin/print_warning_v3', data=form_data)

    if r.status_code == 200:
        # Create the download URL for the AREONET site based on the specified date range
        date_range = '%02d%02d01_%02d%02d%02d'% (year_start, month_start, year_end, month_end, last)
        aeronet_zip_name = date_range + '_' + site['name'] + '.zip'
        aeronet_file_name = date_range + '_' + site['name'] + '.lev15'
        aeronet_zip_url = 'https://aeronet.gsfc.nasa.gov/zip_files/V3/' + aeronet_zip_name
        print('Downloading %s' % aeronet_zip_url)
        r = requests.get(aeronet_zip_url)
        
        if 'application/zip' in r.headers['Content-Type']:
            with open(aeronet_zip_name, 'wb') as areonet_zip:
                areonet_zip.write(r.content)
                areonet_zip.close()

            with zipfile.ZipFile(aeronet_zip_name, 'r') as zip_ref:
                zip_ref.extractall('.')

            # Cleanup ZIP file
            os.remove(aeronet_zip_name)
            return aeronet_file_name
        else:
            print('No AERONET data available for site %s during period of %d/%d - %d/%d' % (site['name'], year_start, month_start, year_end, month_end))


    else:
        print('Could not retrieve AERONET data')
        return None
import pandas as pd
import os

def getAERONETSeries(sites, date_start, date_end): 
    '''
         Retrieve the AERONET data as timeseries
    '''
    (year_start, month_start, day_start) = str(date_start).split('T')[0].split('-')
    (year_end, month_end, day_end ) = str(date_end).split('T')[0].split('-')
    
    if int(year_end) >= 2017:
        if int(year_start) < 2017:
            year_start = 2017
            month_start = 1            
        for site in sites:
            datafile = downloadAERONETdata(site, int(year_start), int(month_start), int(year_end), int(month_end))
            if datafile:
                print('Reading datafile %s for site %s' % (datafile, site['name']) )
                dateparse = lambda x: pd.datetime.strptime(x, "%d:%m:%Y")
                aeronet = pd.read_csv(datafile, skiprows=6,
                                      na_values=['N/A', -999, '-999.000000'],
                                      parse_dates={'times':[0]},
                                      date_parser=dateparse)
                os.remove(datafile)
                return aeronet   
            else:            
                return None
    else:
        print('AREONET sites are only available after 2017')
        return None

        
def showAREONETparams(data):
    '''
         Crate a widget containing all AERONET data parameters that can be plotted. When a selection is 
         made, update the plot.
    '''
    selection = widgets.interactive(plotParam, parameter=data.columns[3:]);
    selection.children[0].layout = widgets.Layout(flex='2 1', margin='10px 0px 10px 10px')
    selection.children[0].description = '<b>AREONET parameter to plot:</b>'
    selection.children[0].style ={'description_width': 'initial'}
    display(selection)<jupyter_output><empty_output><jupyter_text>Map functions<jupyter_code>from geojson import Feature,Point, Polygon, utils
from pyproj import Proj, transform

def createGeometry(site, offset):    
    '''
         Create a GEOSJON object that represents the location of site based on the chosen offset.
    '''
    if offset > 0:
        (lat,lon) = transform(Proj(init='epsg:4326'), Proj(init='epsg:3857'), site['lon'], site['lat'])
        (minX, maxX, minY, maxY) = (
            lat - offset, 
            lat + offset, 
            lon - offset, 
            lon + offset)   
        geom = Polygon([[(minX, minY),( minX, maxY), (maxX, maxY), (maxX, minY), (minX, minY)]])
        geom = utils.map_tuples(lambda c: transform(Proj(init='epsg:3857'), Proj(init='epsg:4326'), c[0], c[1]), geom)
    else:
        geom = Point((site['lon'],site['lat']))
        
    
    return Feature(geometry=geom);'''
         Download the data from the AERONET site (https://aeronet.gsfc.nasa.gov/)
    '''
def addMarker(site_selection, offset_selection):
    '''
         Form callback - Update the map based on selection within the form
    '''
    site_selection_value = [site for site in sites if site['name'] == site_selection]
    addSiteMarker(site_selection_value)
    addOffset(site_selection_value, offset_selection)
def addOffset(sites, offset):
    '''
         Draw the offset on the map based on the selected site and offset
    '''
    site_map.layers = [layer for layer in site_map.layers if type(layer) is not GeoJSON]
    for site in sites:
        geojson = GeoJSON(data=createGeometry(site, offset))
        site_map.add_layer(geojson)
def addSiteMarker(sites): 
    '''
        Add a marker to the map based on the selected site
    '''
    site_map.layers = [layer for layer in site_map.layers if type(layer) is not Marker]
    for site in sites:
        marker = Marker(location=(site['lat'], site['lon']), title=site['name'], draggable=False)  
        site_map.add_layer(marker)
        site_map.center = (site['lat'], site['lon'])
        site_map.zoom = 11<jupyter_output><empty_output><jupyter_text>Plot functions<jupyter_code>def plotParam(parameter):
    '''
         Update the plot based on the selected AERONET parameter
    '''
    plotData(site_data, dataset_selection.value, aeronet_data, parameter)
from matplotlib import pyplot as plt
def plotData(site_data, dataset, aeronet_data, aeronet_param): 
    '''
         Create a plot based on the timeseries and AERONET site data
    '''
    if len(site_data) > 0:
        plot_axis = site_data.plot(legend=True, label=dataset, figsize=(17,4))  
    if not aeronet_data is None:
        aeronet_data = aeronet_data[['times', aeronet_param]]
        aeronet_data.plot(kind='line',x='times',y=aeronet_param, label=aeronet_param, ax=plot_axis)
    plt.show()<jupyter_output><empty_output><jupyter_text>Site map creationAs a first step we create the map that will show us the exact location of the chosen site.<jupyter_code>site_map = Map(center=(45, 0), zoom=2, scroll_wheel_zoom=True)<jupyter_output><empty_output><jupyter_text>Form creationIn the next step we create the form that enables the user to select a site, an offset and a dataset. If a user changes the value of the site or the offset, the map will be updated automically. When submitting the form, the time series will be calculated based on the user's selection.<jupyter_code>site_names = [site['name'] for site in sites]    
site_names.sort()


style = {'description_width': 'initial'}

site_selection = widgets.Dropdown(
    options=site_names,
    layout = widgets.Layout(flex='3 1', margin='10px 10px 10px 0px'),
    description = '<b>Select AERONET site:</b>',
    style = style
)

dataset_selection = widgets.Dropdown(
    options=datasets,
    layout = widgets.Layout(flex='4 1', margin='10px 10px 10px 10px'),
    description = '<b>Select dataset:</b>',
    style = style
)

offset_selection = widgets.Dropdown(
    options=[0,3,5,10, 100, 200, 500, 1000],
    layout = widgets.Layout(flex='2 1', margin='10px 0px 10px 10px'),
    description = '<b>Select site offset (m):</b>',
    style = style
)

box = widgets.Box([site_selection, offset_selection, dataset_selection], layout=widgets.Layout(display='flex', align_items='stretch', overflow='hidden', margin='20px 0px 0px 0px'))
out = widgets.interactive_output(addMarker, {'site_selection': site_selection, 'offset_selection': offset_selection})

display(box)
calculate_button = widgets.interact_manual(calculateTimeSeries)
calculate_button.widget.children[0].description = 'Calculate'
calculate_button.widget.children[0].layout=widgets.Layout(margin='0px 0px 20px 0px')
display(site_map)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10694157094956125, 'fraction_numerical': 0.013483627024327602, 'mean_word_length': 3.386107634543179, 'pattern_counts': {'":': 0, '<': 49, '<?xml version=': 0, '>': 53, 'https://': 13, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '293974', 'license': 'Apache-2.0', 'n_tokens_mistral': 4467, 'n_tokens_neox': 4238, 'n_words': 1240, 'path': 'NextGEOSS/TimeSeries analytics with AERONET sites.ipynb', 'repo_name': 'VITObelgium/notebook-samples', 'size': '14017'}