<jupyter_start><jupyter_text>Tutorial: Climate data access through CDS API and processing in JupyterIn this tutorial we will access and analyse data from the Climate Data Store (CDS) of the Copernicus Climate Change Service (C3S). We will focus on climate extremes in the area surrounding the city of Lille in Northern France.The tutorial comprises the following steps:0. [Prepare environment](prepare): 1) Register with the CDS, 2) obtain a CDS API key, 3) install the CDS API and 4) import Python libraries.1. [Search and download](search_download) data using the CDS API: We will focus on ERA5 reanalysis data of 2 metre (near-surface) temperature.2. [Read data](read_data): Once downloaded, we will read and understand the data, including its variables and coordinates.3. [View and plot](view_plot) maximum temperatures in September 2020.4. [Calculate averages](calculate_averages) of the maximum daily temperatures in September over the period from 1979 to 2019, and compare these with our findings for 2020.   0. Prepare environmentBefore we begin we will prepare our environment. This includes registering with the CDS and obtaining a unique CDS API key. We will then install the CDS API and import various preinstalled libraries. Register with CDSTo register with the Climate Data Store (CDS) visit https://cds.climate.copernicus.eu/ and select **"Login/Register"**   Obtain CDS API keyTo obtain your unique User ID and CDS API key follow the steps at this link:https://cds.climate.copernicus.eu/api-how-toPlease make a note of these as they will be needed to download data from the CDS using the API.  Install CDS APITo install the CDS API, run the following command. We use an exclamation mark to pass the command to the shell (not to the Python interpreter).<jupyter_code>!pip install cdsapi<jupyter_output>Requirement already satisfied: cdsapi in c:\users\cxcs\anaconda3\lib\site-packages (0.3.1)
Requirement already satisfied: tqdm in c:\users\cxcs\anaconda3\lib\site-packages (from cdsapi) (4.47.0)
Requirement already satisfied: requests>=2.5.0 in c:\users\cxcs\anaconda3\lib\site-packages (from cdsapi) (2.24.0)
Requirement already satisfied: certifi>=2017.4.17 in c:\users\cxcs\anaconda3\lib\site-packages (from requests>=2.5.0->cdsapi) (2020.6.20)
Requirement already satisfied: chardet<4,>=3.0.2 in c:\users\cxcs\anaconda3\lib\site-packages (from requests>=2.5.0->cdsapi) (3.0.4)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in c:\users\cxcs\anaconda3\lib\site-packages (from requests>=2.5.0->cdsapi) (1.25.9)
Requirement already satisfied: idna<3,>=2.5 in c:\users\cxcs\anaconda3\lib\site-packages (from requests>=2.5.0->cdsapi) (2.10)<jupyter_text>Import librariesWe will be working with data in NetCDF format. To best handle this data we need a number of libraries for working with multidimensional arrays, in particular Xarray. We will also need libraries for plotting and viewing data, in particular Matplotlib and Cartopy.<jupyter_code># CDS API
import cdsapi

# Libraries for working with multidimensional arrays
import numpy as np
import xarray as xr

# Libraries for plotting and visualising data
import matplotlib.path as mpath
import matplotlib.pyplot as plt
import cartopy.crs as ccrs
from cartopy.mpl.gridliner import LONGITUDE_FORMATTER, LATITUDE_FORMATTER
import cartopy.feature as cfeature

# to suppress SSL warnings
import urllib3
urllib3.disable_warnings()<jupyter_output><empty_output><jupyter_text>To enable us to download data through the CDS API, we need to specify our User ID and CDS API key. Please enter them in the fields below by replacing "UID" with your User ID, and API_KEY with your API key:<jupyter_code># enter your ADS authentication key
URL = "https://cds.climate.copernicus.eu/api/v2"
KEY = "UID:API_KEY"
c = cdsapi.Client(url=URL, key=KEY)<jupyter_output><empty_output><jupyter_text>1. Search and download dataNow that we have installed the API we can download data from the CDS directly into our environment.  Search for dataTo search for data, visit the CDS website: http://cds.climate.copernicus.eu To facilitate your search you can use keywords, or apply various filters. The data we are going to use in this exercise is the ERA5 hourly data on single levels from 1979 to present.   Having selected the correct dataset, we now need to specify what product type, variables, temporal and geographic coverage we are interested in. These can all be selected in the **"Download data"** tab. In this tab a form appears in which we will select the following parameters to download. We will choose a subset area of 1x1 degrees, corresponding to a region of around 111km North/South and 72km East/West in Belgium and Northern France, around the city of Lille:- Product type: `Reanalysis`- Variable: `2m temperature`- Year: `all`- Month: `September`- Day: `all`- Time: `all`- Geographical area: `North: 51`, `East: 4`, `South: 50`, `West: 3`- Format: `NetCDF`At the end of the download form, select **"Show API request"**. This will reveal a block of code, which you can simply copy and paste into a cell of your Jupyter Notebook (see cell below) ... Download data... having copied the API request into the cell below, running this will retrieve and download the data you requested into your local directory.<jupyter_code>c.retrieve(
    'reanalysis-era5-single-levels',
    {
        'product_type': 'reanalysis',
        'format': 'netcdf',
        'variable': '2m_temperature',
        'year': [
            '1979', '1980', '1981',
            '1982', '1983', '1984',
            '1985', '1986', '1987',
            '1988', '1989', '1990',
            '1991', '1992', '1993',
            '1994', '1995', '1996',
            '1997', '1998', '1999',
            '2000', '2001', '2002',
            '2003', '2004', '2005',
            '2006', '2007', '2008',
            '2009', '2010', '2011',
            '2012', '2013', '2014',
            '2015', '2016', '2017',
            '2018', '2019', '2020',
        ],
        'month': '09',
        'day': [
            '01', '02', '03',
            '04', '05', '06',
            '07', '08', '09',
            '10', '11', '12',
            '13', '14', '15',
            '16', '17', '18',
            '19', '20', '21',
            '22', '23', '24',
            '25', '26', '27',
            '28', '29', '30',
        ],
        'time': [
            '00:00', '01:00', '02:00',
            '03:00', '04:00', '05:00',
            '06:00', '07:00', '08:00',
            '09:00', '10:00', '11:00',
            '12:00', '13:00', '14:00',
            '15:00', '16:00', '17:00',
            '18:00', '19:00', '20:00',
            '21:00', '22:00', '23:00',
        ],
        'area': [
            51, 3, 50,
            4,
        ],
    },
    'NFrance_hourly_Sep.nc')<jupyter_output>2021-02-09 21:02:35,475 INFO Welcome to the CDS
2021-02-09 21:02:35,476 INFO Sending request to https://cds.climate.copernicus.eu/api/v2/resources/reanalysis-era5-single-levels
2021-02-09 21:02:35,659 INFO Request is queued
2021-02-09 21:02:36,755 INFO Request is running
2021-02-09 21:43:03,428 INFO Request is completed
2021-02-09 21:43:03,455 INFO Downloading https://download-0012.copernicus-climate.eu/cache-compute-0012/cache/data4/adaptor.mars.internal-1612904555.1840663-3849-3-fd23b77f-8e8a-49a0-adf3-a29e452afc16.nc to NFrance_hourly_Sep.nc (1.6M)
2021-02-09 21:43:08,689 INFO Download rate 305K/s<jupyter_text>2. Read DataNow that we have downloaded the data, we can process it, plot graphs and derive information...We have requested the data in NetCDF format. This is a commonly used format for array-oriented scientific data. To read and process this data we will make use of the Xarray library. Xarray is an open source project and Python package that makes working with labelled multi-dimensional arrays simple and efficient. We will read the data from our NetCDF file into an Xarray **"dataset"**<jupyter_code>filename = r'./NFrance_hourly_Sep.nc'
# Create Xarray Dataset
ds = xr.open_dataset(filename)<jupyter_output><empty_output><jupyter_text>Now we can query our newly created Xarray dataset ...<jupyter_code>ds<jupyter_output><empty_output><jupyter_text>We see that the dataset has one variable called **"t2m"**, which stands for "2 metre temperature", and three coordinates of **longitude**, **latitude** and **time**.Select the icons to the right of the table above to expand the attributes of the coordinates and data variables. What are the units of the temperature data? While an Xarray **dataset** may contain multiple variables, an Xarray **data array** holds a single variable (which may still be multi-dimensional) and its coordinates. To make the processing of the **t2m** data easier, we convert in into an Xarray data array:<jupyter_code>da = ds['t2m']<jupyter_output><empty_output><jupyter_text>Let's convert the units of the 2m temperature data from Kelvin to degrees Celsius. The formula for this is simple: degrees Celsius = Kelvin - 273.15<jupyter_code>t2m_C = da - 273.15<jupyter_output><empty_output><jupyter_text>3. View daily maximum 2m temperature for September 2020We will plot the maximum values of 2m temperature over the subset area of Northern France. First we average over the subset area:<jupyter_code>Lille_t2m = t2m_C.mean(["longitude", "latitude"])<jupyter_output><empty_output><jupyter_text>Now we select only the data for 2020:<jupyter_code>Lille_2020 = Lille_t2m.sel(time='2020')<jupyter_output><empty_output><jupyter_text>We can now calculate the max daily 2m temperature for each day in September 2020:<jupyter_code>Lille_2020_max = Lille_2020.groupby('time.day').max('time')<jupyter_output><empty_output><jupyter_text>Let's plot the results in a chart:<jupyter_code>x = Lille_2020_max.day
y = (np.around(Lille_2020_max.values, 0)).astype(int)

fig = plt.figure(figsize=(10,5))
ax = plt.subplot()
ax.set_ylabel('t2m (Celsius)')
ax.set_xlabel('day')
ax.plot(x, y)
ax.grid(linestyle='--')
for i,j in zip(x,y):
    ax.annotate(str(j),xy=(i,j))
ax.set_title('Max daily t2m for Sep 2020')
print('The maximum temperature in September 2020 in this area was', 
      np.around(Lille_2020_max.max().values, 1), 'degrees Celsius.')<jupyter_output>The maximum temperature in September 2020 in this area was 32.1 degrees Celsius.<jupyter_text>Which day in September had the highest maximum temperature?Is this typical for Northern France? How does this compare with the long term average? We will seek to answer these questions in the next section.  4. Calculate long term average of 2m temperature for September over Northern FranceWe will now seek to discover just how high the temperature for Lille in mid September 2020 was when compared to typical values exptected in this region at this time of year. To do that we will calculate the mean and standard deviation of maximum daily 2m temperature for each day in September for the period of 1979 to 2019, and compare these with our values for 2020. First we select all data prior to 2020. This data has experiment version 1 (consolidated version of ERA5).<jupyter_code>Lille_past = Lille_t2m.loc['1979':'2019']<jupyter_output><empty_output><jupyter_text>Now we calculate the climatology for this data, i.e. the mean and standard deviation of maximum daily values for each of the days in September for a period of several decades (from 1979 to 2019). To do this, we first have to extract the maximum daily value for each day in the time series:<jupyter_code>Lille_max = Lille_past.resample(time='D').max()<jupyter_output><empty_output><jupyter_text>Then we can calculate the mean and standard deviation of this for the 40 year time series for each day in September:<jupyter_code>Lille_m = Lille_max.groupby('time.day').mean('time')
Lille_sd = Lille_max.groupby('time.day').std('time')<jupyter_output><empty_output><jupyter_text>Let's plot this data. We will plot the mean plus and minus one standard deviation to have an idea of the expected range of maximum daily temperatures in this part of France in September:<jupyter_code>y1 = Lille_m
y2 = Lille_m + Lille_sd
y2 = np.squeeze(y2.values)
y3 = Lille_m - Lille_sd
y3 = np.squeeze(y3.values)

fig = plt.figure(figsize=(10,5))
ax = plt.subplot()
ax.set_ylabel('t2m (Celsius)')
ax.set_xlabel('day')
ax.plot(Lille_m.day, y1, color='green', label='t2m mean, shading: +/- SD')
ax.plot(Lille_m.day, y2, color='white')
ax.plot(Lille_m.day, y3, color='white')
ax.fill_between(Lille_m.day, y2, y3, alpha=0.1)
handles, labels = ax.get_legend_handles_labels()
ax.legend(handles, labels)
ax.set_title('t2m climatology for Sep from 1979 to 2019')<jupyter_output><empty_output><jupyter_text>What is the typical range of maximum 2m temperature values for September 15? We will now look more closely at the probability distribution of maximum temperatures for 15 September in this time period. To do this, we will first select only the max daily temperature for 15 September, for each year in the time series:<jupyter_code>Lille_max = Lille_max.dropna('time', how='all')
Lille_15 = Lille_max[14::30]<jupyter_output><empty_output><jupyter_text>We will then plot the histogram of this:<jupyter_code>Lille_15.plot.hist()<jupyter_output><empty_output><jupyter_text>Look at the range of maximum temperatures for 15 September in the period from 1979 to 2019. Has the temperature in this period ever exceeded that of 15 September 2020?The histogram shows the distribution of maximum temperature of one day in each year of the time series, which corresponds to 41 samples. In order to increase the number of samples, let's plot the histogram of maximum temperatures on 15 September, plus or minus three days. This would increase our number of samples by a factor of seven.To do this, we first need to produce an index that takes 15 Sep, plus or minus three days, from every year in the time series:<jupyter_code>years = np.arange(41)
days_in_sep = np.arange(11,18)
index = np.zeros(287)
for i in years:
    index[i*7:(i*7)+7] = days_in_sep + (i*30)
index = index.astype(int)<jupyter_output><empty_output><jupyter_text>Then we apply this index to filter the array of max daily temperature from 1979 to 2019:<jupyter_code>Lille_7days = Lille_max.values[index]<jupyter_output><empty_output><jupyter_text>Now we can plot the histogram of maximum daily temperatures in days 12-18 September from 1979-2019:<jupyter_code>plt.figure(figsize=(10,5))
plt.hist(Lille_7days, bins = np.arange(10,32,1)) 
plt.title("histogram of max temperature in days 12-18 Sep from 1979-2019")
plt.xticks(np.arange(10,32,1))
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10306234550947542, 'fraction_numerical': 0.06152156001098599, 'mean_word_length': 4.477623166603986, 'pattern_counts': {'":': 0, '<': 85, '<?xml version=': 0, '>': 95, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 16, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '21', 'dir': 'jupyter-structured-clean-dedup', 'id': '201740', 'license': 'Apache-2.0', 'n_tokens_mistral': 5121, 'n_tokens_neox': 4505, 'n_words': 1710, 'path': 'POLITO_training.ipynb', 'repo_name': 'stewartchrisecmwf/POLITO_training', 'size': '14564'}