<jupyter_start><jupyter_text>PREREQUISITES In the examples shown in this document, the air pollution datasets of EEA will be used. For this reason, the [EEA-datasets-handler](https://github.com/EnricoPittini/EEA-datasets-handler) library will be used.In particular,the PM10 mean concentrations in Italy are considered, with respect to 2020.<jupyter_code>import EEA_datasets_handler as eea
# Download the datasets 
# IT'S NECESSARY ONLY IF THEY HAVEN'T BEEN DOWNLOADED YET
dest_path = "C:\\Datasets"
countries_cities_dict = {"IT": "all"} 
pollutants = ["PM10"] 
years = [2020]
eea.download_datasets(dest_path, countries_cities_dict, pollutants, years)
# Load the datasets
source_path = "C:\\Datasets\\EEA"
countries_cities_dict = {"IT":"all"}
pollutants = ["PM10"]
years = [2020]
df = eea.load_datasets(source_path,countries_cities_dict,pollutants,years)
df.info()
df
# Process the datasets
df_mean, _, _ = eea.preprocessing(df, fill=True, fill_n_days=10 ,fill_aggr="mean")
df_mean<jupyter_output>C:\Users\Enrico\anaconda3\lib\site-packages\EEA_datasets_handler.py:806: UserWarning: Missing days: ['2020-01-31', '2020-02-01', '2020-02-02', '2020-02-03', '2020-02-04', '2020-02-05', '2020-02-06', '2020-02-07', '2020-02-08', '2020-02-10', '2020-02-11']
  warnings.warn("Missing days: "+str(list(missing_days.strftime('%Y-%m-%d'))))<jupyter_text>In addition, also a metereological dataset will be used (i.e. `meteorological_data_2020`). It will be loaded later on. These data have been obtained using the [ILMETEO](https://www.ilmeteo.it/) Website.  FUNCTIONS TO MANIPULATE DATES Auxiliary functions which work with dates.<jupyter_code>import pandas as pd
days = pd.date_range("2020-01-01","2020-01-04").append(pd.date_range("2020-01-11","2020-03-14"))
print(days)
tsp.find_missing_days(days)
print(tsp.group_days_by(days,criterion="year"))
print()
print(tsp.group_days_by(days,criterion="month"))
print()
print(tsp.group_days_by(days,criterion="season"))
day = pd.Timestamp("2020-04-15")

print(tsp.find_same_month_days(day)) 
print()
print(tsp.find_same_season_days(day))
print(tsp.find_k_years_ago_days(day,k=4,n_days=11))
print()

print(tsp.find_k_years_ago_days(day,k=4,n_days="month"))
print()

print(tsp.find_k_years_ago_days(day,k=4,n_days="season"))
print()
print(tsp.find_current_year_days(day, n_days=11))
print()

print(tsp.find_current_year_days(day, n_days=11, current_day=True))
print()

print(tsp.find_current_year_days(day, n_days="month", current_day=True))
print()

print(tsp.find_current_year_days(day, n_days="season", current_day=False))
print()<jupyter_output>DatetimeIndex(['2020-04-04', '2020-04-05', '2020-04-06', '2020-04-07',
               '2020-04-08', '2020-04-09', '2020-04-10', '2020-04-11',
               '2020-04-12', '2020-04-13', '2020-04-14'],
              dtype='datetime64[ns]', freq='D')

DatetimeIndex(['2020-04-05', '2020-04-06', '2020-04-07', '2020-04-08',
               '2020-04-09', '2020-04-10', '2020-04-11', '2020-04-12',
               '2020-04-13', '2020-04-14', '2020-04-15'],
              dtype='datetime64[ns]', freq='D')

DatetimeIndex(['2020-04-01', '2020-04-02', '2020-04-03', '2020-04-04',
               '2020-04-05', '2020-04-06', '2020-04-07', '2020-04-08',
               '2020-04-09', '2020-04-10', '2020-04-11', '2020-04-12',
               '2020-04-13', '2020-04-14', '2020-04-15'],
              dtype='datetime64[ns]', freq='D')

DatetimeIndex(['2020-03-01', '2020-03-02', '2020-03-03', '2020-03-04',
               '2020-03-05', '2020-03-06', '2020-03-07', '2020-03-08',
               '2020-03-09', '2020-03-10[...]<jupyter_text>FUNCTION TO PLOT A TIME SERIES<jupyter_code>tsp.plot_timeSeries(df_mean, col_name="mean", figsize=(6,6))
tsp.plot_timeSeries(df_mean, col_name="mean", divide="month", figsize=(6,6))
tsp.plot_timeSeries(df_mean, col_name="mean", divide="season", line=False, figsize=(6,6))<jupyter_output><empty_output><jupyter_text>add_timeSeries_dataframe Load `meteorological_data_2020.csv`, which contains meteorological data about Italy in 2020. (It has been obtained using [ILMETEO](https://www.ilmeteo.it/)).<jupyter_code>import pandas as pd
df_meteo = pd.read_csv("meteorological_data_2020.csv",index_col=0)
df_meteo = df_meteo.set_index(pd.DatetimeIndex(df_meteo.index))
df_meteo.info()
df_meteo
df_meteo = df_meteo[["TMEDIA °C","TMIN °C"]] # Focus on two features
df_mean_met, X, y = tsp.add_timeSeries_dataframe(df=df_mean, df_other=df_meteo, y_col="mean")
df_mean_met<jupyter_output><empty_output><jupyter_text>The `add_timeSeries_dataframe` function, like all the other following processing functions, also returns the `X` and `y` numpy arrays.<jupyter_code>X # Contains the explanatory features
y # Contains the response feature. It has been scaled<jupyter_output><empty_output><jupyter_text>*With the input parameter `y_scale`, the user can decide whether to scale or not the response feature.*  add_k_previous_days<jupyter_code>df_mean_temp, X, y = tsp.add_k_previous_days(df=df_mean, col_name="mean", k=5, y_col="mean") 
df_mean_temp.info()
df_mean_temp<jupyter_output><empty_output><jupyter_text>*The names of the added columns have the structure "ColumnName_i", with i from 1 to k*  add_k_years_ago_statistics Get the EEA datasets about the PM10 mean concentrations  in Italy, with respect to 2019 (i.e. one year before 2020).<jupyter_code># Download the datasets 
# IT'S NECESSARY ONLY IF THEY HAVEN'T BEEN DOWNLOADED YET
dest_path = "C:\\Datasets"
countries_cities_dict = {"IT": "all"} 
pollutants = ["PM10"] 
years = [2019]
eea.download_datasets(dest_path, countries_cities_dict, pollutants, years)
# Load the datasets
source_path = "C:\\Datasets\\EEA"
countries_cities_dict = {"IT": "all"}
pollutants = ["PM10"]
years = [2019] 
df_prev_year = eea.load_datasets(source_path, countries_cities_dict, pollutants, years)

df_prev_year.info()
# Process the datasets
df_prev_year_mean, df_prev_year_min, df_prev_year_max = eea.preprocessing(df_prev_year)<jupyter_output><empty_output><jupyter_text>First example: **days_to_select is an odd integer**<jupyter_code>df_mean_temp, X, y = tsp.add_k_years_ago_statistics(df=df_mean, df_k_years_ago=df_prev_year_mean, k=1, days_to_select=11, 
                                                    y_col="mean")<jupyter_output>C:\Users\Enrico\anaconda3\lib\site-packages\timeSeries_processing.py:657: UserWarning: For the day 2020-01-01 only these 1 years ago days have been found: ['2019-01-01', '2019-01-02', '2019-01-03', '2019-01-04', '2019-01-05', '2019-01-06']
  warnings.warn(("For the day "+day.strftime('%Y-%m-%d')+
C:\Users\Enrico\anaconda3\lib\site-packages\timeSeries_processing.py:657: UserWarning: For the day 2020-01-02 only these 1 years ago days have been found: ['2019-01-01', '2019-01-02', '2019-01-03', '2019-01-04', '2019-01-05', '2019-01-06', '2019-01-07']
  warnings.warn(("For the day "+day.strftime('%Y-%m-%d')+
C:\Users\Enrico\anaconda3\lib\site-packages\timeSeries_processing.py:657: UserWarning: For the day 2020-01-03 only these 1 years ago days have been found: ['2019-01-01', '2019-01-02', '2019-01-03', '2019-01-04', '2019-01-05', '2019-01-06', '2019-01-07', '2019-01-08']
  warnings.warn(("For the day "+day.strftime('%Y-%m-%d')+
C:\Users\Enrico\anaconda3\lib\site-packages\timeSeries_processin[...]<jupyter_text>*A warning is given:* - *for each day for which less days than the ones expected are selected;* - *for each day for which no day has been selected.*<jupyter_code>df_mean_temp<jupyter_output><empty_output><jupyter_text>*The names of the added columns have the structure "k_years_ago_ColumnName"* In order to compute the value for a 2020 day, the 11 days of 2019 that are centered on that day are selected.<jupyter_code># 2020-04-15
import pandas as pd
print(df_mean_temp["1_years_ago_mean"].loc[pd.Timestamp("2020-04-15")])
df_prev_year_mean["mean"].loc[pd.date_range('2019-04-10','2019-04-20')].mean()
# 2020-10-22
print(df_mean_temp["1_years_ago_mean"].loc[pd.Timestamp("2020-10-22")])
df_prev_year_mean["mean"].loc[pd.date_range('2019-10-17','2019-10-27')].mean()
# 2020-01-01: example with less than 11 days
print(df_mean_temp["1_years_ago_mean"].loc[pd.Timestamp("2020-01-01")])
df_prev_year_mean["mean"].loc[pd.date_range('2019-01-01','2019-01-06')].mean()
# 2020-12-27: example with less than 11 days
print(df_mean_temp["1_years_ago_mean"].loc[pd.Timestamp("2020-12-27")])
df_prev_year_mean["mean"].loc[pd.date_range('2019-12-22','2019-12-31')].mean()<jupyter_output>24.0776657907545<jupyter_text>*With the input parameter `scale`, the user can decide which is the statistical aggregation to be used.*  **days_to_select is "month" (/"season")**<jupyter_code>df_mean_temp, _, _ = tsp.add_k_years_ago_statistics(df = df_mean, df_k_years_ago=df_prev_year_mean, k=1, 
                                                    days_to_select="month", y_col="mean")
df_mean_temp<jupyter_output><empty_output><jupyter_text>In order to compute the value for a 2020 day, the days of the same month in 2019 are selected.<jupyter_code># January
import pandas as pd
df_prev_year_mean["mean"].loc[pd.date_range('2019-01-01','2019-01-31')].mean()<jupyter_output><empty_output><jupyter_text>**More columns** The same logic is applied individually on each column of `df_prev_year_mean`<jupyter_code>df_prev_year_mean["COLUMN"] = df_prev_year_mean["mean"]*2 # Put a second column
df_mean_temp, _ ,_ = tsp.add_k_years_ago_statistics(df_mean, df_k_years_ago=df_prev_year_mean, k=1, days_to_select=11, 
                                                    y_col="mean")
df_mean_temp<jupyter_output><empty_output><jupyter_text>With the input parameter `columns_to_select`, the user can specify which are the columns to be taken into account.<jupyter_code>df_mean_temp,_,_ = tsp.add_k_years_ago_statistics(df_mean, df_k_years_ago=df_prev_year_mean, k=1, days_to_select=11, 
                                                  columns_to_select = ["mean"], y_col="mean", )
df_mean_temp<jupyter_output><empty_output><jupyter_text>**days_to_select is a function** It's a predicate that decides which days have to be selected<jupyter_code># Predicate of selection
f = lambda day, df, prev_day, prev_df : abs(df["mean"][day]-prev_df["mean"][prev_day])<3

df_mean_temp, _, _ = tsp.add_k_years_ago_statistics(df=df_mean, df_k_years_ago=df_prev_year_mean, k=1, 
                                                    days_to_select=f, y_col="mean")
df_mean_temp<jupyter_output><empty_output><jupyter_text>For each 2020 day, the 2019 days with similar PM10 concentration are selected (i.e. the 2019 days whose PM10 concentration differ from the PM10 concentration of the 2020 day less than 3). **replace_miss** By default, the parameter `replace_miss` is True. I.e. the missing days are filled.<jupyter_code># replace_miss True
f = lambda day,df,prev_day,prev_df : abs(df["mean"][day]-prev_df["mean"][prev_day])<3

df_mean_temp, _, _ = tsp.add_k_years_ago_statistics(df=df_mean, df_k_years_ago=df_prev_year_mean, k=1, 
                                                    days_to_select=f, replace_miss=True, y_col="mean")

df_mean_temp<jupyter_output>C:\Users\Enrico\anaconda3\lib\site-packages\timeSeries_processing.py:654: UserWarning: No 1 years ago days have been found for the day 2020-01-01
  warnings.warn("No "+str(k)+" years ago days have been found for the day " + day.strftime('%Y-%m-%d'))
C:\Users\Enrico\anaconda3\lib\site-packages\timeSeries_processing.py:654: UserWarning: No 1 years ago days have been found for the day 2020-01-16
  warnings.warn("No "+str(k)+" years ago days have been found for the day " + day.strftime('%Y-%m-%d'))<jupyter_text>*The 2020-01-01 is a missing day, and it has been filled.*<jupyter_code># replace_miss False
f = lambda day,df,prev_day,prev_df : abs(df["mean"][day]-prev_df["mean"][prev_day])<3

df_mean_temp, _, _ = tsp.add_k_years_ago_statistics(df=df_mean, df_k_years_ago=df_prev_year_mean, k=1, 
                                                    days_to_select=f, replace_miss=False, y_col="mean")

df_mean_temp<jupyter_output>C:\Users\Enrico\anaconda3\lib\site-packages\timeSeries_processing.py:654: UserWarning: No 1 years ago days have been found for the day 2020-01-01
  warnings.warn("No "+str(k)+" years ago days have been found for the day " + day.strftime('%Y-%m-%d'))
C:\Users\Enrico\anaconda3\lib\site-packages\timeSeries_processing.py:654: UserWarning: No 1 years ago days have been found for the day 2020-01-16
  warnings.warn("No "+str(k)+" years ago days have been found for the day " + day.strftime('%Y-%m-%d'))<jupyter_text>add_current_year_statistics First example: **days_to_select is an integer.**<jupyter_code>df_mean_temp, _, _ = tsp.add_current_year_statistics(df_mean, df_current_year=df_mean, days_to_select=11, y_col="mean")<jupyter_output>C:\Users\Enrico\anaconda3\lib\site-packages\timeSeries_processing.py:871: UserWarning: No current year days have been found for the day 2020-01-01
  warnings.warn("No current year days have been found for the day " + day.strftime('%Y-%m-%d'))
C:\Users\Enrico\anaconda3\lib\site-packages\timeSeries_processing.py:874: UserWarning: For the day 2020-01-02 only these current year days have been found: ['2020-01-01']
  warnings.warn(("For the day "+day.strftime('%Y-%m-%d')+
C:\Users\Enrico\anaconda3\lib\site-packages\timeSeries_processing.py:874: UserWarning: For the day 2020-01-03 only these current year days have been found: ['2020-01-01', '2020-01-02']
  warnings.warn(("For the day "+day.strftime('%Y-%m-%d')+
C:\Users\Enrico\anaconda3\lib\site-packages\timeSeries_processing.py:874: UserWarning: For the day 2020-01-04 only these current year days have been found: ['2020-01-01', '2020-01-02', '2020-01-03']
  warnings.warn(("For the day "+day.strftime('%Y-%m-%d')+
C:\Users\Enrico\anaconda3\li[...]<jupyter_text>*The same warnings of the previous function are given.*<jupyter_code>df_mean_temp<jupyter_output><empty_output><jupyter_text>*The names of the added columns have the structure "current_year_ColumnName"*<jupyter_code>df_mean<jupyter_output><empty_output><jupyter_text>In order to compute the value for a day, the 11 days preceding that day are selected.<jupyter_code># 2020-04-15
import pandas as pd
print(df_mean_temp["current_year_mean"].loc[pd.Timestamp("2020-04-15")])
df_mean["mean"].loc[pd.date_range('2020-04-04','2020-04-14')].mean()
# 2020-10-22
print(df_mean_temp["current_year_mean"].loc[pd.Timestamp("2020-10-22")])
df_mean["mean"].loc[pd.date_range('2020-10-11','2020-10-21')].mean()
# 2020-01-03: example with less than 11 days
print(df_mean_temp["current_year_mean"].loc[pd.Timestamp("2020-01-03")])
df_mean["mean"].loc[pd.date_range('2020-01-01','2020-01-02')].mean()
# 2020-01-05: example with less than 11 days
print(df_mean_temp["current_year_mean"].loc[pd.Timestamp("2020-01-05")])
df_mean["mean"].loc[pd.date_range('2020-01-01','2020-01-04')].mean()<jupyter_output>60.938575393321<jupyter_text>*With the input parameter `scale`, the user can decide which is the statistical aggregation to be used.*  **replace_miss** By default, the parameter `replace_miss` is True. I.e. the missing days are filled.The days for which no preceding day is found are deleted from the dataset.<jupyter_code>df_mean_temp, _, _ = tsp.add_current_year_statistics(df_mean, df_current_year=df_mean, days_to_select=11, y_col="mean")
df_mean_temp<jupyter_output><empty_output><jupyter_text>*No preceding day is present for the day 2020-01-01: this day has been removed.* If `replace_miss` is False, the missing values are kept as Nan.<jupyter_code>df_mean_temp, _, _ = tsp.add_current_year_statistics(df_mean, df_current_year=df_mean, days_to_select=11, replace_miss=False,
                                                     y_col="mean")
df_mean_temp<jupyter_output><empty_output><jupyter_text>**current_day** By default the parameter `current_day` is False. I.e. each day is not itself selected, but only the preceding days are selected.Otherwise, if it's True, each day is itself selected.<jupyter_code>df_mean_temp,_,_ = tsp.add_current_year_statistics(df_mean, df_current_year=df_mean, days_to_select=11, current_day=True, 
                                                   y_col="mean")
df_mean_temp<jupyter_output><empty_output><jupyter_text>In order to compute the value for a day, the day itself and the 10 days preceding that day are selected.<jupyter_code># 2020-04-15
print(df_mean_temp["current_year_mean"].loc[pd.Timestamp("2020-04-15")])
df_mean["mean"].loc[pd.date_range('2020-04-05','2020-04-15')].mean()
# 2020-10-22
print(df_mean_temp["current_year_mean"].loc[pd.Timestamp("2020-10-22")])
df_mean["mean"].loc[pd.date_range('2020-10-12','2020-10-22')].mean()
# 2020-01-03: example with less than 11 days
print(df_mean_temp["current_year_mean"].loc[pd.Timestamp("2020-01-03")])
df_mean["mean"].loc[pd.date_range('2020-01-01','2020-01-03')].mean()
# 2020-01-05: example with less than 11 days
print(df_mean_temp["current_year_mean"].loc[pd.Timestamp("2020-01-05")])
df_mean["mean"].loc[pd.date_range('2020-01-01','2020-01-05')].mean()<jupyter_output>54.38927215262471<jupyter_text>It's important to notice that, despite the fact that `replcae_miss` is True, the day 2020-01-01 hasn't been removed: this is because it isn't a missing day anymore (it has been selected at least one day, which is the day itself). **days_to_select is "month" (/"season")**<jupyter_code>df_mean_temp, _, _ = tsp.add_current_year_statistics(df_mean, df_current_year=df_mean, days_to_select="month", y_col="mean")
df_mean_temp<jupyter_output><empty_output><jupyter_text>In order to compute the value for a day, the preceding days that are in the same month are selected.<jupyter_code># 2020-04-15
print(df_mean_temp["current_year_mean"].loc[pd.Timestamp("2020-04-15")])
df_mean["mean"].loc[pd.date_range('2020-04-01','2020-04-14')].mean()
# 2020-10-22
print(df_mean_temp["current_year_mean"].loc[pd.Timestamp("2020-10-22")])
df_mean["mean"].loc[pd.date_range('2020-10-01','2020-10-21')].mean()<jupyter_output>16.255132480170204<jupyter_text>*If `curret_day` is True, the day itself is also taken* **days_to_select is a function** It's a predicate that decides which preceding days have to be selected. (*If `curret_day` is True, the day itself can also be taken*).<jupyter_code># Predicate of selection
f = lambda day, df, current_day ,current_df: abs(df["mean"].loc[day]-current_df["mean"].loc[current_day])<3

df_mean_temp,_,_ = tsp.add_current_year_statistics(df=df_mean, df_current_year=df_mean, days_to_select=f, y_col="mean")
df_mean_temp<jupyter_output><empty_output><jupyter_text>For each day, the preceding days with similar PM10 concentration are selected (i.e. the preceding days whose PM10 concentration differ from the PM10 concentration of the current day less than 3).<jupyter_code>df_mean[:10]
# 2020-01-04: 2020-01-03 and 2020-01-02 are selected
print(df_mean_temp["current_year_mean"].loc[pd.Timestamp("2020-01-04")])
(56.675791+55.216906)/2
# 2020-01-09: 2020-01-08 and 2020-01-07 are selected
print(df_mean_temp["current_year_mean"].loc[pd.Timestamp("2020-01-09")])
(40.759729+38.646087)/2
# 2020-01-05: is a missing day. The miss value is filled taking into account all the previous days.
print(df_mean_temp["current_year_mean"].loc[pd.Timestamp("2020-01-05")])
(76.974569+56.675791+55.216906+54.887035)/4<jupyter_output>60.938575393321<jupyter_text>**More columns** The same logic is applied individually for each column of `df_current_year`. With the parameter `columns_to_select` the user can specify which are the columns that have to be used.This is shown in the examples for the previous function : `add_k_years_ago_statistics` **EXAMPLE WITH METEOROLOGICAL  DATA** Load `meteorological_data_2020.csv`, which contains meteorological data about Italy in 2020. (It has been obtained using [ILMETEO](https://www.ilmeteo.it/)).<jupyter_code>import pandas as pd
df_meteo = pd.read_csv("meteorological_data_2020.csv",index_col=0)
df_meteo = df_meteo.set_index(pd.DatetimeIndex(df_meteo.index))
df_meteo.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
DatetimeIndex: 366 entries, 2020-01-01 to 2020-12-31
Data columns (total 12 columns):
 #   Column             Non-Null Count  Dtype  
---  ------             --------------  -----  
 0   TMEDIA °C          366 non-null    float64
 1   TMIN °C            366 non-null    float64
 2   TMAX °C            366 non-null    float64
 3   PUNTORUGIADA °C    366 non-null    float64
 4   UMIDITA %          366 non-null    float64
 5   VISIBILITA km      366 non-null    float64
 6   VENTOMEDIA km/h    366 non-null    float64
 7   VENTOMAX km/h      366 non-null    float64
 8   RAFFICA km/h       366 non-null    float64
 9   PRESSIONESLM mb    366 non-null    float64
 10  PRESSIONEMEDIA mb  366 non-null    float64
 11  PIOGGIA mm         366 non-null    float64
dtypes: float64(12)
memory usage: 37.2 KB<jupyter_text>Dataset with both the PM10 mean concentrations and the meteorological features<jupyter_code>df_mean_met, _, _ = tsp.add_timeSeries_dataframe(df_mean, df_meteo, y_col="mean")
df_mean_met
f = lambda day,df,d,df_current_day: abs(df_current_day["TMEDIA °C"].loc[day]-df_current_day["TMEDIA °C"].loc[d])<0.5 

df_mean_temp,_,_ = tsp.add_current_year_statistics(df=df_mean, df_current_year=df_mean_met, 
                                                   days_to_select=f, replace_miss=False, # Miss values are not replaced
                                                   columns_to_select=["mean"], # Only the column "mean" is taken into account
                                                   y_col="mean")<jupyter_output>C:\Users\Enrico\anaconda3\lib\site-packages\timeSeries_processing.py:871: UserWarning: No current year days have been found for the day 2020-01-01
  warnings.warn("No current year days have been found for the day " + day.strftime('%Y-%m-%d'))
C:\Users\Enrico\anaconda3\lib\site-packages\timeSeries_processing.py:871: UserWarning: No current year days have been found for the day 2020-01-04
  warnings.warn("No current year days have been found for the day " + day.strftime('%Y-%m-%d'))
C:\Users\Enrico\anaconda3\lib\site-packages\timeSeries_processing.py:871: UserWarning: No current year days have been found for the day 2020-01-07
  warnings.warn("No current year days have been found for the day " + day.strftime('%Y-%m-%d'))
C:\Users\Enrico\anaconda3\lib\site-packages\timeSeries_processing.py:871: UserWarning: No current year days have been found for the day 2020-01-08
  warnings.warn("No current year days have been found for the day " + day.strftime('%Y-%m-%d'))
C:\Users\Enrico\anaconda3\li[...]<jupyter_text>For each day, the preceding days which have similar mean temperatures are selected (i.e. the difference between the temperatures is less than 0.5).<jupyter_code>df_mean_temp
df_mean_met[["TMEDIA °C"]][:10]
df_mean_temp[:10]
# 2020-01-03: 2020-01-02 and 2020-01-01 are selected
(56.675791+76.974569)/2
# 2020-01-05: only 2020-01-04 is selected
54.887035
# 2020-01-09: only 2020-01-06 is selected
26.099399<jupyter_output><empty_output><jupyter_text>add_upTo_k_years_ago_statistics Get the EEA datasets about the PM10 mean concentrations in Italy, with respect to 2018-2019-2020 (i.e. up to two years before 2020).<jupyter_code># Download the datasets 
# IT'S NECESSARY ONLY IF THEY HAVEN'T BEEN DOWNLOADED YET
dest_path = "C:\\Datasets"
countries_cities_dict = {"IT": "all"} 
pollutants = ["PM10"] 
years = [2018, 2019, 2020]
eea.download_datasets(dest_path, countries_cities_dict, pollutants, years)
# Load the datasets
source_path = "C:\\Datasets\\EEA"
countries_cities_dict = {"IT":"all"}
pollutants = ["PM10"]
years = [2018, 2019, 2020] 
df_full = eea.load_datasets(source_path,countries_cities_dict,pollutants,years)
df_full.info()
df_full_mean,df_full_min,df_full_max = eea.preprocessing(df_full)
df_mean_temp,_,_ = tsp.add_upTo_k_years_ago_statistics(df_mean, df_upTo_k_years_ago=df_full_mean, k=2, days_to_select=11, 
                                                       y_col="mean")
df_mean_temp<jupyter_output><empty_output><jupyter_text>*The names of the added columns have the structure "upTo_k_years_ago_ColumnName".* Basically, the function `add_k_years_ago_statistics` is applied for each of the specified previous years (i.e. for 2019 and for 2018) and then the mean of the computed values is calculated: in this way, for each day of 2020 an unique value is obtained, which sums up the two previous years.  The semantics of the parameters `days_to_select`, `stat`, `columns_to_select`, `replace_miss` are the same seen for the function  `add_k_years_ago_statistics`. **current_year** By default,  the parameter `current_year` is False. I.e. the current year (2020 in our case) is not considered. Otherwise, if `current_year` is True, the current year is considered: that means that not only the function `add_k_years_ago_statistics` is applied two times, but also the function `add_current_year_statistics` is applied one time. The final values in the DataFrame are computed calculating the mean of the values produced by these three functions.<jupyter_code>df_mean_temp,_,_ = tsp.add_upTo_k_years_ago_statistics(df_mean, df_upTo_k_years_ago=df_full_mean, k=2, current_year=True, 
                                                       days_to_select=11, current_day=False, y_col="mean")
df_mean_temp<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13528112924848198, 'fraction_numerical': 0.08220960492074758, 'mean_word_length': 4.973386716910033, 'pattern_counts': {'":': 6, '<': 130, '<?xml version=': 0, '>': 125, 'https://': 4, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 34, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '34', 'dir': 'jupyter-structured-clean-dedup', 'id': '103411', 'license': 'MIT', 'n_tokens_mistral': 10460, 'n_tokens_neox': 8956, 'n_words': 1991, 'path': 'notebooks/timeSeries-processing ~ Examples.ipynb', 'repo_name': 'EnricoPittini/timeSeries-processing', 'size': '25362'}