<jupyter_start><jupyter_text>Analogous YearsThe Analogous Years application enables users to compare events from a set period of time, to those of the same date range in other years. The application will compute ranks of similarity between the specified period, and the same period from previous or future years. 0. Gro API ClientTo get started with `Analogous Years`, users have to install `Gro API Client`as detailed [here](https://developers.gro-intelligence.com/installation.html). 0.1 Analogous Years' FunctionsFollowing the step mentioned above, users can run the following cell to import some of the necessary libraries from the `analogous_years` package.<jupyter_code>import os
from api.client.samples.analogous_years import run_analogous_years
from api.client.samples.analogous_years.lib import final_ranks_computation, get_transform_data
import pandas as pd
pd.options.mode.chained_assignment = None<jupyter_output><empty_output><jupyter_text>0.2 Access the Gro API access tokenAssuming that users have saved the Gro API access token as an environment variable named `GROAPI_TOKEN` (and have imported `os` in the previous step), users may run the following cell to get the access token for later usage.<jupyter_code>ACCESS_TOKEN = os.environ['GROAPI_TOKEN']<jupyter_output><empty_output><jupyter_text>1. InputMultiple different inputs can be used in determining the ranks (refer to the appendix), but to compute these ranks, a user must provide: 1.1 Gro Data SeriesSingle or multiple Gro Data Series defined by `metric_id`, `item_id`, `source_id`, `frequency_id` for a particular region given by a `region_id`. 1.2 Time PeriodA time period determined by an `initial_date` and a `final_date`. The two dates must be within 1 year of each other and in the `YYYY-MM-DD` format. ExampleAs an example, if the user wants to know which period of time is most similar to the time period between 1st January 2019 and 31st October 2019 in the [US Corn Belt States (region_id=100000100)](https://app.gro-intelligence.com/dictionary/regions/100000100) with respect to thefollowing Gro Data Series -1. Rainfall, TRMM ([metric_id=2100031](https://app.gro-intelligence.com/dictionary/metrics/2100031), [item_id=2039](https://app.gro-intelligence.com/dictionary/items/2039), [source_id=35](https://app.gro-intelligence.com/dictionary/sources/35), frequency_id=1)2. Land Temperature, MODIS ([metric_id=2540047](https://app.gro-intelligence.com/dictionary/metrics/2540047), [item_id=3457](https://app.gro-intelligence.com/dictionary/items/3457), [source_id=26](https://app.gro-intelligence.com/dictionary/sources/26), frequency_id=1)3. Soil moisture, SMOS ([metric_id=15531082](https://app.gro-intelligence.com/dictionary/items/15531082), [item_id=7382](https://app.gro-intelligence.com/dictionary/items/7382), [source_id=43](https://app.gro-intelligence.com/dictionary/sources/43), frequency_id=1)Note: 1. `frequency_id: 1` gives us daily values. In absence of daily values, the application up-samples to daily frequency(ies).2. A convenient way to incorporate these is to copy the dictionaries associated to the data series mentioned above from [Gro webapp display](https://app.gro-intelligence.com/displays/za9MlQYRM) by exporting the api client code snippets of the display. The copied data series will be stored in `data_series_list` as outlined below.<jupyter_code># Rainfall (modeled) - Precipitation Quantity - US Corn Belt States (NASA TRMM 3B42RT)
data_series_1 = {'metric_id': 2100031, 
                 'item_id': 2039, 
                 'region_id': 100000100, 
                 'partner_region_id': 0, 
                 'source_id': 35, 
                 'frequency_id': 1, 
                 'unit_id': 2}
# Land temperature (daytime, modeled) - Temperature - US Corn Belt States (NASA MODIS MOD11 LST)
data_series_2 = {'metric_id': 2540047, 
                 'item_id': 3457, 
                 'region_id': 100000100, 
                 'partner_region_id': 0, 
                 'source_id': 26, 
                 'frequency_id': 1,
                 'unit_id': 36}

# Soil moisture - Availability in soil (volume/volume) - US Corn Belt States (ESA SMOS CLF33D)
data_series_3 = {'metric_id': 15531082, 
            'item_id': 7382, 
            'region_id': 100000100, 
            'partner_region_id': 0, 
            'source_id': 43, 
            'frequency_id': 1}


data_series_list = [data_series_1, data_series_2, data_series_3]


initial_date = '2019-01-01'
final_date = '2019-10-31'<jupyter_output><empty_output><jupyter_text>2. OutputThe output will be a pandas dataframe containg the ranks computed by ensembling several distance calculation method.<jupyter_code>result = final_ranks_computation.analogous_years(
    ACCESS_TOKEN, data_series_list, initial_date, final_date)
result<jupyter_output>Feature Extraction: 100%|██████████| 9/9 [00:02<00:00,  3.36it/s]
Feature Extraction: 100%|██████████| 9/9 [00:02<00:00,  3.85it/s]
Feature Extraction: 100%|██████████| 9/9 [00:02<00:00,  3.82it/s]<jupyter_text>3. Appendix 3.1 Methods Of Rank CalculationThe analogy score between two different time periods can be measured in multiple ways. Here, the program can calculate ranks based on 2 primary approaches - 1. Ranks based on differences between extracted features:     1. Distance between cumulative sums.     2. Distance between more features extracted from time series.         Note: For the purpose of this package we have used `tsfresh` package to     extract data from time series. 2. Point wise differences:     1. Euclidean distance between stacked time periods.     2. Dynamic Time Warping distance between stacked time periods.    Finally, the program returns a composite rank by default based on the default (`cumulative`, `euclidean`, `ts-features`) methods or user specified methods.  3.2 Additional Options1. Weights: Users can input different weights associated with different data-series. By default, all the data series are equally weighted.2. Methods: Users have an option to choose from the following methods for distance computation `cumulative`, `euclidean`, `ts-features`, `dtw`. The default methods for rank generation are `cumulative`, `euclidean`, `ts-features`. `dtw` method is intentionally left out of the default setting as it is computationally expensive to run dynamic time warping algorithm on one `item-metric` tuple and in most situations `dtw` ranks are highly correlated with the `euclidean` ranks.3. All Ranks: Users have an option to output multiple individual ranks or an ensemble rankbased on their methods list. By default only the ensemble rank will be generated.4. Multivariate El Niño Southern Oscillation (ENSO) index can also be included in the rank computation, along with the weight that the user wants to give for the ENSO index. The weight of ENSO index is set to 1 by default (same as other data series).5. Start Date: Users have an option to exclude time periods before a specified date. By default the earliest date from which data is available for all entities will be used to compute ranks.<jupyter_code>additional_options_result = final_ranks_computation.analogous_years(
    ACCESS_TOKEN, data_series_list, initial_date, final_date, 
    methods_list=['cumulative', 'euclidean', 'ts-features', 'dtw'], 
    all_ranks=True, weights=[0.2, 0.3, 0.4], enso=True,
    enso_weight=0.1, provided_start_date='2015-01-01')
additional_options_result<jupyter_output>Feature Extraction: 100%|██████████| 5/5 [00:01<00:00,  3.25it/s]
Feature Extraction: 100%|██████████| 5/5 [00:01<00:00,  3.91it/s]
Feature Extraction: 100%|██████████| 5/5 [00:01<00:00,  3.65it/s]
Feature Extraction: 100%|██████████| 5/5 [00:02<00:00,  2.10it/s]<jupyter_text>4. ReportIn this section we have computed the `spearman's rank correlation` between the ranks generated by different methods. We have also used scatterplots to visualize the correlation between the ranks. 4.1 Correlation MatrixFrom our past observations we have found that `dtw_rank` is highly correlated with `euclidean_rank`. However, `dtw_rank` calculation takes an extremely long time when compared with the other methods. Hence, for computational efficiency we have removed `dtw` from the default `list_of_methods`.<jupyter_code>correlation_matrix = additional_options_result.corr(method='spearman')
correlation_matrix<jupyter_output><empty_output><jupyter_text>4.2 Scatterplots Between RanksWe have generated scatterplots between ranks which may help advanced users to understand if some of the methods are more consistent for their specific data-series.<jupyter_code>%matplotlib inline
import seaborn as sns
import matplotlib.pyplot as plt
sns.set(style="ticks")
sns_plot = sns.pairplot(additional_options_result, diag_kind=None)
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09316628701594533, 'fraction_numerical': 0.04407744874715262, 'mean_word_length': 4.73921568627451, 'pattern_counts': {'":': 0, '<': 34, '<?xml version=': 0, '>': 27, 'https://': 12, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '562485', 'license': 'MIT', 'n_tokens_mistral': 2885, 'n_tokens_neox': 2491, 'n_words': 935, 'path': 'api/client/samples/analogous_years/get_started_with_analogous_years.ipynb', 'repo_name': 'eric-gro/api-client', 'size': '8780'}