<jupyter_start><jupyter_text>Multiple models comparison This notebook will run three forecasting algorithms on the same dataset and compare their performances.The algorithms are:  - Prophet  - ETS  - DeepAR+   Setup<jupyter_code>import boto3
from time import sleep
import pandas as pd
import seaborn as sns
import pprint
pp = pprint.PrettyPrinter(indent=2)  # Better display for dictionaries<jupyter_output><empty_output><jupyter_text>The line below will retrieve your shared variables from the first notebook.<jupyter_code>%store -r<jupyter_output><empty_output><jupyter_text>The last part of the setup process is to validate that your account can communicate with Amazon Forecast, the cell below does just that.<jupyter_code>forecast = boto3.client(service_name='forecast')
forecastquery = boto3.client(service_name='forecastquery')<jupyter_output><empty_output><jupyter_text>Create the predictors The first step is to create a dictionary where to store useful information about the algorithms: their name, ARN and eventually their performance metrics.<jupyter_code>algos = ['Prophet', 'ETS', 'Deep_AR_Plus']

predictors = {a:{} for a in algos}

for p in predictors:
    predictors[p]['predictor_name'] = project + '_' + p + '_algo'
    predictors[p]['algorithm_arn'] = 'arn:aws:forecast:::algorithm/' + p

pp.pprint(predictors)<jupyter_output><empty_output><jupyter_text>Here we also define our forecast horizon: the number of time points to be predicted in the future. For weekly data, a value of 12 means 12 weeks. Our example is hourly data, we try forecast the next day, so we can set to 24.<jupyter_code>forecastHorizon = 24<jupyter_output><empty_output><jupyter_text>The following function actually creates the predictor as specified by several parameters. We will call this function once for each of the 3 algorithms.<jupyter_code>def create_predictor_response(pred_name, algo_arn, forecast_horizon):
    response=forecast.create_predictor(PredictorName=pred_name, 
                                       AlgorithmArn=algo_arn,
                                       ForecastHorizon=forecast_horizon,
                                       PerformAutoML= False,
                                       PerformHPO=False,
                                       EvaluationParameters= {"NumberOfBacktestWindows": 1, 
                                                              "BackTestWindowOffset": 24}, 
                                       InputDataConfig= {"DatasetGroupArn": datasetGroupArn},
                                       FeaturizationConfig= {"ForecastFrequency": "H", 
                                                             "Featurizations": 
                                                             [
                                                                 {"AttributeName": "target_value", 
                                                                  "FeaturizationPipeline": 
                                                                  [
                                                                      {"FeaturizationMethodName": "filling", 
                                                                       "FeaturizationMethodParameters": 
                                                                       {"frontfill": "none", 
                                                                        "middlefill": "zero", 
                                                                        "backfill": "zero"}
                                                                      }
                                                                  ]
                                                                 }
                                                             ]
                                                            }
                                      )
    return response<jupyter_output><empty_output><jupyter_text>For all 3 algorithms, we invoke their creation and wait until they are complete. We also store their performance in our dictionary.<jupyter_code>for p in predictors.keys():
    predictor_response = create_predictor_response(predictors[p]['predictor_name'], predictors[p]['algorithm_arn'], forecastHorizon)
    
    predictorArn=predictor_response['PredictorArn']
    
    # wait for the predictor to be actually created
    print('------------------ Creating ' + p)
    while True:
        predictorStatus = forecast.describe_predictor(PredictorArn=predictorArn)['Status']
        print(predictorStatus)
        if predictorStatus != 'ACTIVE' and predictorStatus != 'CREATE_FAILED':
            sleep(30)
        else:
            predictors[p]['predictor_arn'] = predictorArn  # save it, just for reference
            break
            
    # compute and store performance metrics, then proceed with the next algorithm        
    predictors[p]['accuracy'] = forecast.get_accuracy_metrics(PredictorArn=predictorArn)<jupyter_output><empty_output><jupyter_text>**TODO:** (Bar?)plot RMSE, 0.9-, 0.5- and 0.1-quantile LossValues for each algorithm This is what we stored so far for DeepAR+:<jupyter_code>pp.pprint(predictors['Deep_AR_Plus'])<jupyter_output><empty_output><jupyter_text>Visualize results We use `seaborn` as it interacts well with `pandas` DataFrames. Looping over our dictionary, we can retrieve the Root Mean Square Error (RMSE) for each predictor and plot it as a bar plot.<jupyter_code>scores = pd.DataFrame(columns=['predictor', 'RMSE'])
for p in predictors:
    score = predictors[p]['accuracy']['PredictorEvaluationResults'][0]['TestWindows'][0]['Metrics']['RMSE']
    scores = scores.append(pd.DataFrame({'predictor':[p], 'RMSE':[score]}), ignore_index=True)
fig = sns.barplot(data=scores, x='predictor', y='RMSE').set_title('Root Mean Square Error')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08163265306122448, 'fraction_numerical': 0.004630423598010633, 'mean_word_length': 1.8914229053049083, 'pattern_counts': {'":': 12, '<': 37, '<?xml version=': 0, '>': 37, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 45, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '146665', 'license': 'MIT-0', 'n_tokens_mistral': 1405, 'n_tokens_neox': 1327, 'n_words': 379, 'path': 'notebooks/Multiple Models Notebook.ipynb', 'repo_name': 'ardilacarlosh/amazon-forecast-samples', 'size': '5831'}