<jupyter_start><jupyter_text>Copyright (c) Microsoft Corporation. All rights reserved.Licensed under the MIT License. ![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/forecasting-beer-remote/auto-ml-forecasting-beer-remote.png)  Automated Machine Learning**Beer Production Forecasting** Contents1. [Introduction](Introduction)1. [Setup](Setup)1. [Data](Data)1. [Train](Train)1. [Evaluate](Evaluate)  IntroductionThis notebook demonstrates demand forecasting for Beer Production Dataset using AutoML.AutoML highlights here include using Deep Learning forecasts, Arima, Prophet,  Remote Execution and Remote Inferencing, and working with the `forecast` function. Please also look at the additional forecasting notebooks, which document lagging, rolling windows, forecast quantiles, other ways to use the forecast function, and forecaster deployment.Make sure you have executed the [configuration](../../../configuration.ipynb) before running this notebook.An Enterprise workspace is required for this notebook. To learn more about creating an Enterprise workspace or upgrading to an Enterprise workspace from the Azure portal, please visit our [Workspace page.](https://docs.microsoft.com/azure/machine-learning/service/concept-workspaceupgrade)Notebook synopsis:1. Creating an Experiment in an existing Workspace2. Configuration and remote run of AutoML for a time-series model exploring Regression learners, Arima, Prophet and DNNs4. Evaluating the fitted model using a rolling test   Setup<jupyter_code>import os
import azureml.core
import pandas as pd
import numpy as np
import logging
import warnings

from pandas.tseries.frequencies import to_offset

# Squash warning messages for cleaner output in the notebook
warnings.showwarning = lambda *args, **kwargs: None

from azureml.core.workspace import Workspace
from azureml.core.experiment import Experiment
from azureml.train.automl import AutoMLConfig
from matplotlib import pyplot as plt
from sklearn.metrics import mean_absolute_error, mean_squared_error
from azureml.train.estimator import Estimator<jupyter_output><empty_output><jupyter_text>This sample notebook may use features that are not available in previous versions of the Azure ML SDK.<jupyter_code>print("This notebook was created using version 1.16.0 of the Azure ML SDK")
print("You are currently using version", azureml.core.VERSION, "of the Azure ML SDK")<jupyter_output>This notebook was created using version 1.16.0 of the Azure ML SDK
You are currently using version 1.30.0 of the Azure ML SDK<jupyter_text>As part of the setup you have already created a Workspace. To run AutoML, you also need to create an Experiment. An Experiment corresponds to a prediction problem you are trying to solve, while a Run corresponds to a specific approach to the problem.<jupyter_code>ws = Workspace.from_config()

# choose a name for the run history container in the workspace
experiment_name = 'beer-remote-cpu'

experiment = Experiment(ws, experiment_name)

output = {}
output['Subscription ID'] = ws.subscription_id
output['Workspace'] = ws.name
output['Resource Group'] = ws.resource_group
output['Location'] = ws.location
output['Run History Name'] = experiment_name
pd.set_option('display.max_colwidth', -1)
outputDf = pd.DataFrame(data = output, index = [''])
outputDf.T<jupyter_output><empty_output><jupyter_text>Using AmlComputeYou will need to create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecturecompute-target) for your AutoML run. In this tutorial, you use `AmlCompute` as your training compute resource.<jupyter_code>from azureml.core.compute import ComputeTarget, AmlCompute
from azureml.core.compute_target import ComputeTargetException

# Choose a name for your CPU cluster
cpu_cluster_name = "beer-cluster"

# Verify that cluster does not exist already
try:
    compute_target = ComputeTarget(workspace=ws, name=cpu_cluster_name)
    print('Found existing cluster, use it.')
except ComputeTargetException:
    compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_D2_V2',
                                                           max_nodes=4)
    compute_target = ComputeTarget.create(ws, cpu_cluster_name, compute_config)

compute_target.wait_for_completion(show_output=True)<jupyter_output>InProgress......
SucceededProvisioning operation finished, operation "Succeeded"
Succeeded
AmlCompute wait for completion finished

Minimum number of nodes requested have been provisioned<jupyter_text>DataRead Beer demand data from file, and preview data. Let's set up what we know about the dataset. **Target column** is what we want to forecast.**Time column** is the time axis along which to predict.**Time series identifier columns** are identified by values of the columns listed `time_series_id_column_names`, for example "store" and "item" if your data has multiple time series of sales, one series for each combination of store and item sold.This dataset has only one time series. Please see the [orange juice notebook](https://github.com/Azure/MachineLearningNotebooks/tree/master/how-to-use-azureml/automated-machine-learning/forecasting-orange-juice-sales) for an example of a multi-time series dataset.<jupyter_code>import pandas as pd
from pandas import DataFrame
from pandas import Grouper
from pandas import concat
from pandas.plotting import register_matplotlib_converters

register_matplotlib_converters()
plt.figure(figsize=(20, 10))
plt.tight_layout()

plt.subplot(2, 1, 1)
plt.title('Beer Production By Year')
df = pd.read_csv("Beer_no_valid_split_train.csv", parse_dates=True, index_col= 'DATE').drop(columns='grain')
test_df = pd.read_csv("Beer_no_valid_split_test.csv", parse_dates=True, index_col= 'DATE').drop(columns='grain')
plt.plot(df)

plt.subplot(2, 1, 2)
plt.title('Beer Production By Month')
groups = df.groupby(df.index.month)
months = concat([DataFrame(x[1].values) for x in groups], axis=1)
months = DataFrame(months)
months.columns = range(1,13)
months.boxplot()

plt.show()
target_column_name = 'BeerProduction'
time_column_name = 'DATE'
time_series_id_column_names = []
freq = 'M' #Monthly data<jupyter_output><empty_output><jupyter_text>Split Training data into Train and Validation set and Upload to Datastores<jupyter_code>from helper import split_fraction_by_grain
from helper import split_full_for_forecasting

train, valid = split_full_for_forecasting(df, time_column_name)
train.to_csv("train.csv")
valid.to_csv("valid.csv")
test_df.to_csv("test.csv")

datastore = ws.get_default_datastore()
datastore.upload_files(files = ['./train.csv'], target_path = 'beer-dataset/tabular/', overwrite = True,show_progress = True)
datastore.upload_files(files = ['./valid.csv'], target_path = 'beer-dataset/tabular/', overwrite = True,show_progress = True)
datastore.upload_files(files = ['./test.csv'], target_path = 'beer-dataset/tabular/', overwrite = True,show_progress = True)

from azureml.core import Dataset
train_dataset = Dataset.Tabular.from_delimited_files(path = [(datastore, 'beer-dataset/tabular/train.csv')])
valid_dataset = Dataset.Tabular.from_delimited_files(path = [(datastore, 'beer-dataset/tabular/valid.csv')])
test_dataset = Dataset.Tabular.from_delimited_files(path = [(datastore, 'beer-dataset/tabular/test.csv')])<jupyter_output>Uploading an estimated of 1 files
Uploading ./train.csv
Uploaded ./train.csv, 1 files out of an estimated total of 1
Uploaded 1 files
Uploading an estimated of 1 files
Uploading ./valid.csv
Uploaded ./valid.csv, 1 files out of an estimated total of 1
Uploaded 1 files
Uploading an estimated of 1 files
Uploading ./test.csv
Uploaded ./test.csv, 1 files out of an estimated total of 1
Uploaded 1 files<jupyter_text>Setting forecaster maximum horizon The forecast horizon is the number of periods into the future that the model should predict. Here, we set the horizon to 12 periods (i.e. 12 months). Notice that this is much shorter than the number of months in the test set; we will need to use a rolling test to evaluate the performance on the whole test set. For more discussion of forecast horizons and guiding principles for setting them, please see the [energy demand notebook](https://github.com/Azure/MachineLearningNotebooks/tree/master/how-to-use-azureml/automated-machine-learning/forecasting-energy-demand).<jupyter_code>forecast_horizon = 12<jupyter_output><empty_output><jupyter_text>TrainInstantiate a AutoMLConfig object. This defines the settings and data used to run the experiment.|Property|Description||-|-||**task**|forecasting||**primary_metric**|This is the metric that you want to optimize. Forecasting supports the following primary metrics spearman_correlationnormalized_root_mean_squared_errorr2_scorenormalized_mean_absolute_error|**iteration_timeout_minutes**|Time limit in minutes for each iteration.||**training_data**|Input dataset, containing both features and label column.||**label_column_name**|The name of the label column.||**enable_dnn**|Enable Forecasting DNNs|This step requires an Enterprise workspace to gain access to this feature. To learn more about creating an Enterprise workspace or upgrading to an Enterprise workspace from the Azure portal, please visit our [Workspace page.](https://docs.microsoft.com/azure/machine-learning/service/concept-workspaceupgrade).<jupyter_code>from azureml.automl.core.forecasting_parameters import ForecastingParameters
forecasting_parameters = ForecastingParameters(
    time_column_name=time_column_name, forecast_horizon=forecast_horizon
)

automl_config = AutoMLConfig(task='forecasting',                             
                             primary_metric='normalized_root_mean_squared_error',
                             experiment_timeout_hours = 1,
                             training_data=train_dataset,
                             label_column_name=target_column_name,
                             validation_data=valid_dataset, 
                             verbosity=logging.INFO,
                             compute_target=compute_target,
                             max_concurrent_iterations=4,
                             max_cores_per_iteration=-1,
                             enable_dnn=True,
                             forecasting_parameters=forecasting_parameters)<jupyter_output><empty_output><jupyter_text>We will now run the experiment, starting with 10 iterations of model search. The experiment can be continued for more iterations if more accurate results are required. Validation errors and current status will be shown when setting `show_output=True` and the execution will be synchronous.<jupyter_code>remote_run = experiment.submit(automl_config, show_output= False)
remote_run
# If you need to retrieve a run that already started, use the following code
# from azureml.train.automl.run import AutoMLRun
# remote_run = AutoMLRun(experiment = experiment, run_id = '<replace with your run id>')
remote_run.wait_for_completion()<jupyter_output><empty_output><jupyter_text>Displaying the run objects gives you links to the visual tools in the Azure Portal. Go try them!  Retrieve the Best Model for Each AlgorithmBelow we select the best pipeline from our iterations. The get_output method on automl_classifier returns the best run and the fitted model for the last fit invocation. There are overloads on get_output that allow you to retrieve the best run and fitted model for any logged metric or a particular iteration.<jupyter_code>from helper import get_result_df
summary_df = get_result_df(remote_run)
summary_df
from azureml.core.run import Run
from azureml.widgets import RunDetails
forecast_model = 'TCNForecaster'
if not forecast_model in summary_df['run_id']:
    forecast_model = 'ForecastTCN'
    
best_dnn_run_id = summary_df['run_id'][forecast_model]
best_dnn_run = Run(experiment, best_dnn_run_id)
best_dnn_run.parent
RunDetails(best_dnn_run.parent).show() 
best_dnn_run
RunDetails(best_dnn_run).show()<jupyter_output><empty_output><jupyter_text>Evaluate on Test Data We now use the best fitted model from the AutoML Run to make forecasts for the test set.  We always score on the original dataset whose schema matches the training set schema.<jupyter_code>from azureml.core import Dataset
test_dataset = Dataset.Tabular.from_delimited_files(path = [(datastore, 'beer-dataset/tabular/test.csv')])
# preview the first 3 rows of the dataset
test_dataset.take(5).to_pandas_dataframe()
compute_target = ws.compute_targets['beer-cluster']
test_experiment = Experiment(ws, experiment_name + "_test")
import os
import shutil

script_folder = os.path.join(os.getcwd(), 'inference')
os.makedirs(script_folder, exist_ok=True)
shutil.copy('infer.py', script_folder)
from helper import run_inference

test_run = run_inference(test_experiment, compute_target, script_folder, best_dnn_run, test_dataset, valid_dataset, forecast_horizon,
                 target_column_name, time_column_name, freq)
RunDetails(test_run).show()
from helper import run_multiple_inferences

summary_df = run_multiple_inferences(summary_df, experiment, test_experiment, compute_target, script_folder, test_dataset, 
                  valid_dataset, forecast_horizon, target_column_name, time_column_name, freq)
for run_name, run_summary in summary_df.iterrows():
    print(run_name)
    print(run_summary)
    run_id = run_summary.run_id
    test_run_id = run_summary.test_run_id
    test_run = Run(test_experiment, test_run_id)
    test_run.wait_for_completion()
    test_score = test_run.get_metrics()[run_summary.primary_metric]
    summary_df.loc[summary_df.run_id == run_id, 'Test Score'] = test_score
    print("Test Score: ", test_score)
summary_df<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07930409568684306, 'fraction_numerical': 0.00572671257702066, 'mean_word_length': 5.311070448307411, 'pattern_counts': {'":': 0, '<': 43, '<?xml version=': 0, '>': 43, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '12700', 'license': 'Apache-2.0', 'n_tokens_mistral': 4062, 'n_tokens_neox': 3871, 'n_words': 1224, 'path': 'TCNForecaster Demo/auto-ml-forecasting-beer-remote.ipynb', 'repo_name': 'azeltov/JnJ_TCNForecaster', 'size': '13795'}