<jupyter_start><jupyter_text>AutoML: Train "the best" Time-Series Forecasting model for the Energy Demand Dataset.**Requirements** - In order to benefit from this tutorial, you will need:- A basic understanding of Machine Learning- An Azure account with an active subscription. [Create an account for free](https://azure.microsoft.com/free/?WT.mc_id=A261C142F)- An Azure ML workspace. [Check this notebook for creating a workspace](../../../resources/workspace/workspace.ipynb) - A Compute Cluster. [Check this notebook to create a compute cluster](../../../resources/compute/compute.ipynb)- A python environment- Installed Azure Machine Learning Python SDK v2 - [install instructions](../../../README.md) - check the getting started section**Learning Objectives** - By the end of this tutorial, you should be able to:- Connect to your AML workspace from the Python SDK- Create an `AutoML time-series forecasting Job` with the 'forecasting()' factory-fuction.- Train the model using AmlCompute by submitting/running the AutoML forecasting training job- Obtaing the model and score predictions with it**Motivations** - This notebook explains how to setup and run an AutoML forecasting job. This is one of the nine ML-tasks supported by AutoML. Other ML-tasks are 'regression', 'classification', 'image classification', 'image object detection', 'nlp text classification', etc.In this example we use the associated New York City energy demand dataset to showcase how you can use AutoML for a simple forecasting problem and explore the results. The goal is predict the energy demand for the next 48 hours based on historic time-series data.  1. Connect to Azure Machine Learning WorkspaceThe [workspace](https://docs.microsoft.com/en-us/azure/machine-learning/concept-workspace) is the top-level resource for Azure Machine Learning, providing a centralized place to work with all the artifacts you create when you use Azure Machine Learning. In this section we will connect to the workspace in which the job will be run. 1.1. Import the required libraries<jupyter_code># Import required libraries
from azure.identity import DefaultAzureCredential
from azure.ai.ml import MLClient

from azure.ai.ml.constants import AssetTypes
from azure.ai.ml import automl
from azure.ai.ml import Input<jupyter_output><empty_output><jupyter_text>1.2. Configure workspace details and get a handle to the workspaceTo connect to a workspace, we need identifier parameters - a subscription, resource group and workspace name. We will use these details in the `MLClient` from `azure.ai.ml` to get a handle to the required Azure Machine Learning workspace. We use the default [default azure authentication](https://docs.microsoft.com/en-us/python/api/azure-identity/azure.identity.defaultazurecredential?view=azure-python) for this tutorial. Check the [configuration notebook](../../configuration.ipynb) for more details on how to configure credentials and connect to a workspace.<jupyter_code>credential = DefaultAzureCredential()
ml_client = None
try:
    ml_client = MLClient.from_config(credential)
except Exception as ex:
    print(ex)
    # Enter details of your AML workspace
    subscription_id = "<SUBSCRIPTION_ID>"
    resource_group = "<RESOURCE_GROUP>"
    workspace = "<AML_WORKSPACE_NAME>"
    ml_client = MLClient(credential, subscription_id, resource_group, workspace)<jupyter_output><empty_output><jupyter_text>Show Azure ML Workspace information<jupyter_code>import pandas as pd

workspace = ml_client.workspaces.get(name=ml_client.workspace_name)

output = {}
output["Workspace"] = ml_client.workspace_name
output["Subscription ID"] = ml_client.connections._subscription_id
output["Resource Group"] = workspace.resource_group
output["Location"] = workspace.location
pd.set_option("display.max_colwidth", None)
outputDf = pd.DataFrame(data=output, index=[""])
outputDf.T<jupyter_output><empty_output><jupyter_text>2. DataWe will use energy consumption [data from New York City](http://mis.nyiso.com/public/P-58Blist.htm) for model training. The data is stored in a tabular format and includes energy demand and basic weather data at an hourly frequency. With Azure Machine Learning MLTables you can keep a single copy of data in your storage, easily access data during model training, share data and collaborate with other users. Below, we will upload the data by creating an MLTable to be used for training.**NOTE:** In this PRIVATE PREVIEW we're defining the MLTable in a separate folder and .YAML file.In later versions, you'll be able to do it all in Python APIs.<jupyter_code># Training MLTable defined locally, with local data to be uploaded
my_training_data_input = Input(
    type=AssetTypes.MLTABLE, path="./data/training-mltable-folder"
)

# Training MLTable defined locally, with local data to be uploaded
my_validation_data_input = Input(
    type=AssetTypes.MLTABLE, path="./data/validation-mltable-folder"
)

# WITH REMOTE PATH
# my_training_data_input  = Input(type=AssetTypes.MLTABLE, path="azureml://datastores/workspaceblobstore/paths/my-forecasting-mltable")<jupyter_output><empty_output><jupyter_text>3. Configure and run the AutoML Forecasting training jobIn this section we will configure and run the AutoML job, for training the model. 3.1 Configure the job through the forecasting() factory function forecasting() function parameters:The `forecasting()` factory function allows user to configure AutoML for the forecasting task for the most common scenarios with the following properties.- `target_column_name` - The name of the column to target for predictions. It must always be specified. This parameter is applicable to 'training_data', 'validation_data' and 'test_data'.- `primary_metric` - The metric that AutoML will optimize for model selection.- `training_data` - The data to be used for training. It should contain both training feature columns and a target column. Optionally, this data can be split for segregating a validation or test dataset. You can use a registered MLTable in the workspace using the format ':' OR you can use a local file or folder as a MLTable. For e.g Input(mltable='my_mltable:1') OR Input(mltable=MLTable(local_path="./data"))The parameter 'training_data' must always be provided.- `compute` - The compute on which the AutoML job will run. In this example we are using a compute called 'cpu-cluster' present in the workspace. You can replace it any other compute in the workspace. - `name` - The name of the Job/Run. This is an optional property. If not specified, a random name will be generated.- `experiment_name` - The name of the Experiment. An Experiment is like a folder with multiple runs in Azure ML Workspace that should be related to the same logical machine learning experiment. set_limits() parameters:This is an optional configuration method to configure limits parameters such as timeouts.         - timeout_minutes - Maximum amount of time in minutes that the whole AutoML job can take before the job terminates. This timeout includes setup, featurization and training runs but does not include the ensembling and model explainability runs at the end of the process since those actions need to happen once all the trials (children jobs) are done. If not specified, the default job's total timeout is 6 days (8,640 minutes). To specify a timeout less than or equal to 1 hour (60 minutes), make sure your dataset's size is not greater than 10,000,000 (rows times column) or an error results.- trial_timeout_minutes - Maximum time in minutes that each trial (child job) can run for before it terminates. If not specified, a value of 1 month or 43200 minutes is used.    - max_trials - The maximum number of trials/runs each with a different combination of algorithm and hyperparameters to try during an AutoML job. If not specified, the default is 1000 trials. If using 'enable_early_termination' the number of trials used can be smaller.    - max_concurrent_trials - Represents the maximum number of trials (children jobs) that would be executed in parallel. It's a good practice to match this number with the number of nodes your cluster.    - enable_early_termination - Whether to enable early termination if the score is not improving in the short term.       Specialized Forecasting ParametersTo define forecasting parameters for your experiment training, you can leverage the .set_forecast_settings() method. The table below details the forecasting parameters we will be passing into our experiment.|Property|Description||-|-||**time_column_name**|The name of your time column.||**forecast_horizon**|The forecast horizon is how many periods forward you would like to forecast. This integer horizon is in units of the timeseries frequency (e.g. daily, weekly).||**frequency**|Forecast frequency. This optional parameter represents the period with which the forecast is desired, for example, daily, weekly, yearly, etc. Use this parameter for the correction of time series containing irregular data points or for padding of short time series. The frequency needs to be a pandas offset alias. Please refer to [pandas documentation](https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.htmldateoffset-objects) for more information.  Advanced Forecasting Training  Using lags and rolling window featuresThis training is also using the **target lags**, that is the previous values of the target variables, meaning the prediction uses a horizon. We therefore must still specify the `forecast_horizon` that the model will learn to forecast. The `target_lags` keyword specifies how far back we will construct the lags of the target variable, and the `target_rolling_window_size` specifies the size of the rolling window over which we will generate the `max`, `min` and `sum` features.This notebook uses the .set_training(blocked_training_algorithms=...) parameter to exclude some models that take a longer time to train on this dataset.  You can choose to remove models from the blocked_training_algorithms list but you may need to increase the trial_timeout_minutes parameter value to get results.<jupyter_code># general job parameters
compute_name = "cpu-cluster"
max_trials = 5
exp_name = "dpv2-forecasting-experiment"
# Create the AutoML forecasting job with the related factory-function.

forecasting_job = automl.forecasting(
    compute=compute_name,
    # name="dpv2-forecasting-job-02",
    experiment_name=exp_name,
    training_data=my_training_data_input,
    # validation_data = my_validation_data_input,
    target_column_name="demand",
    primary_metric="NormalizedRootMeanSquaredError",
    n_cross_validations=3,
    enable_model_explainability=True,
    tags={"my_custom_tag": "My custom value"},
)

# Limits are all optional
forecasting_job.set_limits(
    timeout_minutes=600,
    trial_timeout_minutes=20,
    max_trials=max_trials,
    # max_concurrent_trials = 4,
    # max_cores_per_trial: -1,
    enable_early_termination=True,
)

# Specialized properties for Time Series Forecasting training
forecasting_job.set_forecast_settings(
    time_column_name="timeStamp",
    forecast_horizon=48,
    frequency="H",
    target_lags=[12],
    target_rolling_window_size=4,
    # ADDITIONAL FORECASTING TRAINING PARAMS ---
    # time_series_id_column_names=["tid1", "tid2", "tid2"],
    # short_series_handling_config=ShortSeriesHandlingConfiguration.DROP,
    # use_stl="season",
    # seasonality=3,
)

# Training properties are optional
forecasting_job.set_training(blocked_training_algorithms=["ExtremeRandomTrees"])<jupyter_output><empty_output><jupyter_text>2.2 Run the CommandUsing the `MLClient` created earlier, we will now run this Command in the workspace.<jupyter_code># Submit the AutoML job
returned_job = ml_client.jobs.create_or_update(
    forecasting_job
)  # submit the job to the backend

print(f"Created job: {returned_job}")
# Wait until AutoML training runs are finished
ml_client.jobs.stream(returned_job.name)<jupyter_output><empty_output><jupyter_text>3. Retrieve the Best Trial (Best Model's trial/run)Use the MLFLowClient to access the results (such as Models, Artifacts, Metrics) of a previously completed AutoML Trial.  Initialize MLFlow ClientThe models and artifacts that are produced by AutoML can be accessed via the MLFlow interface. Initialize the MLFlow client here, and set the backend as Azure ML, via. the MLFlow Client.*IMPORTANT*, you need to have installed the latest MLFlow packages with:    pip install azureml-mlflow    pip install mlflow  Obtain the tracking URI for MLFlow<jupyter_code>import mlflow

# Obtain the tracking URL from MLClient
MLFLOW_TRACKING_URI = ml_client.workspaces.get(
    name=ml_client.workspace_name
).mlflow_tracking_uri

print(MLFLOW_TRACKING_URI)
# Set the MLFLOW TRACKING URI

mlflow.set_tracking_uri(MLFLOW_TRACKING_URI)

print("\nCurrent tracking uri: {}".format(mlflow.get_tracking_uri()))
from mlflow.tracking.client import MlflowClient

# Initialize MLFlow client
mlflow_client = MlflowClient()<jupyter_output><empty_output><jupyter_text>Get the AutoML parent Job<jupyter_code>job_name = returned_job.name

# Example if providing an specific Job name/ID
# job_name = "591640e8-0f88-49c5-adaa-39b9b9d75531"

# Get the parent run
mlflow_parent_run = mlflow_client.get_run(job_name)

print("Parent Run: ")
print(mlflow_parent_run)
# Print parent run tags. 'automl_best_child_run_id' tag should be there.
print(mlflow_parent_run.data.tags)<jupyter_output><empty_output><jupyter_text>Get the AutoML best child run<jupyter_code># Get the best model's child run

best_child_run_id = mlflow_parent_run.data.tags["automl_best_child_run_id"]
print("Found best child run id: ", best_child_run_id)

best_run = mlflow_client.get_run(best_child_run_id)

print("Best child run: ")
print(best_run)<jupyter_output><empty_output><jupyter_text>Get best model run's metricsAccess the results (such as Models, Artifacts, Metrics) of a previously completed AutoML Run.<jupyter_code>import pandas as pd

pd.DataFrame(best_run.data.metrics, index=[0]).T<jupyter_output><empty_output><jupyter_text>Download the best model locallyAccess the results (such as Models, Artifacts, Metrics) of a previously completed AutoML Run.<jupyter_code># Create local folder
local_dir = "./artifact_downloads"
if not os.path.exists(local_dir):
    os.mkdir(local_dir)
# Download run's artifacts/outputs
local_path = mlflow_client.download_artifacts(
    best_run.info.run_id, "outputs", local_dir
)
print("Artifacts downloaded in: {}".format(local_path))
print("Artifacts: {}".format(os.listdir(local_path)))
import os

# Show the contents of the MLFlow model folder
os.listdir("./artifact_downloads/outputs/mlflow-model")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07285354395043235, 'fraction_numerical': 0.00646830530401035, 'mean_word_length': 5.422387407083516, 'pattern_counts': {'":': 1, '<': 48, '<?xml version=': 0, '>': 48, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '192989', 'license': 'MIT', 'n_tokens_mistral': 4107, 'n_tokens_neox': 3952, 'n_words': 1619, 'path': 'sdk/jobs/automl-standalone-jobs/automl-forecasting-task-energy-demand/automl-forecasting-task-energy-demand-advanced-mlflow.ipynb', 'repo_name': 'jplummer01/azureml-examples', 'size': '14687'}