<jupyter_start><jupyter_text>Hyper-parameter optimization using HyperOpt and nested runs in MLflowThis notebook demonstrates how to use MLflow for tracking experiment using MLflow in Azure ML when you want to do hyper-parameter optimization using child runs. In MLflow a run can start child runs which will get logged inside of it. This is useful when doing hyper-parameters cause you can get one dedicated run for each trial, while keeping all the runs controlled by a parent run which orchestrates the parameters search. In this example we will show also how you can get artifacts from the child run and log them inside of the parent. We are going to use the library `HyperOpt` for doing the optimization.> **Important:** You may also want to explore using the parameter sweep component from Azure ML for doing hyper-parameter optimization in job-based runs. Check [CLI (v2) sweep job](https://docs.microsoft.com/en-us/azure/machine-learning/reference-yaml-job-sweephttps://docs.microsoft.com/en-us/azure/machine-learning/reference-yaml-job-sweep) for more details.<jupyter_code># Ensure you have the dependencies for this notebook
%pip install -r xgboost_nested_runs.txt
import warnings

warnings.simplefilter("ignore")<jupyter_output><empty_output><jupyter_text>Configuring the experiment Let's get started. It's always a good idea to start by configuring the name of the experiment we are working with in MLflow. Experiments allows you to organize runs in a comprehensive way so you can compare different experiment's runs with different parameters and configuration. MLflow configures the default experiment named "Default" but you can change this name.<jupyter_code>import mlflow

mlflow.set_experiment(experiment_name="heart-condition-classifier")<jupyter_output><empty_output><jupyter_text>Exploring the data<jupyter_code>import pandas as pd
file_url = "http://storage.googleapis.com/download.tensorflow.org/data/heart.csv"
df = pd.read_csv(file_url)
df<jupyter_output><empty_output><jupyter_text>As we can see, some of the variables are categorical. To make it simpler for our model to handle these values, let's use their encoded values:<jupyter_code>df["thal"] = df["thal"].astype("category").cat.codes<jupyter_output><empty_output><jupyter_text>The encoded values looks then as follows:<jupyter_code>df["thal"].unique()<jupyter_output><empty_output><jupyter_text>Let's split our dataset in train and test, so we can assess the performance of the model without overfitting the dataset.<jupyter_code>from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
    df.drop("target", axis=1), df["target"], test_size=0.3
)<jupyter_output><empty_output><jupyter_text>Defining the search spaceSince we are going to do hyper-parameter optimization, we need to define the search space we are going to use for each parameter. In this simple example, we are optimizing the following parameters:<jupyter_code>from hyperopt import hp

space = {
    "learning_rate": hp.uniform("learning_rate", 1e-5, 1e-1),
    "max_leaves": hp.choice("max_leaves ", options=[2, 5, 9, 10, 20, 30]),
}<jupyter_output><empty_output><jupyter_text>Training a model We are going to place all the training code inside of a function so we can use for searching the best parameters configuration. HyperOpt requires us to have **the first positional parameter of this function to receive the parameters of each trial**. Here we are placing them in an argument called `params` which is a dictionary. The rest of the parameters needs to be named parameters. You need to comply with this convention.The function needs to return either a float value which correspond to the loss of the trial, or a dictionary which specific keys. Read the documentation of HyperOpt for other options and details. We are using the dictionary option here cause we want to return some extra information about each trial which is the run ID in MLflow corresponding to the trial. Pay attention how this is indicated in a key `attachments`. This will be handy later to download assets for each run.It's worth noticing how the MLflow run is started with the parameter `nested=True`. This indicates to MLflow that the current run is a child run of a higher level one. The parent run controls the parameter optimization while each child focus on training one particular model. Also, notice how we are using `autolog` to log metrics, parameters and models automatically for us.<jupyter_code>from xgboost import XGBClassifier
from sklearn.metrics import accuracy_score, recall_score, roc_curve, auc
from typing import Any, Dict
from hyperopt import STATUS_OK


def train(
    params: Dict[str, Any], X_train: pd.DataFrame, X_test, y_train: pd.DataFrame, y_test
) -> Dict[str, Any]:
    run_name = "_".join(params.keys())
    with mlflow.start_run(nested=True, run_name=f"hyperopt_{run_name}") as child_run:
        # Using autolog for xgboost model
        mlflow.xgboost.autolog(silent=True)

        # Using autolog for the metrics calculated with scikit-learn
        mlflow.sklearn.autolog(silent=True)

        model = XGBClassifier(use_label_encoder=False, eval_metric="logloss", **params)
        model.fit(X_train, y_train, eval_set=[(X_test, y_test)], verbose=False)
        y_pred = model.predict(X_test)

        # Calculate some metrics
        accuracy = accuracy_score(y_test, y_pred)
        recall = recall_score(y_test, y_pred)
        fpr, tpr, thresholds = roc_curve(y_test, y_pred, pos_label=2)
        auc_metric = auc(fpr, tpr)

        # Return a dictionary HyperOpt can use to better inform the search
        # We are also returning the run_id of each iteration so we can use it
        # later to access the best run from Mlflow.
        return {
            "status": STATUS_OK,
            "loss": -recall,
            "attachments": {"run_id": child_run.info.run_id},
        }<jupyter_output><empty_output><jupyter_text>Creating a search functionWe are going to create a search function for doing the hyper-parameter optimization. This is not required -  you can place all the code directly on notebook's cells but it is a good choice. The function will:1. Start a new MLflow run to do the optimization this parent run is the one in charge of doing the overall optimization. Notice that if you are running inside of an AML job, you are should not start the run manually cause it is already done for you.2. Creates a `fmin` object which will do the optimization using HyperOpt.3. Gets the best trial and best run ID from MLflow.4. Downloads the best model from the child run which was better and register it in the parent run.5. Logs the best child run in the parent run along with the best hyper-parameters configuration we found.6. Logs some extra artifacts (see details later).<jupyter_code>import hyperopt.plotting
from hyperopt import fmin, tpe, rand, Trials
from functools import partial
from matplotlib import pyplot as plt


def search(
    space: Dict[str, Any], X_train: pd.DataFrame, X_test, y_train: pd.DataFrame, y_test
):
    with mlflow.start_run() as run:
        print("Working in MLflow run:", run.info.run_id)
        trials = Trials()
        best_params = fmin(
            fn=partial(
                train, X_train=X_train, X_test=X_test, y_train=y_train, y_test=y_test
            ),
            space=space,
            algo=tpe.suggest,
            max_evals=5,
            trials=trials,
        )
        best_run_id = trials.trial_attachments(trials.best_trial)["run_id"]
        best_run = mlflow.get_run(best_run_id)

        client = mlflow.tracking.MlflowClient()
        best_model = client.download_artifacts(best_run_id, path="model")

        mlflow.log_param("best_run_id", best_run_id)
        mlflow.log_params({f"best_{p}": v for p, v in best_params.items()})
        mlflow.log_metric("best_loss", trials.best_trial["result"]["loss"])
        mlflow.log_artifacts(local_dir=best_model, artifact_path="model")

        fig = plt.figure()
        ax = fig.add_subplot()
        hyperopt.plotting.main_plot_histogram(trials, do_show=False)
        mlflow.log_figure(fig, "loss_histogram.png")

        return trials, best_params<jupyter_output><empty_output><jupyter_text>Some things to notice about `fmin`:- `fmin` is used to minimize the loss according to the objective indicated in the function `train`. If you pay closer look at how it is indicated, we wrapped the function with `partial`. This is because the function receives extra parameters we are not looking to optimize, but they are directly passed to the function. We are passing here the train and test splits for the features and labels.- `algo` is indicated with `tpe.suggest`, meaning that we are relying on HyperOpt to guess which is the best algorithm to use based on the space we are using.- `max_evals` indicates how many trials will be executed. We are not using early termination in this example, but you are welcome to explore it.- `trials` is passed to the `fmin` function. This is not required but when the argument is present, all the debug information about the trials is stored there. It's a best practice to always indicate it.Some things to notice in the `search` function:- The function `fmin` returns the best parameters combination, but it doesn't provide information about which is the best run. For that, we use `trials.best_trial`. We also added `run_id` as an attachment, so we can use it to get the best run inside of MLflow. This is convenient because we don't have to query all the child runs to guess which is the best one.- We are downloading the best model so we can log it in the parent run. This is done using the function `client.download_artifacts` for the child run and then `mlflow.log_artifacts` for the parent run. So the best model is logged twice, inside of the parent run and also inside of the child run that generated it.- For further reference, the best run ID is also logged as a parameters with the method `mlflow.log_param`.- HyperOpt also has some features for plotting the results. We are logging this plots using `mlflow.log_figure` which will store these plots as `png` files automatically for us. Let's run the search:<jupyter_code>trials, best_params = search(space, X_train, X_test, y_train, y_test)<jupyter_output>Working in MLflow run: 7eca10a6-d166-4714-a251-10f1585eb101
100%|███████████████████████████████████████████████████████████████████████████████████| 5/5 [01:05<00:00, 13.15s/trial, best loss: -0.6]
Showing Histogram of 5 jobs<jupyter_text>Exploring the results in Azure ML Using StudioWhen child runs are used, they are displayed inside of the experiment's run in the tab `child jobs`:![](assets/hyperopt_child_runs.png)You can then view any of the child runs we created and explore metrics, parameters and more:![](assets/hyperopt_child.png)  Exploring child runs programmaticallyWe can also access the runs programmatically using MLflow client:<jupyter_code>hyperopt_run = mlflow.last_active_run()<jupyter_output><empty_output><jupyter_text>We can see all the child runs by using the tag `mlflow.parentRunId` that gets automatically populated by MLflow and Azure ML.<jupyter_code>child_runs = mlflow.search_runs(
    filter_string=f"tags.mlflow.parentRunId='{hyperopt_run.info.run_id}'"
)
child_runs<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07692987104751811, 'fraction_numerical': 0.006006006006006006, 'mean_word_length': 4.520721599219892, 'pattern_counts': {'":': 7, '<': 49, '<?xml version=': 0, '>': 50, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '226623', 'license': 'MIT', 'n_tokens_mistral': 3303, 'n_tokens_neox': 3143, 'n_words': 1358, 'path': 'notebooks/using-mlflow/train-with-mlflow/xgboost_nested_runs.ipynb', 'repo_name': 'jplummer01/azureml-examples', 'size': '11322'}