<jupyter_start><jupyter_text>Neptune + Optuna Before you start Install dependencies<jupyter_code>! pip install lightgbm==2.2.3 neptune-client[optuna] optuna==2.8.0 plotly==4.4.1<jupyter_output><empty_output><jupyter_text>Import libraries<jupyter_code>import lightgbm as lgb
import optuna

from sklearn.datasets import load_breast_cancer
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import train_test_split<jupyter_output><empty_output><jupyter_text>Create a sample `objective` function for Optuna<jupyter_code>def objective(trial):
    data, target = load_breast_cancer(return_X_y=True)
    train_x, test_x, train_y, test_y = train_test_split(data, target, test_size=0.25)
    dtrain = lgb.Dataset(train_x, label=train_y)

    param = {
        "verbose": -1,
        "objective": "binary",
        "metric": "binary_logloss",
        "num_leaves": trial.suggest_int("num_leaves", 2, 256),
        "feature_fraction": trial.suggest_uniform("feature_fraction", 0.2, 1.0),
        "bagging_fraction": trial.suggest_uniform("bagging_fraction", 0.2, 1.0),
        "min_child_samples": trial.suggest_int("min_child_samples", 3, 100),
    }

    gbm = lgb.train(param, dtrain)
    preds = gbm.predict(test_x)
    accuracy = roc_auc_score(test_y, preds)

    return accuracy<jupyter_output><empty_output><jupyter_text>Quickstart  Create a Neptune Run Create a Neptune project and get your API token (optional)To log metadata to the Neptune project, you need the `project` name and the `api_token`.To make this example easy to follow, we have created a public project 'common/optuna-integration' and a shared user 'neptuner' with the API token '<PASSWORD>'.```pythonrun = neptune.init(api_token='<PASSWORD>', project='common/optuna-integration')```Add a snippet at the top of your script:<jupyter_code>import neptune.new as neptune

run = neptune.init(
    api_token="<PASSWORD>", project="common/optuna-integration"
)  # you can pass your credentials here<jupyter_output><empty_output><jupyter_text>Running this cell creates a Run in Neptune, and you can log model building metadata to it.**Click on the link above to open the Run in Neptune UI**. For now, it is empty, but you should keep the tab open to see what happens next.  Create a Neptune project and get your API token (optional)To log to your own Neptune project:* [Create a Neptune account](https://neptune.ai/register)* [Find your API token](https://docs.neptune.ai/getting-started/installationauthentication-neptune-api-token)* [Find your project name](https://docs.neptune.ai/getting-started/installationsetting-the-project-name)* Pass your credentials to `project` and `api_token` arguments of `neptune.init()````pythonrun = neptune.init(api_token='', project='')  pass your credentials```  Initialize the NeptuneCallback<jupyter_code>import neptune.new.integrations.optuna as optuna_utils

neptune_callback = optuna_utils.NeptuneCallback(run)<jupyter_output><empty_output><jupyter_text>Pass the NeptuneCallback to Optuna Study `.optimize()` method<jupyter_code>study = optuna.create_study(direction="maximize")
study.optimize(objective, n_trials=20, callbacks=[neptune_callback])<jupyter_output><empty_output><jupyter_text>You can view the logging live in the Neptune tab once Optuna you run the below cell  Stop logging When you track your ML runs with Neptune in Jupyter notebooks, you need to stop the Run by `run.stop()` explicitly.If you are running Neptune in regular '.py' scripts, it will stop automatically when your code finishes.<jupyter_code>run.stop()<jupyter_output><empty_output><jupyter_text>More Options Customize which plots you want to log and how oftenBy default, `NeptuneCallback` creates and logs all of the plots from `optuna.visualizations`, but it adds overhead to your Optuna sweep.You can decide which plots to create and log and how often you want to do that with:* `plot_update_freq` argument: pass integer k to update plots every k trials or 'never' to not log any plots* `log_plot_contour`, `log_plot_slice`, and other `log_{OPTUNA_PLOT_FUNCTION}` arguments: pass 'False', and the plots will not be created or logged<jupyter_code># Create a Neptune Run
run = neptune.init(
    api_token="<PASSWORD>", project="common/optuna-integration"
)  # you can pass your credentials here

# Create a NeptuneCallback for Optuna
neptune_callback = optuna_utils.NeptuneCallback(
    run,
    plots_update_freq=10,  # create/log plots every 10 trials
    log_plot_slice=False,  # do not create/log plot_slice
    log_plot_contour=False,  # do not create/log plot_contour
)

# Pass NeptuneCallback to Optuna Study .optimize()
study = optuna.create_study(direction="maximize")
study.optimize(objective, n_trials=50, callbacks=[neptune_callback])

# Stop logging to a Neptune Run
run.stop()<jupyter_output><empty_output><jupyter_text>Log charts and study object after the sweepIf you want to log study metadata after the Study was finished you can use the `.log_study_metadata()`.`.log_study_metadata()` function logs the same things that `NeptuneCallback` logs, and you can customize what is logged with similar flags.<jupyter_code># Create a Run
run = neptune.init(
    api_token="<PASSWORD>", project="common/optuna-integration"
)  # you can pass your credentials here

# Run Optuna with Neptune Callback
study = optuna.create_study(direction="maximize")
study.optimize(objective, n_trials=10)

# Log Optuna charts and study object after the sweep is complete
optuna_utils.log_study_metadata(study, run, log_plot_contour=False)

# Stop logging
run.stop()<jupyter_output><empty_output><jupyter_text>Load the Optuna Study from an existing Neptune RunIf you logged the Optuna Study to Neptune, you can load the Study directly from the Run with `load_study_from_run()` function and continue working with it.It works both for Optuna `InMemoryStorage` and database storage.<jupyter_code># Fetch an existing Neptune Run
run = neptune.init(
    api_token="<PASSWORD>",
    project="common/optuna-integration",  # you can pass your credentials here
    run="NEP1-513",
)  # You can pass Run ID for some other Run

# Run Optuna with Neptune Callback
study = optuna_utils.load_study_from_run(run)

# Create callback to log advanced options during the sweep
neptune_callback = optuna_utils.NeptuneCallback(run)

# Continue logging to the same run
study.optimize(objective, n_trials=10, callbacks=[neptune_callback])

# Stop logging
run.stop()<jupyter_output><empty_output><jupyter_text>Keep track of both study-level and trial-level RunsYou can log trial-level information to separate Neptune Runs and have a main Run for the study-level information.**Warning**The sweep will take longer as each trial-level Run needs to synchronize with Neptune.   Create a unique sweep ID<jupyter_code>import uuid

sweep_id = uuid.uuid1()
print("sweep-id: ", sweep_id)<jupyter_output><empty_output><jupyter_text>Create a study-level Neptune Run<jupyter_code>run_study_level = neptune.init(
    api_token="<PASSWORD>", project="common/optuna-integration"
)  # you can pass your credentials here<jupyter_output><empty_output><jupyter_text>Log the sweep ID to the study-level Run You can also add a tag 'study-level' to distinguish between the study-level and trial-level runs for the sweap.<jupyter_code>run_study_level["sys/tags"].add("study-level")
run_study_level["sweep-id"] = sweep_id<jupyter_output><empty_output><jupyter_text>Create an objective function that logs each trial to Neptune as a RunInside of the objective function, you need to:create a trial-level Neptune Run* log the sweep ID and a tag 'trial-level' to distinguish between study-level and trial-level Runs* log parameters and scores to the trial-level Run* stop the trial-level Run<jupyter_code>def objective_with_logging(trial):
    data, target = load_breast_cancer(return_X_y=True)
    train_x, test_x, train_y, test_y = train_test_split(data, target, test_size=0.25)
    dtrain = lgb.Dataset(train_x, label=train_y)

    param = {
        "verbose": -1,
        "objective": "binary",
        "metric": "binary_logloss",
        "num_leaves": trial.suggest_int("num_leaves", 2, 256),
        "feature_fraction": trial.suggest_uniform("feature_fraction", 0.2, 1.0),
        "bagging_fraction": trial.suggest_uniform("bagging_fraction", 0.2, 1.0),
        "min_child_samples": trial.suggest_int("min_child_samples", 3, 100),
    }

    # create a trial-level Run
    run_trial_level = neptune.init(
        api_token="<PASSWORD>", project="common/optuna-integration"
    )

    # log sweep id to trial-level Run
    run_trial_level["sys/tags"].add("trial-level")
    run_trial_level["sweep-id"] = sweep_id

    # log parameters of a trial-level Run
    run_trial_level["parameters"] = param

    # run model training
    gbm = lgb.train(param, dtrain)
    preds = gbm.predict(test_x)
    accuracy = roc_auc_score(test_y, preds)

    # log score of a trial-level Run
    run_trial_level["score"] = accuracy

    # stop trial-level Run
    run_trial_level.stop()

    return accuracy<jupyter_output><empty_output><jupyter_text>Create a study-level NeptuneCallback<jupyter_code>neptune_callback = optuna_utils.NeptuneCallback(run_study_level)<jupyter_output><empty_output><jupyter_text>Pass the NeptuneCallback to the `study.optimize()` method and run the parameter sweep<jupyter_code>study = optuna.create_study(direction="maximize")
study.optimize(objective_with_logging, n_trials=20, callbacks=[neptune_callback])<jupyter_output><empty_output><jupyter_text>Stop logging to the Neptune Run<jupyter_code>run_study_level.stop()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0955108842828872, 'fraction_numerical': 0.007082595562962191, 'mean_word_length': 5.617505168849069, 'pattern_counts': {'":': 14, '<': 77, '<?xml version=': 0, '>': 77, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '468759', 'license': 'MIT', 'n_tokens_mistral': 3183, 'n_tokens_neox': 3045, 'n_words': 879, 'path': 'integrations-and-supported-tools/optuna/notebooks/Neptune_Optuna_integration.ipynb', 'repo_name': 'neptune-ai/examples', 'size': '9593'}