<jupyter_start><jupyter_text>Hyperparameter Optimization Tutorial     The lab client library integrates various hyperparameter optimization libraries to make it simple to tune hyperparameters. **In this notebook:*** Create fasttext classification model based on Facebooks Fasttext Implementation and optimize Hyperparameters via:    * [Sklearn Grid Search](Run-Simple-Grid-Search)    * [Sklearn Random Search](Run-Simple-Random-Search)    * [Hyperopt Optimizer](Run-Hyperopt-Optimizer)    * [Skopt Optimizer](Run-Skopt-Optimizer)    * [Optuna Optimizer](Run-Optuna-Optimizer)  DependenciesInstall, load, and initialize all required dependencies for this experiment. Install Dependencies- _Please use a Python 3 kernel for the notebook_To build fastText for Python:``` bash%%bashmkdir /resources/fasttext-0-2cd /resources/fasttext-0-2git clone https://github.com/facebookresearch/fastText.gitcd fastTextpip install .```  Import Dependencies<jupyter_code># System libraries
from __future__ import division, print_function
import logging, os, sys, time

# Enable logging
logging.basicConfig(format='[%(levelname)s] %(message)s', level=logging.INFO, stream=sys.stdout)

# Intialize tqdm to always use the notebook progress bar
import tqdm
tqdm.tqdm = tqdm.tqdm_notebook

%load_ext autoreload
%autoreload 2

# Third-party libraries
import numpy as np
import pandas as pd

import fastText

# Lab libraries
from lab_client import Environment
from lab_client.commons import file_utils<jupyter_output><empty_output><jupyter_text>Initialize EnvironmentTo run this code you need to have created a project within the Lab instance and replace LAB_PROJECT with your project!<jupyter_code># Initialize environment
env = Environment(project="LAB_PROJECT"  # Lab project you want to work on
                  # Only required in stand-alone workspace deployments
                  # lab_endpoint="LAB_ENDPOINT", # Lab endpoint url: e.g. http://10.2.3.45:8091
                  # lab_api_token="LAB_API_TOKEN"
                 )

# Initialize experiment
exp = env.create_experiment('Hyperopt Tutorial')<jupyter_output><empty_output><jupyter_text>Load DataDownload, explore, and prepare all required data for the experiment in this section.To run this code you need to have uploaded this dataset to the Lab instance!<jupyter_code># Get data from remote storage and read into basic pandas dataframe
dataset_key = 'datasets/news-categorized.csv'
text_corpus_path = env.get_file(dataset_key)

# Load dataset via pandas
df = pd.read_csv(text_corpus_path, sep=";")

# Configure dataset transfomration
dataset_config = {
    'train_size':0.75
}

# Split the dataset into train (95%), and test (5%) based on dataset configuration
train_df, test_df = np.split(df.sample(frac=1, random_state=2), [int(dataset_config['train_size']*len(df))])

# Create training data file for fasttext - fasttext requires a text file as input
train_data_path = exp.create_file_path('train_fasttext.csv')
with open(train_data_path, 'w') as f:
    for index, row in train_df.iterrows():
        if row["label"] and row["text"]:
            f.write("__label__" + row["label"].strip() + ' ' +  str(row["text"]) +  '\n')

# Show sample
train_df.head()<jupyter_output><empty_output><jupyter_text>Define Model TrainingImplementation of model training and unified model creation<jupyter_code>from unified_model.predefined_models.fasttext_models import FasttextClassifier

# use this if you are using the not preprocessed content
def create_unified_model(ft_model_path: str, model_name: str):
    def preprocess(data, **kwargs):
        return str(data)
    
    return FasttextClassifier(ft_classifier_path=ft_model_path, 
                              name=model_name, 
                              transform_func=preprocess)


def train(exp, params, artifacts):
    start = time.time()
    
    # train model
    model = fastText.train_supervised(
        input = artifacts['train_data_path'], 
        lr = params['learning_rate'],
        dim = params['vector_dim'],
        ws = params['window_size'],
        epoch = params['epochs'],
        minCount = params['min_count'],
        neg = params['negativ_sampling'],
        wordNgrams = params['word_ngrams'],
        bucket = params['bucket'],
        thread = params['threads'],
        verbose = 2
    )
    
    exp.log_metric("train_time", time.time()-start)
    
    # Save Model
    model_name = file_utils.get_filename(dataset_key) + "_ft_classifier"
    model_path = exp.create_file_path(model_name)
    
    model.save_model(model_path)
    unified_model = create_unified_model(model_path, model_name)
    
    # add trained model instance to experiment, so it can be accessed
    exp.add_artifact("trained_model", unified_model)
    
    # Test Model
    test_df = artifacts["test_data"]
    exp.log_metrics(unified_model.evaluate(test_df['text'].tolist(), test_df['label'].tolist(), per_label=False))
    
    # return the most descriptive metric (main objective) for the experiment
    accuracy = exp.metrics["micro_recall@k5"]
    print("Model training finished with accuracy: " + str(accuracy))
    return accuracy<jupyter_output><empty_output><jupyter_text>Run Param Optimization Experiments  Default Experiment Configuration<jupyter_code># Define parameter configuration for experiment run
params = {
    'learning_rate': 0.1,
    'vector_dim': 300,
    'window_size': 7,
    'epochs': 10,
    'min_count': 3,
    'negativ_sampling': 7,
    'word_ngrams': 1,
    'bucket': 0,
    'threads': 30
}

# Add dataset configuration to experiment parameters
params.update(dataset_config)

# add dataframes and file paths to experiment (will be logged and accesible within the experiment)
artifacts = {
    "train_data": train_df,
    "test_data": test_df,
    "train_data_path": train_data_path
}<jupyter_output><empty_output><jupyter_text>Optional: Run single training<jupyter_code># Run experiment with default params and sync all metadata
exp.run_exp(train, params, artifacts)<jupyter_output><empty_output><jupyter_text>Run Simple Grid SearchSimple grid search using [Sklearn's ParamGrid](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.ParameterGrid.html). Please refer to this [user guide](https://scikit-learn.org/stable/modules/grid_search.htmlgrid-search) on how to define the parameter space.<jupyter_code>from lab_client.utils.experiment_optimizer import GridSearchOptimizer

# Parameter Space for Hyperparameter Optimization
param_space = {
    'epochs': [5, 10, 20, 50],
    'min_count': [2, 5]
}

# Initialize and run Optimizer
optimizer = GridSearchOptimizer(exp)
experiments = optimizer.run(train, params, artifacts, param_space)
# Visualize Results
optimizer.plot_results()<jupyter_output><empty_output><jupyter_text>Run Simple Random SearchSimple random search using [Sklearn's ParamSampler](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.ParameterSampler.html). Please refer to this [user guide](https://scikit-learn.org/stable/modules/grid_search.htmlrandomized-parameter-optimization) on how to define the parameter space.<jupyter_code>from lab_client.utils.experiment_optimizer import RandomSearchOptimizer

# Parameter Space for Hyperparameter Optimization
param_space = {
    'epochs': [5, 10, 20, 50],
    'min_count': [2, 5]
}

# Initialize and run Optimizer
optimizer = RandomSearchOptimizer(exp)
experiments = optimizer.run(train, params, artifacts, param_space, max_evals=5)
# Visualize Results
optimizer.plot_results()<jupyter_output><empty_output><jupyter_text>Run Hyperopt OptimizerOptimization based on the [hyperopt](http://hyperopt.github.io/hyperopt/) library using the Tree of Parzen Estimators (TPE) algorithm. Please refer to this [user guide](https://github.com/hyperopt/hyperopt/wiki/FMin) for information on how to define the parameter space and configure the optimizer.<jupyter_code>from lab_client.utils.experiment_optimizer import HyperoptOptimizer

# Parameter Space for Hyperparameter Optimization
from hyperopt import hp
param_space = {
    'epochs': hp.choice("epochs", [5, 10, 20, 50]),
    'min_count': hp.choice("min_count", [2,5])
}

# Initialize and run Optimizer
# You can provide additional named params to the HyperoptOptimizer that will be used in the fmin function
# Refer to this docu for information on available parameters: https://github.com/hyperopt/hyperopt/wiki/FMin
optimizer = HyperoptOptimizer(exp) 
experiments = optimizer.run(train, params, artifacts, param_space, max_evals=5)

# Visualize Results
optimizer.plot_results()<jupyter_output><empty_output><jupyter_text>Run Skopt OptimizerOptimization based on the [skopt](https://scikit-optimize.github.io/) library that provides several algorithms for sequential model-based optimization. Please refer to this [user guide](https://scikit-optimize.github.io/optimizer/index.html) for information on how to define the parameter space and configure the optimizer.<jupyter_code>from lab_client.utils.experiment_optimizer import SkoptOptimizer

# Parameter Space for Hyperparameter Optimization
# Information on param space configuration: https://scikit-optimize.github.io/notebooks/hyperparameter-optimization.html
from skopt.space import Real, Integer, Categorical
param_space  = [
    Categorical([5, 10, 20, 50], name='epochs'),
    Categorical([2,5], name='min_count')
]

# Initialize and run Optimizer
# You can provide additional named params to the SkoptOptimizer to configure the minimize func of the selecte algorithm
# Refer to this docu for information on available parameters and algos: https://scikit-optimize.github.io/optimizer/index.html
optimizer = SkoptOptimizer(exp) 
experiments = optimizer.run(train, params, artifacts, param_space, max_evals=10)

# Visualize Results
optimizer.plot_results()<jupyter_output><empty_output><jupyter_text>Run Optuna OptimizerOptimization based on the [optuna](https://github.com/pfnet/optuna) library that provides several algorithms parameter optimization. Please refer to this [user guide](https://optuna.readthedocs.io/en/stable/index.html) for information on how to define the parameter space and configure the optimizer.<jupyter_code>from lab_client.utils.experiment_optimizer import OptunaOptimizer

# Parameter Selection Function for Hyperparameter Optimization
# Information on param space configuration: https://optuna.readthedocs.io/en/stable/tutorial/configurations.html
def select_params(trial):
    param_selection = {}
    param_selection["epochs"] = trial.suggest_categorical('epochs', [5, 10, 20, 50])
    param_selection["min_count"] = trial.suggest_categorical('min_count', [2, 5])
    return param_selection

# Initialize and run Optimizer
# You can provide additional named params to the OptunaOptimizer to configure the optuna Study
# Refer to this docu for information on available parameters and algos: https://optuna.readthedocs.io/en/stable/index.html
optimizer = OptunaOptimizer(exp) 
experiments = optimizer.run(train, params, artifacts, select_params, max_evals=5)

# Visualize Results
optimizer.plot_results()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09415262636273539, 'fraction_numerical': 0.00810883863411118, 'mean_word_length': 5.142778085224128, 'pattern_counts': {'":': 3, '<': 45, '<?xml version=': 0, '>': 45, 'https://': 15, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '178364', 'license': 'Apache-2.0', 'n_tokens_mistral': 3372, 'n_tokens_neox': 3245, 'n_words': 906, 'path': 'services/lab-workspace/docker-res/tutorials/tutorials/hyperparameter-optimization-tutorial.ipynb', 'repo_name': 'Felipe-Renck/machine-learning-lab', 'size': '11099'}