<jupyter_start><jupyter_text>Copyright (c) 2020-2021. All rights reserved.Licensed under the MIT License. Troubleshooting HPO for fine-tuning pre-trained language models 1. IntroductionIn this notebook, we demonstrate a procedure for troubleshooting HPO failure in fine-tuning pre-trained language models (introduced in the following paper):*[An Empirical Study on Hyperparameter Optimization for Fine-Tuning Pre-trained Language Models](https://arxiv.org/abs/2106.09204). <NAME>, <NAME>. To appear in ACL-IJCNLP 2021*Notes:*In this notebook, we only run each experiment 1 time for simplicity, which is different from the paper (3 times). To reproduce the paper's result, please run 3 repetitions and take the average scores.*Running this notebook takes about one hour.FLAML requires `Python>=3.6`. To run this notebook example, please install flaml with the `notebook` and `nlp` options:```bashpip install flaml[nlp]```Our paper was developed under transformers version 3.4.0. We uninstall and reinstall transformers==3.4.0:<jupyter_code>!pip install flaml[nlp]
!pip install transformers==3.4.0
from flaml.nlp import AutoTransformers<jupyter_output><empty_output><jupyter_text>2. Initial Experimental Study  Load dataset Load the dataset using AutoTransformer.prepare_data. In this notebook, we use the Microsoft Research Paraphrasing Corpus (MRPC) dataset and the Electra model as an example:<jupyter_code>autohf = AutoTransformers()
preparedata_setting = {
                          "dataset_subdataset_name": "glue:mrpc",
                          "pretrained_model_size": "google/electra-base-discriminator:base",
                          "data_root_path": "data/",
                          "max_seq_length": 128,
                       }
autohf.prepare_data(**preparedata_setting)<jupyter_output>console_args has no attribute pretrained_model_size, continue
console_args has no attribute dataset_subdataset_name, continue
console_args has no attribute algo_mode, continue
console_args has no attribute space_mode, continue
console_args has no attribute search_alg_args_mode, continue
console_args has no attribute algo_name, continue
console_args has no attribute pruner, continue
console_args has no attribute resplit_mode, continue
console_args has no attribute rep_id, continue
console_args has no attribute seed_data, continue
console_args has no attribute seed_transformers, continue
console_args has no attribute learning_rate, continue
console_args has no attribute weight_decay, continue<jupyter_text>Running grid searchFirst, we run grid search using Electra. By specifying `algo_mode="grid"`, AutoTransformers will run the grid search algorithm. By specifying `space_mode="grid"`, AutoTransformers will use the default grid search configuration recommended by the Electra paper:<jupyter_code>import transformers
autohf_settings = {
                      "resources_per_trial": {"gpu": 1, "cpu": 1},
                      "num_samples": 1,
                      "time_budget": 100000,  # unlimited time budget
                      "ckpt_per_epoch": 5,
                      "fp16": True,
                      "algo_mode": "grid",  # set the search algorithm to grid search
                      "space_mode": "grid", # set the search space to the recommended grid space
                      "transformers_verbose": transformers.logging.ERROR
                   }
validation_metric, analysis = autohf.fit(**autohf_settings)<jupyter_output><empty_output><jupyter_text>Get the time for running grid search:<jupyter_code>GST = autohf.last_run_duration
print("grid search for {} took {} seconds".format(autohf.jobid_config.get_jobid_full_data_name(), GST))<jupyter_output>grid search for glue_mrpc took 106.57789206504822 seconds<jupyter_text>After the HPO run finishes, generate the predictions and save it as a .zip file to be submitted to the glue website. Here we will need the library AzureUtils which is for storing the output information (e.g., analysis log, .zip file) locally and uploading the output to an azure blob container (e.g., if multiple jobs are executed in a cluster). If the azure key and container information is not specified, the output information will only be saved locally.<jupyter_code>predictions, test_metric = autohf.predict()
from flaml.nlp import AzureUtils

print(autohf.jobid_config)

azure_utils = AzureUtils(root_log_path="logs_test/", autohf=autohf)
azure_utils.write_autohf_output(valid_metric=validation_metric,
                                predictions=predictions,
                                duration=GST)
print(validation_metric)<jupyter_output>remove_columns_ is deprecated and will be removed in the next major version of datasets. Use the dataset.remove_columns method instead.<jupyter_text>The validation F1/accuracy we got was 92.4/89.5. After the above steps, you will find a .zip file for the predictions under data/result/. Submit the .zip file to the glue website. The test F1/accuracy we got was 90.4/86.7. As an example, we only run the experiment one time, but in general, we should run the experiment multiple repetitions and report the averaged validation and test accuracy.  Running Random SearchNext, we run random search with the same time budget as grid search:<jupyter_code>def tune_hpo(time_budget, this_hpo_space):
    autohf_settings = {
                          "resources_per_trial": {"gpu": 1, "cpu": 1},
                          "num_samples": -1,
                          "time_budget": time_budget,
                          "ckpt_per_epoch": 5,
                          "fp16": True,
                          "algo_mode": "hpo",  # set the search algorithm mode to hpo
                          "algo_name": "rs",
                          "space_mode": "cus", # customized search space (this_hpo_space)
                          "hpo_space": this_hpo_space,
                          "transformers_verbose": transformers.logging.ERROR
                       }
    validation_metric, analysis = autohf.fit(**autohf_settings)
    predictions, test_metric = autohf.predict()
    azure_utils = AzureUtils(root_log_path="logs_test/", autohf=autohf)
    azure_utils.write_autohf_output(valid_metric=validation_metric,
                                    predictions=predictions,
                                    duration=GST)
    print(validation_metric)
hpo_space_full = {
                      "learning_rate": {"l": 3e-5, "u": 1.5e-4, "space": "log"},
                      "warmup_ratio": {"l": 0, "u": 0.2, "space": "linear"},
                      "num_train_epochs": [3],
                      "per_device_train_batch_size": [16, 32, 64],
                      "weight_decay": {"l": 0.0, "u": 0.3, "space": "linear"},
                      "attention_probs_dropout_prob": {"l": 0, "u": 0.2, "space": "linear"},
                      "hidden_dropout_prob": {"l": 0, "u": 0.2, "space": "linear"},
                  }

tune_hpo(GST, hpo_space_full)<jupyter_output><empty_output><jupyter_text>The validation F1/accuracy we got was 93.5/90.9. Similarly, we can submit the .zip file to the glue website. The test F1/accuaracy we got was 81.6/70.2.   3. Troubleshooting HPO FailuresSince the validation accuracy is larger than grid search while the test accuracy is smaller, HPO has overfitting. We reduce the search space:<jupyter_code>hpo_space_fixwr = {
                       "learning_rate": {"l": 3e-5, "u": 1.5e-4, "space": "log"},
                       "warmup_ratio": [0.1],
                       "num_train_epochs": [3],
                       "per_device_train_batch_size": [16, 32, 64],
                       "weight_decay": {"l": 0.0, "u": 0.3, "space": "linear"},
                       "attention_probs_dropout_prob": {"l": 0, "u": 0.2, "space": "linear"},
                       "hidden_dropout_prob": {"l": 0, "u": 0.2, "space": "linear"},
                   }
tune_hpo(GST, hpo_space_fixwr)<jupyter_output><empty_output><jupyter_text>The validation F1/accuracy we got was 92.6/89.7, the test F1/accuracy was 85.9/78.7, therefore overfitting still exists and we further reduce the space:<jupyter_code>hpo_space_min = {
                   "learning_rate": {"l": 3e-5, "u": 1.5e-4, "space": "log"},
                   "warmup_ratio": [0.1],
                   "num_train_epochs": [3],
                   "per_device_train_batch_size": [16, 32, 64],
                   "weight_decay": [0.0],
                   "attention_probs_dropout_prob": [0.1],
                   "hidden_dropout_prob": [0.1],
                }
tune_hpo(GST, hpo_space_min)<jupyter_output><empty_output><jupyter_text>The validation F1/accuracy we got was 90.4/86.7, test F1/accuracy was 83.0/73.0. Since the validation accuracy is below grid search, we increase the budget to 4 * GST:<jupyter_code>hpo_space_min = {
                   "learning_rate": {"l": 3e-5, "u": 1.5e-4, "space": "log"},
                   "warmup_ratio": [0.1],
                   "num_train_epochs": [3],
                   "per_device_train_batch_size": [32],
                   "weight_decay": [0.0],
                   "attention_probs_dropout_prob": [0.1],
                   "hidden_dropout_prob": [0.1],
                 }
tune_hpo(4 * GST, hpo_space_min)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09651967142239516, 'fraction_numerical': 0.023670557717250323, 'mean_word_length': 2.8156701030927835, 'pattern_counts': {'":': 87, '<': 36, '<?xml version=': 0, '>': 37, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '21629', 'license': 'MIT', 'n_tokens_mistral': 2873, 'n_tokens_neox': 2659, 'n_words': 808, 'path': 'notebook/research/acl2021.ipynb', 'repo_name': 'Jimmy-INL/FLAML', 'size': '9259'}