<jupyter_start><jupyter_text>This notebook uses the Huggingface transformers library to finetune a transformer model.**Requirements.** This notebook has additional requirements:```bashpip install -r transformers_requirements.txt```  Tokenizer<jupyter_code>from transformers import AutoTokenizer
MODEL_CHECKPOINT = "distilbert-base-uncased"
tokenizer = AutoTokenizer.from_pretrained(MODEL_CHECKPOINT, use_fast=True)
tokenizer("this is a test")<jupyter_output><empty_output><jupyter_text>Data<jupyter_code>TASK = "cola"
import datasets
raw_dataset = datasets.load_dataset("glue", TASK)
# define tokenization function used to process data
COLUMN_NAME = "sentence"
def tokenize(examples):
    return tokenizer(examples[COLUMN_NAME], truncation=True)
encoded_dataset = raw_dataset.map(tokenize, batched=True)
encoded_dataset["train"][0]<jupyter_output><empty_output><jupyter_text>Model<jupyter_code>from transformers import AutoModelForSequenceClassification
NUM_LABELS = 2
model = AutoModelForSequenceClassification.from_pretrained(MODEL_CHECKPOINT, num_labels=NUM_LABELS)
model<jupyter_output><empty_output><jupyter_text>Metric<jupyter_code>metric = datasets.load_metric("glue", TASK)
metric
def compute_metrics(eval_pred):
    predictions, labels = eval_pred
    predictions = np.argmax(predictions, axis=1)
    return metric.compute(predictions=predictions, references=labels)<jupyter_output><empty_output><jupyter_text>Training (aka Finetuning)<jupyter_code>from transformers import Trainer
from transformers import TrainingArguments
args = TrainingArguments(
    output_dir='output',
    do_eval=True,
    )
trainer = Trainer(
    model=model,
    args=args,
    train_dataset=encoded_dataset["train"],
    eval_dataset=encoded_dataset["validation"],
    tokenizer=tokenizer,
    compute_metrics=compute_metrics,
)
trainer.train()<jupyter_output><empty_output><jupyter_text>Hyperparameter Optimization`flaml.tune` is a module for economical hyperparameter tuning. It frees users from manually tuning many hyperparameters for a software, such as machine learning training procedures. The API is compatible with ray tune. Step 1. Define training methodWe define a function `train_distilbert(config: dict)` that accepts a hyperparameter configuration dict `config`. The specific configs will be generated by flaml's search algorithm in a given search space.<jupyter_code>import flaml

def train_distilbert(config: dict):

    # Define tokenize method
    tokenizer = AutoTokenizer.from_pretrained(MODEL_CHECKPOINT, use_fast=True)
    def tokenize(examples):
        return tokenizer(examples[COLUMN_NAME], truncation=True)
    # Load CoLA dataset and apply tokenizer
    cola_raw = load_dataset("glue", TASK)
    cola_encoded = cola_raw.map(tokenize, batched=True)
    # QUESTION: Write processed data to disk?
    train_dataset, eval_dataset = cola_encoded["train"], cola_encoded["validation"]

    model = AutoModelForSequenceClassification.from_pretrained(
        MODEL_CHECKPOINT, num_labels=NUM_LABELS
    )

    metric = load_metric("glue", TASK)

    training_args = TrainingArguments(
        output_dir='.',
        do_eval=False,
        **config,
    )

    trainer = Trainer(
        model,
        training_args,
        train_dataset=train_dataset,
        eval_dataset=eval_dataset,
        tokenizer=tokenizer,
        compute_metrics=compute_metrics,
    )

    # train model
    trainer.train()

    # evaluate model
    eval_output = trainer.evaluate()

    # report the metric to optimize
    flaml.tune.report(
        loss=eval_output["eval_loss"],
        matthews_correlation=eval_output["eval_matthews_correlation"],
        )<jupyter_output><empty_output><jupyter_text>Step 2. Define the searchWe are now ready to define our search. This includes:- The `search_space` for our hyperparameters- The metric and the mode ('max' or 'min') for optimization- The constraints (`n_cpus`, `n_gpus`, `num_samples`, and `time_budget_s`)<jupyter_code>max_num_epoch = 4
search_space = {
        # You can mix constants with search space objects.
        "num_train_epochs": flaml.tune.loguniform(1, max_num_epoch),
        "learning_rate": flaml.tune.loguniform(1e-6, 1e-4),
        "adam_epsilon": flaml.tune.loguniform(1e-9, 1e-7),
    }
# optimization objective
HP_METRIC, MODE = "matthews_correlation", "max"

# resources
num_cpus = 2
num_gpus = 2

# constraints
num_samples = -1    # number of trials, -1 means unlimited
time_budget_s = 3600    # time budget in seconds<jupyter_output><empty_output><jupyter_text>Step 3. Launch with `flaml.tune.run`We are now ready to laungh the tuning using `flaml.tune.run`:<jupyter_code>import time
import ray
start_time = time.time()
ray.init(num_cpus=num_cpus, num_gpus=num_gpus)

print("Tuning started...")
analysis = flaml.tune.run(
    train_distilbert,
    config=search_space,
    init_config={
        "num_train_epochs": 1,
    },
    metric=HP_METRIC,
    mode=MODE,
    report_intermediate_result=False,
    # uncomment the following if report_intermediate_result = True
    # max_resource=max_num_epoch, min_resource=1,
    resources_per_trial={"gpu": 1},
    local_dir='logs/',
    num_samples=num_samples,
    time_budget_s=time_budget_s,
    use_ray=True,
)

ray.shutdown()
best_trial = analysis.get_best_trial(HP_METRIC, MODE, "all")
metric = best_trial.metric_analysis[HP_METRIC][MODE]
print(f"n_trials={len(analysis.trials)}")
print(f"time={time.time()-start_time}")
print(f"Best model eval {HP_METRIC}: {metric:.4f}")
print(f"Best model parameters: {best_trial.config}")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09727207465900933, 'fraction_numerical': 0.005025125628140704, 'mean_word_length': 5.354618015963512, 'pattern_counts': {'":': 5, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '110234', 'license': 'Apache-2.0', 'n_tokens_mistral': 1826, 'n_tokens_neox': 1781, 'n_words': 383, 'path': 'notebook/finetune_transformer_demo.ipynb', 'repo_name': 'dan0nchik/FLAML', 'size': '5572'}