<jupyter_start><jupyter_text>Frictionless hyperparameter tuning with W&B and Jupyter Notebooks> How use W&B sweeps ro run the experiment in a notebook- toc: true - badges: false- comments: true- author: <NAME>- categories: [wandb, jupyter]- image: images/wandb_jupyter.png<jupyter_code># hide
# ## NOTE: UNCOMMENT AND RUN THIS CELL IF YOU NEED TO INSTALL/ UPGRADE TSAI
stable = False # True: stable version in pip, False: latest version from github
if stable: 
    !pip install tsai -U >> /dev/null
else:      
    !pip install git+https://github.com/timeseriesAI/tsai.git -U >> /dev/null
# ## NOTE: REMEMBER TO RESTART (NOT RECONNECT/ RESET) THE KERNEL/ RUNTIME ONCE THE INSTALLATION IS FINISHED
#hide
from tsai.all import *
from fastai.callback.wandb import WandbCallback<jupyter_output><empty_output><jupyter_text>IntroductionI often find myself coding a machine learning experiment in a Jupyter Notebook, and at the same time, using [Weights & Biases (wandb)](https://www.wandb.com/) to visualize and track the results of the runs. When the experiment is finished, I always have questions such as: How will the performance be affected by the parameter a? What if I change the number of items of the dataset, or change the dataset completely?Hyperpameter tuning with [wandb sweeps](https://docs.wandb.com/sweeps) is a great tool to solve these questions. However, sweeping requires that you define a specific training program, as a separate python file. I find this to be redundant, specially when the code for training is already in the Jupyter Notebook. Furthermore, if I make some changes in the original notebook, I have to be sure that I change the sweep script too.This post shows a trick to execute a Jupyter Notebook as the program of a wandb sweep. This provides a frictionless way of using your Jupyter Notebooks both for single runs and sweep functions. We won't use any separate configuration or script file, everything will be done between Jupyter and wandb. This post assumes that the reader has basic knowledge on both how Jupyter Notebooks and wandb sweeps work.As use case we will perform a time series classification task with deep neural networks using the wonderful library [tsai](https://github.com/timeseriesAI/tsai). This is all the code needed to train a classifier in `tsai` for the dataset [NATOPS](http://www.timeseriesclassification.com/description.php?Dataset=NATOPS):<jupyter_code>#hide_output
from tsai.all import *

dsid = 'NATOPS' 
X, y, splits = get_UCR_data(dsid, return_split=False)
learn = TSClassifier(X, y, splits=splits, bs=[64, 128], 
                     batch_tfms=[TSStandardize()], arch=InceptionTime, 
                     metrics=accuracy)
learn.fit_one_cycle(25, lr_max=1e-3)
learn.plot_metrics()<jupyter_output><empty_output><jupyter_text>We will do a hyperparameter search over the two arguments of the call to `fit_one_cycle`, that is, the number of epocs (`n_epochs`) and the maximum learning rate passed to the one-cycle schedule (`lr_max`). In the next section, we'll see how to organize the notebook so that it is ready to be used as the program of a sweep. Then, we'll configure it to be run in a local server (e.g, an instance of JupyterLab). Finally, for Colab users, we'll see a workaround to make this work with a subtle difference.  Refactoring the notebook for wandb sweeps As explained in the [wandb documentation](https://docs.wandb.ai/guides/sweeps/python-api): "two components work together in a sweep: a controller on the central sweep server, which picks out new hyperparameter combinations to try, and **agents**, running in any number of processes on any number of machines, which query the server for hyperparameters, use them to run model training, and then report the results back to the controller."Each time the agent queries values of the hyperparameters for a new *trial*, those will be *injected* as part of the configuration of the wandb run that the training program must have. Once the program executes the call to `wandb.init` to begin the syncing, the object `wandb.config` will contain them, and any line of code that depends on that config will use the values pof that trial.But, what happens if we had already defined a configuration object like the one below, before the call to `wandb.init` to play manually with different values?<jupyter_code>config = {
    'n_epochs' : 25,
    'lr_max' : 1e-3,
    'bs' : 64
}
#hide_output
import wandb
run = wandb.init(config=config, mode='disabled')<jupyter_output><empty_output><jupyter_text>The good thing is that, nothing happens! Even if we have set values for `n_epochs` and `lr_max` before calling `wandb.init`, the sweep agent will override them with the values of the new trial. This is a [common question](https://docs.wandb.ai/guides/sweeps/faq) explained in the wandb docs, and it is exactly what allows us to use the same exact notebook for both single runs and sweeps. Parameters that are not part of the sweep, such as `bs` in this example, can be part of the config object as well and of course they will be kept there by the sweep agent.The last thing we have to do is, as in every sweep, replace our magic numbers (at least the ones that are part of the sweep) with the corresponding reference to the config variable:<jupyter_code>#hide_output
from tsai.all import *

dsid = 'NATOPS' 
X, y, splits = get_UCR_data(dsid, return_split=False)
learn = TSClassifier(X, y, splits=splits, bs=[config['bs'], 128], 
                     batch_tfms=[TSStandardize()], arch=InceptionTime, 
                     metrics=accuracy, cbs=[WandbCallback(log_preds=False)])
learn.fit_one_cycle(config['n_epochs'], lr_max=config['lr_max'])
learn.plot_metrics()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.072778166550035, 'fraction_numerical': 0.0031490552834149755, 'mean_word_length': 4.671626984126984, 'pattern_counts': {'":': 0, '<': 18, '<?xml version=': 0, '>': 23, 'https://': 6, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '185641', 'license': 'Apache-2.0', 'n_tokens_mistral': 1667, 'n_tokens_neox': 1607, 'n_words': 765, 'path': '_notebooks/2022-01-03-sweeps.ipynb', 'repo_name': 'vrodriguezf/ml-totd', 'size': '5736'}