<jupyter_start><jupyter_text>[![Github](https://img.shields.io/github/stars/labmlai/annotated_deep_learning_paper_implementations?style=social)](https://github.com/labmlai/annotated_deep_learning_paper_implementations)[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/transformers/switch/experiment.ipynb)                     Switch TransformerThis is an experiment training Shakespeare dataset with a small Switch Transformer. Install the `labml-nn` package<jupyter_code>!pip install labml-nn<jupyter_output>Collecting labml-nn
[?25l  Downloading https://files.pythonhosted.org/packages/fa/de/b8bea1493162cc4a845d043a84bce937f68751b9a463b2318672e0a46fed/labml_nn-0.4.79-py3-none-any.whl (117kB)

[K     |██▉                             | 10kB 19.5MB/s eta 0:00:01
[K     |█████▋                          | 20kB 13.7MB/s eta 0:00:01
[K     |████████▍                       | 30kB 9.9MB/s eta 0:00:01
[K     |███████████▏                    | 40kB 8.5MB/s eta 0:00:01
[K     |██████████████                  | 51kB 5.5MB/s eta 0:00:01
[K     |████████████████▊               | 61kB 5.8MB/s eta 0:00:01
[K     |███████████████████▌            | 71kB 6.2MB/s eta 0:00:01
[K     |██████████████████████▎         | 81kB 6.6MB/s eta 0:00:01
[K     |█████████████████████████       | 92kB 6.2MB/s eta 0:00:01
[K     |███████████████████████████▉    | 102kB 5.3MB/s eta 0:00:01
[K     |██████████████████████████████▋ | 112kB 5.3MB/s eta 0:00:01
[K     |████████████████████████████████| 122kB 5.3MB/s 
[...]<jupyter_text>Imports<jupyter_code>import torch
import torch.nn as nn

from labml import experiment
from labml.configs import option
from labml_helpers.module import Module
from labml_nn.transformers.switch.experiment import Configs<jupyter_output><empty_output><jupyter_text>Create an experiment<jupyter_code>experiment.create(name="switch_transformer")<jupyter_output><empty_output><jupyter_text>Initialize configurations<jupyter_code>conf = Configs()<jupyter_output><empty_output><jupyter_text>Set experiment configurations and assign a configurations dictionary to override configurations<jupyter_code>experiment.configs(conf,
                # A dictionary of configurations to override
                {'tokenizer': 'character',
                'text': 'tiny_shakespeare',
                'optimizer.learning_rate': 1.,
                'optimizer.optimizer': 'Noam',
                'prompt': 'It is',
                'prompt_separator': '',

                'transformer': 'switch_transformer',
                'is_scale_prob': False,
                'n_experts': 4,

                'drop_tokens': True,
                'capacity_factor': 1.2,

                'train_loader': 'shuffled_train_loader',
                'valid_loader': 'shuffled_valid_loader',

                'seq_len': 64,
                'epochs': 128,
                'batch_size': 32,
                'inner_iterations': 25,
                })<jupyter_output><empty_output><jupyter_text>Set PyTorch models for loading and saving<jupyter_code>experiment.add_pytorch_models({'model': conf.model})<jupyter_output><empty_output><jupyter_text>Start the experiment and run the training loop.<jupyter_code># Start the experiment
with experiment.start():
    conf.run()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.17332944266121972, 'fraction_numerical': 0.04960606944849723, 'mean_word_length': 3.0520094562647753, 'pattern_counts': {'":': 0, '<': 28, '<?xml version=': 0, '>': 28, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '248894', 'license': 'MIT', 'n_tokens_mistral': 1466, 'n_tokens_neox': 1188, 'n_words': 182, 'path': 'labml_nn/transformers/switch/experiment.ipynb', 'repo_name': 'techthiyanes/annotated_deep_learning_paper_implementations', 'size': '3427'}