<jupyter_start><jupyter_text>Advanced Tutorial 5: SchedulerIn this tutorial, we will talk about:* [Scheduler](ta05scheduler)    * [Concept](ta05concept)    * [EpochScheduler](ta05epoch)    * [RepeatScheduler](ta05repeat)* [Things You Can Schedule](ta05things)    * [Datasets](ta05dataset)    * [Batch Size](ta05batch)    * [NumpyOps](ta05numpy)    * [Optimizers](ta05optimizer)    * [TensorOps](ta05tensor)    * [Traces](ta05trace)* [Related Apphub Examples](ta05apphub)   Scheduler   ConceptDeep learning training is getting more complicated every year. One major aspect of this complexity is time-dependent training. For example:* Using different datasets for different training epochs.* Applying different preprocessing for different epochs.* Training different networks on different epochs. * ...The list goes on and on. In order to provide an easy way for users to accomplish time-dependent training, we provide the `Scheduler` class which can help you schedule any part of the training. Please note that the basic time unit that `Scheduler` can handle is `epochs`. If users want arbitrary scheduling cycles, the simplest way is to customize the length of one epoch in `Estimator` using max_train_steps_per_epoch.   EpochSchedulerThe most straightforward way to schedule things is through an epoch-value mapping. For example, If users want to schedule the batch size in the following way:* epoch 1 - batchsize 16* epoch 2 - batchsize 32* epoch 3 - batchsize 32* epoch 4 - batchsize 64* epoch 5 - batchsize 64You can do the following:<jupyter_code>from fastestimator.schedule import EpochScheduler
batch_size = EpochScheduler(epoch_dict={1:16, 2:32, 4:64})
for epoch in range(1, 6):
    print("At epoch {}, batch size is {}".format(epoch, batch_size.get_current_value(epoch)))<jupyter_output>At epoch 1, batch size is 16
At epoch 2, batch size is 32
At epoch 3, batch size is 32
At epoch 4, batch size is 64
At epoch 5, batch size is 64<jupyter_text>RepeatSchedulerIf your schedule follows a repeating pattern, then you don't want to specify that for all epochs. `RepeatScheduler` is here to help you. Let's say we want the batch size on odd epochs to be 32, and on even epochs to be 64:<jupyter_code>from fastestimator.schedule import RepeatScheduler
batch_size = RepeatScheduler(repeat_list=[32, 64])

for epoch in range(1, 6):
    print("At epoch {}, batch size is {}".format(epoch, batch_size.get_current_value(epoch)))<jupyter_output>At epoch 1, batch size is 32
At epoch 2, batch size is 64
At epoch 3, batch size is 32
At epoch 4, batch size is 64
At epoch 5, batch size is 32<jupyter_text>Things You Can Schedule:   DatasetsScheduling training or evaluation datasets is very common in deep learning. For example, in curriculum learning people will train on an easy dataset first and then gradually move on to harder datasets. For illustration purposes, let's use two different instances of the same MNIST dataset:<jupyter_code>from fastestimator.dataset.data import mnist, cifar10
from fastestimator.schedule import EpochScheduler

train_data1, eval_data = mnist.load_data()
train_data2, _ = mnist.load_data()
train_data = EpochScheduler(epoch_dict={1:train_data1, 3: train_data2})<jupyter_output><empty_output><jupyter_text>Batch SizeWe can also schedule the batch size on different epochs, which may help resolve GPU resource constraints.<jupyter_code>batch_size = RepeatScheduler(repeat_list=[32,64])<jupyter_output><empty_output><jupyter_text>NumpyOpsPreprocessing operators can also be scheduled. For illustration purpose, we will apply a `Rotation` for the first two epochs and then not apply it for the third epoch:<jupyter_code>from fastestimator.op.numpyop.univariate import ExpandDims, Minmax
from fastestimator.op.numpyop.multivariate import Rotate
import fastestimator as fe

rotate_op = EpochScheduler(epoch_dict={1:Rotate(image_in="x", image_out="x",limit=30), 3:None})

pipeline = fe.Pipeline(train_data=train_data, 
                       eval_data=eval_data,
                       batch_size=batch_size, 
                       ops=[ExpandDims(inputs="x", outputs="x"), rotate_op, Minmax(inputs="x", outputs="x")])<jupyter_output><empty_output><jupyter_text>OptimizersFor fast convergence, some people like to use different optimizers at different training phases. In our example, we will use `adam` for the first epoch and `sgd` for the second epoch.<jupyter_code>from fastestimator.architecture.tensorflow import LeNet

model_1 = fe.build(model_fn=LeNet, optimizer_fn=EpochScheduler(epoch_dict={1:"adam", 2: "sgd"}), model_name="m1")<jupyter_output><empty_output><jupyter_text>TensorOpsWe can schedule `TensorOps` just like `NumpyOps`. Let's define another model `model_2` such that:* epoch 1-2: train `model_1`* epoch 3: train `model_2`<jupyter_code>from fastestimator.op.tensorop.model import ModelOp, UpdateOp
from fastestimator.op.tensorop.loss import CrossEntropy

model_2 = fe.build(model_fn=LeNet, optimizer_fn="adam", model_name="m2")

model_map = {1: ModelOp(model=model_1, inputs="x", outputs="y_pred"), 
             3: ModelOp(model=model_2, inputs="x", outputs="y_pred")}

update_map = {1: UpdateOp(model=model_1, loss_name="ce"), 3: UpdateOp(model=model_2, loss_name="ce")}

network = fe.Network(ops=[EpochScheduler(model_map),
                          CrossEntropy(inputs=("y_pred", "y"), outputs="ce"),
                          EpochScheduler(update_map)])<jupyter_output><empty_output><jupyter_text>Traces`Traces` can also be scheduled. For example, we will save `model_1` at the end of second epoch and save `model_3` at the end of third epoch:<jupyter_code>from fastestimator.trace.io import ModelSaver
import tempfile

save_folder = tempfile.mkdtemp()

#Disable model saving by setting None on 3rd epoch:
modelsaver1 = EpochScheduler({2:ModelSaver(model=model_1,save_dir=save_folder), 3:None})

modelsaver2 = EpochScheduler({3:ModelSaver(model=model_2,save_dir=save_folder)})

traces=[modelsaver1, modelsaver2]<jupyter_output><empty_output><jupyter_text>Let the training beginNothing special in here, create the estimator then start the training:<jupyter_code>estimator = fe.Estimator(pipeline=pipeline, network=network, traces=traces, epochs=3, log_steps=300)
estimator.fit()<jupyter_output>______           __  ______     __  _                 __            
   / ____/___ ______/ /_/ ____/____/ /_(_)___ ___  ____ _/ /_____  _____
  / /_  / __ `/ ___/ __/ __/ / ___/ __/ / __ `__ \/ __ `/ __/ __ \/ ___/
 / __/ / /_/ (__  ) /_/ /___(__  ) /_/ / / / / / / /_/ / /_/ /_/ / /    
/_/    \__,_/____/\__/_____/____/\__/_/_/ /_/ /_/\__,_/\__/\____/_/     
                                                                        

FastEstimator-Start: step: 1; num_device: 1; logging_interval: 300; 
FastEstimator-Train: step: 1; ce: 0.016785031; 
FastEstimator-Train: step: 300; ce: 0.16430134; steps/sec: 697.26; 
FastEstimator-Train: step: 600; ce: 0.023913195; steps/sec: 728.45; 
FastEstimator-Train: step: 900; ce: 0.042380013; steps/sec: 732.24; 
FastEstimator-Train: step: 1200; ce: 0.0014684915; steps/sec: 723.88; 
FastEstimator-Train: step: 1500; ce: 0.020901386; steps/sec: 728.1; 
FastEstimator-Train: step: 1800; ce: 0.0114256; steps/sec: 724.26; 
FastEstimator-Train: step: 187[...]
{'fraction_non_alphanumeric': 0.10266317096729681, 'fraction_numerical': 0.03725679591555126, 'mean_word_length': 4.575384615384616, 'pattern_counts': {'":': 0, '<': 34, '<?xml version=': 0, '>': 34, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '338893', 'license': 'Apache-2.0', 'n_tokens_mistral': 2512, 'n_tokens_neox': 2276, 'n_words': 686, 'path': 'tutorial/advanced/t05_scheduler.ipynb', 'repo_name': 'purujitb/fastestimator', 'size': '7247'}