<jupyter_start><jupyter_text>RoadMap 13 - Torch NN - Optimizers and handling Learning rate    To use torch.optim you have to construct an optimizer object, that will hold the current state and will update the parameters based on the computed gradients.    To construct an Optimizer you have to give it an iterable containing the parameters (all should be Variable s) to optimize. Then, you can specify optimizer-specific options such as the learning rate, weight decay, etc.         Contents        1. Adding parameters to optimizers    2. Taking an optimization step    3. Base optimizer class    4. Optimizers            4.1.  Adadelta            4.2.  Adagrad            4.3.  Adam            4.4.  Sparse Adam            4.5.  Adamax            4.6.  Averaged Storchastic Gradient Descent            4.7.  Limited Memeory Broyden–Fletcher–Goldfarb–Shanno            4.8.  RMSProp            4.9.  Resilient backpropagation    5. Adusting Learning rate            5.1.  Lambda-LR            5.2.  Step-LR            5.3.  Multistep-LR            5.4.  Exponential-LR            5.5.  Cosine Annealing LR            5.6.  Plateau-LR<jupyter_code>import os
import sys
import torch
import numpy as np
import torch.nn as nn
from torchvision import transforms, datasets
from PIL import Image
import cv2
import matplotlib.pyplot as plt
import torchvision<jupyter_output><empty_output><jupyter_text>Extra Blog Resources1. https://towardsdatascience.com/types-of-optimization-algorithms-used-in-neural-networks-and-ways-to-optimize-gradient-95ae5d39529f    2. https://algorithmia.com/blog/introduction-to-optimizers3. https://medium.com/datadriveninvestor/overview-of-different-optimizers-for-neural-networks-e0ed119440c34. https://www.deeplearningbook.org/contents/optimization.html<jupyter_code># Input
image_name = "dog.jpg"
image_pil = Image.open(image_name)

transform = transforms.Compose([transforms.ToTensor()])
image = transform(image_pil).float()
image_nchw = torch.unsqueeze(image, 0)
print(image_nchw.size())
plt.imshow(image_pil)<jupyter_output>torch.Size([1, 3, 576, 768])<jupyter_text>Adding parameters to optimizers    Optimizer s also support specifying per-parameter options.         To do this, instead of passing an iterable of Variable s, pass in an iterable of dicts.         Each of them will define a separate parameter group, and should contain a params key, containing a list of parameters belonging to it.         optim.SGD([                {'params': model.base.parameters()},                {'params': model.classifier.parameters(), 'lr': 1e-3}            ], lr=1e-2, momentum=0.9)  Taking an optimization Step    All optimizers implement a step() method, that updates the parameters.        It can be dones as follows:            optimizer.step()  Base Optimizer class and function listClass - torch.optim.OptimizerFunctions    1. add_param_group - Add a param group to the Optimizer's param_groups. This can be useful when fine tuning a pre-trained network as frozen layers can be made trainable and added to the Optimizer as training progresses.            - param_group (dict) – Specifies what Tensors should be optimized along with group                2. load_state_dict - Loads the optimizer state.            - state_dict (dict) – optimizer state. Should be an object returned from a call to state_dict()                3. state_dict - Returns the state of the optimizer as a dict.        4. step - Performs a single optimization step             - closure (callable) – A closure that reevaluates the model and returns the loss. Optional for most optimizers.                 5. zero_grad - Clears the gradients of all optimized torch.Tensor s.<jupyter_code># Creating a sample model

import torch.nn as nn
import torch.nn.functional as F

'''
input -> conv2d -> relu -> conv2d -> relu -> output
'''
class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
       x = F.relu(self.conv1(x))
       return F.relu(self.conv2(x))
model = Model()
# AdaDelta Optimizer

'''
1. torch.optim.Adadelta - Implements Adadelta algorithm.
        - params (iterable) – iterable of parameters to optimize or dicts defining parameter groups
        - rho (float, optional) – coefficient used for computing a running average of squared gradients (default: 0.9)
        - eps (float, optional) – term added to the denominator to improve numerical stability (default: 1e-6)
        - lr (float, optional) – coefficient that scale delta before it is applied to the parameters (default: 1.0)
        - weight_decay (float, optional) – weight decay (L2 penalty) (default: 0)
'''

adadelta_optim = torch.optim.Adadelta(model.parameters(), lr=1.0, rho=0.9, eps=1e-06, weight_decay=0)
# Adagrad Optimizers

'''
2. torch.optim.Adagrad - Implements Adagrad algorithm.
        - params (iterable) – iterable of parameters to optimize or dicts defining parameter groups
        - lr (float, optional) – learning rate (default: 1e-2)
        - lr_decay (float, optional) – learning rate decay (default: 0)
        - weight_decay (float, optional) – weight decay (L2 penalty) (default: 0)

'''

adagrad_optim = torch.optim.Adagrad(model.parameters(), lr=1.0, lr_decay=0.001, weight_decay=0.00005)
# Adam Optimizers

'''
3. torch.optim.Adam - Implements Adam algorithm.
        - params (iterable) – iterable of parameters to optimize or dicts defining parameter groups
        - lr (float, optional) – learning rate (default: 1e-3)
        - betas (Tuple[float, float], optional) – coefficients used for computing running averages of gradient and 
            its square (default: (0.9, 0.999))
        - eps (float, optional) – term added to the denominator to improve numerical stability (default: 1e-8)
        - weight_decay (float, optional) – weight decay (L2 penalty) (default: 0)
        - amsgrad (boolean, optional) – whether to use the AMSGrad variant of this algorithm from the paper On the Convergence of Adam and Beyond

'''

adam_optim = torch.optim.Adam(model.parameters(), lr=0.001, betas=(0.9, 0.999), eps=1e-08, weight_decay=0, amsgrad=False)
# Sparse Adam Optimizers

'''
4. torch.optim.SparseAdam - Implements lazy version of Adam algorithm suitable for sparse tensors.
        - params (iterable) – iterable of parameters to optimize or dicts defining parameter groups
        - lr (float, optional) – learning rate (default: 1e-3)
        - betas (Tuple[float, float], optional) – coefficients used for computing running averages of gradient 
            and its square (default: (0.9, 0.999))
        - eps (float, optional) – term added to the denominator to improve numerical stability (default: 1e-8)

'''

sparse_adam_optim = torch.optim.SparseAdam(model.parameters(), lr=0.001, betas=(0.9, 0.999), eps=1e-08)
# Adamax Optimizers

'''
5. torch.optim.Adamax - Implements Adamax algorithm (a variant of Adam based on infinity norm). It has been proposed 
    in Adam: A Method for Stochastic Optimization.
        - params (iterable) – iterable of parameters to optimize or dicts defining parameter groups
        - lr (float, optional) – learning rate (default: 2e-3)
        - betas (Tuple[float, float], optional) – coefficients used for computing running averages of gradient and its square
        - eps (float, optional) – term added to the denominator to improve numerical stability (default: 1e-8)
        - weight_decay (float, optional) – weight decay (L2 penalty) (default: 0)

'''

adamax_optim = torch.optim.Adamax(model.parameters(), lr=0.002, betas=(0.9, 0.999), eps=1e-08, weight_decay=0)
# Averaged Stochastic Gradient Descent Optimizers

'''
6. torch.optim.ASGD - Implements Averaged Stochastic Gradient Descent. It has been proposed 
    in Acceleration of stochastic approximation by averaging.
        - params (iterable) – iterable of parameters to optimize or dicts defining parameter groups
        - lr (float, optional) – learning rate (default: 1e-2)
        - lambd (float, optional) – decay term (default: 1e-4)
        - alpha (float, optional) – power for eta update (default: 0.75)
        - t0 (float, optional) – point at which to start averaging (default: 1e6)
        - weight_decay (float, optional) – weight decay (L2 penalty) (default: 0)

'''

asgd_optim = torch.optim.ASGD(model.parameters(), lr=0.01, lambd=0.0001, alpha=0.75, t0=1000000.0, weight_decay=0)
# Limited Memeory Broyden–Fletcher–Goldfarb–Shanno (BFGS) optimizers

'''
7. torch.optim.LBFGS - Implements Limited Memeory Broyden–Fletcher–Goldfarb–Shanno (BFGS) algorithm
        - lr (float) – learning rate (default: 1)
        - max_iter (int) – maximal number of iterations per optimization step (default: 20)
        - max_eval (int) – maximal number of function evaluations per optimization step (default: max_iter * 1.25).
        - tolerance_grad (float) – termination tolerance on first order optimality (default: 1e-5).
        - tolerance_change (float) – termination tolerance on function value/parameter changes (default: 1e-9).
        - history_size (int) – update history size (default: 100).

'''

lbgfs_optim = torch.optim.LBFGS(model.parameters(), lr=1, max_iter=20, max_eval=None, tolerance_grad=1e-05, \
                                tolerance_change=1e-09, history_size=100, line_search_fn=None)
# RMS-Prop optimizers

'''
8. torch.optim.RMSprop - Implements RMSprop algorithm.
        - params (iterable) – iterable of parameters to optimize or dicts defining parameter groups
        - lr (float, optional) – learning rate (default: 1e-2)
        - momentum (float, optional) – momentum factor (default: 0)
        - alpha (float, optional) – smoothing constant (default: 0.99)
        - eps (float, optional) – term added to the denominator to improve numerical stability (default: 1e-8)
        - centered (bool, optional) – if True, compute the centered RMSProp, the gradient is normalized by an 
            estimation of its variance
        - weight_decay (float, optional) – weight decay (L2 penalty) (default: 0)

'''

rmsprop_optim = torch.optim.RMSprop(model.parameters(), lr=0.01, alpha=0.99, eps=1e-08, weight_decay=0, momentum=0,\
                                    centered=False)
# Resilient backpropagation Optimizers

'''
9. torch.optim.Rprop - Implements the resilient backpropagation algorithm.
        - params (iterable) – iterable of parameters to optimize or dicts defining parameter groups
        - lr (float, optional) – learning rate (default: 1e-2)
        - etas (Tuple[float, float], optional) – pair of (etaminus, etaplis), that are multiplicative increase and 
            decrease factors (default: (0.5, 1.2))
        - step_sizes (Tuple[float, float], optional) – a pair of minimal and maximal allowed step sizes 
            (default: (1e-6, 50))

'''


rprop_optim = torch.optim.Rprop(model.parameters(), lr=0.01, etas=(0.5, 1.2), step_sizes=(1e-06, 50))
# Stochastic gradient descent Optimizers

'''
10. torch.optim.SGD - Implements stochastic gradient descent (optionally with momentum).
        - params (iterable) – iterable of parameters to optimize or dicts defining parameter groups
        - lr (float) – learning rate
        - momentum (float, optional) – momentum factor (default: 0)
        - weight_decay (float, optional) – weight decay (L2 penalty) (default: 0)
        - dampening (float, optional) – dampening for momentum (default: 0)
        - nesterov (bool, optional) – enables Nesterov momentum (default: False)

'''

sgd_optim = torch.optim.SGD(model.parameters(), lr=0.01, momentum=0, dampening=0, weight_decay=0, nesterov=False)<jupyter_output><empty_output><jupyter_text>Extra Blog Resources1. https://machinelearningmastery.com/learning-rate-for-deep-learning-neural-networks/2. https://towardsdatascience.com/understanding-learning-rates-and-how-it-improves-performance-in-deep-learning-d0d4059c1c103. https://www.jeremyjordan.me/nn-learning-rate/4. https://www.deeplearningwizard.com/deep_learning/boosting_models_pytorch/lr_scheduling/5. https://www.pyimagesearch.com/2019/07/22/keras-learning-rate-schedules-and-decay/  Adjusting Learning Rate 1. torch.optim.lr_scheduler provides several methods to adjust the learning rate based on the number of epochs.2. torch.optim.lr_scheduler.ReduceLROnPlateau allows dynamic learning rate reducing based on some validation measurements.<jupyter_code># Lambda-LR based learning rate adjustments

'''
1. torch.optim.lr_scheduler.LambdaLR - Sets the learning rate of each parameter group to the initial lr 
    times a given function. When last_epoch=-1, sets initial lr as lr.
        - optimizer (Optimizer) – Wrapped optimizer.
        - lr_lambda (function or list) – A function which computes a multiplicative factor given an integer 
            parameter epoch, or a list of such functions, one for each group in optimizer.param_groups.
        - last_epoch (int) – The index of last epoch. Default: -1.

'''

lambda1 = lambda epoch: epoch // 30
lambda2 = lambda epoch: 0.95 ** epoch
scheduler = torch.optim.lr_scheduler.LambdaLR(sgd_optim, lr_lambda=lambda1)
# Step-LR based learning rate adjustments

'''
2. torch.optim.lr_scheduler.StepLR - Sets the learning rate of each parameter group to the initial 
    lr decayed by gamma every step_size epochs. When last_epoch=-1, sets initial lr as lr.
        - optimizer (Optimizer) – Wrapped optimizer.
        - step_size (int) – Period of learning rate decay.
        - gamma (float) – Multiplicative factor of learning rate decay. Default: 0.1.
        - last_epoch (int) – The index of last epoch. Default: -1.

'''

scheduler = torch.optim.lr_scheduler.StepLR(sgd_optim, step_size=30, gamma=0.1)
# Multi-Step-LR based learning rate adjustments

'''
3. torch.optim.lr_scheduler.MultiStepLR - Set the learning rate of each parameter group to the initial lr 
    decayed by gamma once the number of epoch reaches one of the milestones. 
    When last_epoch=-1, sets initial lr as lr.
        - optimizer (Optimizer) – Wrapped optimizer.
        - milestones (list) – List of epoch indices. Must be increasing.
        - gamma (float) – Multiplicative factor of learning rate decay. Default: 0.1.
        - last_epoch (int) – The index of last epoch. Default: -1.

'''

scheduler = torch.optim.lr_scheduler.MultiStepLR(sgd_optim, milestones=[30,80], gamma=0.1)
# Exponential-LR based learning rate adjustment

'''
4. torch.optim.lr_scheduler.ExponentialLR - Set the learning rate of each parameter group to the initial lr 
    decayed by gamma every epoch. When last_epoch=-1, sets initial lr as lr.
        - optimizer (Optimizer) – Wrapped optimizer.
        - gamma (float) – Multiplicative factor of learning rate decay.
        - last_epoch (int) – The index of last epoch. Default: -1.

'''

scheduler = torch.optim.lr_scheduler.ExponentialLR(sgd_optim, gamma=0.1, last_epoch=-1)
# Cosine-Annealing-LR based learning rate adjustment

'''
5. torch.optim.lr_scheduler.CosineAnnealingLR
        - optimizer (Optimizer) – Wrapped optimizer.
        - T_max (int) – Maximum number of iterations.
        - eta_min (float) – Minimum learning rate. Default: 0.
        - last_epoch (int) – The index of last epoch. Default: -1.

'''

scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(sgd_optim, T_max=200, eta_min=0, last_epoch=-1)
# Reduce Learning rate on Plateau based learning rate adustment

'''
6. torch.optim.lr_scheduler.ReduceLROnPlateau - Reduce learning rate when a metric has stopped improving. 
    Models often benefit from reducing the learning rate by a factor of 2-10 once learning stagnates. 
    This scheduler reads a metrics quantity and if no improvement is seen for a ‘patience’ number of epochs, 
    the learning rate is reduced.
        - optimizer (Optimizer) – Wrapped optimizer.
        - mode (str) – One of min, max. In min mode, lr will be reduced when the quantity monitored has stopped decreasing;
            in max mode it will be reduced when the quantity monitored has stopped increasing. Default: ‘min’.
        - factor (float) – Factor by which the learning rate will be reduced. new_lr = lr * factor. Default: 0.1.
        - patience (int) – Number of epochs with no improvement after which learning rate will be reduced. 
            For example, if patience = 2, then we will ignore the first 2 epochs with no improvement,
            and will only decrease the LR after the 3rd epoch if the loss still hasn’t improved then. Default: 10.
        - verbose (bool) – If True, prints a message to stdout for each update. Default: False.
        - threshold (float) – Threshold for measuring the new optimum, to only focus on significant changes. 
            Default: 1e-4.
        - threshold_mode (str) – One of rel, abs. In rel mode, 
            dynamic_threshold = best * ( 1 + threshold ) in ‘max’ mode or best * ( 1 - threshold ) in min mode. 
            In abs mode, dynamic_threshold = best + threshold in max mode or best - threshold in min mode. 
            Default: ‘rel’.
        - cooldown (int) – Number of epochs to wait before resuming normal operation after lr has been reduced. 
            Default: 0.
        - min_lr (float or list) – A scalar or a list of scalars. A lower bound on the learning rate of all 
            param groups or each group respectively. Default: 0.
        - eps (float) – Minimal decay applied to lr. If the difference between new and old lr is smaller than eps, 
            the update is ignored. Default: 1e-8.

'''

scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(sgd_optim, mode='min', factor=0.1, patience=10, \
                                                       verbose=False, threshold=0.0001, threshold_mode='rel', \
                                                       cooldown=0, min_lr=0, eps=1e-08)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09254915102770331, 'fraction_numerical': 0.023458445040214475, 'mean_word_length': 3.1104224058769514, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 21, 'https://': 9, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '329201', 'license': 'Apache-2.0', 'n_tokens_mistral': 5518, 'n_tokens_neox': 5171, 'n_words': 1957, 'path': 'M) RoadMap 13 - Torch NN 7 - Optimizers and learning rate adjustment.ipynb', 'repo_name': 'Charlottecuc/Pytorch_Tutorial', 'size': '17904'}