<jupyter_start><jupyter_text>"[Draft] Optimizer Code Supplement Using fastai"> "Building the Optimizers explained by <NAME> using fastai"- toc: true- branch: master- badges: true- comments: true- author: <NAME>- categories: [fastai]  Introduction This post is meant to be a supporting guide that can be used with the incredibly detailed [post from <NAME>](https://ruder.io/optimizing-gradient-descent/index.html).  The goal is to implement each of these optimizers using fastai and run them on a small problem.    The sample problem that will be used is the one created by Jeremy and Rachel in the graddesc spreadsheet.  Because this post is generated using Jupyter Notebooks, you can also interact with any of these concepts and modify them as you see fit.  In as many places as possible, I will try to connect the code back to the concepts by utilizing the greek letters Sebastian (and usually the paper) uses<jupyter_code>#hide
from fastai.vision.all import *
a_real = 2
b_real = 30<jupyter_output><empty_output><jupyter_text>`a_real` is a weight and `b_real` is a bias they form the equation $a\_real*x + b\_real= y$  Both a and b are parameters and will be learned by our optimizer through the loss function.  In a real world problem, these values would be unknown and there would typically be a lot more weights, but the concept will remain the same whether there are 2 parameters or [175 billion](https://www.google.com/search?client=firefox-b-1-d&q=how+many+parameters+does+gpt+3+have).<jupyter_code>#hide
class MySimpleModel(torch.nn.Module):
    def __init__(self):
        super(MySimpleModel, self).__init__()
        self.linear = nn.Linear(1,1)
        self.linear.weight.data = nn.Parameter(torch.tensor([[1.0]]))
        self.linear.bias.data = nn.Parameter(torch.tensor([1.0]))
        self.a = self.linear.weight
        self.b = self.linear.bias
        
    def forward(self, x):
        x = x.unsqueeze(1)
        x = self.linear(x)
        return x
#hide
class OptimizerInsight(Callback):
    def before_batch(self):
        if self.training:
            self.a_guess+=[float(self.model.a)]
            self.b_guess+=[float(self.model.b)]
            #plt.plot(theta_real,b_real, 'ro', markersize=5)
            #print(self.model.theta.tolist())
    def before_epoch(self):
        try:
            self.a_guess = [self.a_guess[-1]]
        except:
            self.a_guess = []
        try:
            self.b_guess = [self.b_guess[-1]]
        except:
            self.b_guess = []
        
    def after_epoch(self):
        plt.plot(self.a_guess, self.b_guess)
#hide
x_list = torch.tensor([14.0,86,28,51,28,29,72,62,84,15,42,62,47,35,9,38,44,99,13,21,28,20,8,64,99,70,27,17,8])
y_list = torch.tensor([58.0,202,86,132,86,88,174,154,198,60,114,154,124,100,48,106,118,228,56,72,86,70,46,158,228,170,84,64,46])
b_list = [(x,y) for x,y in zip(x_list, y_list)]
#hide
dl = DataLoader(b_list, bs=4)
#setting validation set equal to training dataset.  For this example, we are only interested in the training dataset
dls = DataLoaders(dl,dl)<jupyter_output><empty_output><jupyter_text>Gradient Descent Variants  [Batch Gradient Descent](https://ruder.io/optimizing-gradient-descent/index.htmlbatchgradientdescent) Forumula:  $\theta = \theta - \eta \cdot \nabla_\theta J( \theta)$  Code:  ```pythondef sgd_step(p, lr, **kwargs):    p.data.add_(p.grad.data, alpha=-lr)```Translation:    $\theta$ = p    $\eta$ = lr    $\nabla_\theta J( \theta)$ = p.grad.data<jupyter_code>def _do_epoch_BGD(self):
    self._do_epoch_train()
    self._step()
    self('after_step')
    self.opt.zero_grad()
    self._do_epoch_validate()
def _do_one_batch_BGD(self):
    self.pred = self.model(*self.xb)
    self('after_pred')
    if len(self.yb): self.loss = self.loss_func(self.pred, *self.yb)
    self('after_loss')
    if not self.training or not len(self.yb): return
    self('before_backward')
    self._backward()
    self('after_backward')
model = MySimpleModel()
loss_func = mse #F.mse_loss
𝜂 = 0.00005
partial_learner = partial(Learner,dls, model, loss_func, cbs=[OptimizerInsight])
learn = partial_learner(SGD, lr=𝜂)
learn._do_epoch = partial(_do_epoch_BGD,learn)
learn._do_one_batch = partial(_do_one_batch_BGD,learn)
learn.fit(10)<jupyter_output><empty_output><jupyter_text>[Stochastic Gradient Descent](https://ruder.io/optimizing-gradient-descent/index.htmlstochasticgradientdescent) Formula:  $\theta = \theta - \eta \cdot \nabla_\theta J( \theta; x^{(i)}; y^{(i)})$    Code:  ```pythondef sgd_step(p, lr, **kwargs):    p.data.add_(p.grad.data, alpha=-lr)```Translation:    $\theta$ = p    $\eta$ = lr    $\nabla_\theta J( \theta; x^{(i)}; y^{(i)})$ = p.grad.data<jupyter_code>def _do_epoch_SGD(self):
    self._do_epoch_train()
    self._do_epoch_validate()
def _do_one_batch_SGD(self):
    #pdb.set_trace()
    for i,b in enumerate(zip(self.xb[0][None].T,self.yb[0][None].T)):
        x,y = b
        self.pred = self.model(x)
        self('after_pred')
        if len(y): self.loss = self.loss_func(self.pred, y)
        self('after_loss')
        if not self.training or not len(self.yb): return
        self('before_backward')
        self._backward()
        self('after_backward')
        self._step()
        self('after_step')
        self.opt.zero_grad()
model = MySimpleModel()
loss_func = mse #F.mse_loss
𝜂 = 0.0001
partial_learner = partial(Learner,dls, model, loss_func, cbs=[OptimizerInsight])
learn = partial_learner(SGD, lr=𝜂)
learn._do_epoch = partial(_do_epoch_SGD,learn)
learn._do_one_batch = partial(_do_one_batch_SGD,learn)
learn.fit(10)<jupyter_output><empty_output><jupyter_text>[Mini-batch Gradient Descent](https://ruder.io/optimizing-gradient-descent/index.htmlminibatchgradientdescent) $\theta = \theta - \eta \cdot \nabla_\theta J( \theta; x^{(i:i+n)}; y^{(i:i+n)})$ Comments about Batch Gradient Descent, Stochastic Gradient Descent, and Mini-batch Gradient Descent:  All three of these use itentical code for the optimizer itself.  The huge difference between these three Optimizers is when the step actually occurs.<jupyter_code>def _do_epoch_MBGD(self):
    self._do_epoch_train()
    self._do_epoch_validate()
def _do_one_batch_MBGD(self):
    self.pred = self.model(*self.xb)
    self('after_pred')
    if len(self.yb): self.loss = self.loss_func(self.pred, *self.yb)
    self('after_loss')
    if not self.training or not len(self.yb): return
    self('before_backward')
    self._backward()
    self('after_backward')
    self._step()
    self('after_step')
    self.opt.zero_grad()
model = MySimpleModel()
loss_func = mse #F.mse_loss
𝜂 = 0.0001
partial_learner = partial(Learner,dls, model, loss_func, cbs=[OptimizerInsight])
learn = partial_learner(SGD, lr=𝜂)
learn._do_epoch = partial(_do_epoch_MBGD,learn)
learn._do_one_batch = partial(_do_one_batch_MBGD,learn)
learn.fit(10)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13442192739466394, 'fraction_numerical': 0.02536812946493658, 'mean_word_length': 4.0702143385070215, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 25, 'https://': 5, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '34434', 'license': 'Apache-2.0', 'n_tokens_mistral': 2543, 'n_tokens_neox': 2438, 'n_words': 551, 'path': '_notebooks/2020-09-07-Optimizer_Code_Supplement.ipynb', 'repo_name': 'kevinbird15/Blog', 'size': '6881'}