<jupyter_start><jupyter_text>Import Packages<jupyter_code>import numpy as np
import tensorflow as tf
import tensorflow_addons as tfa

from tensorflow.keras import optimizers
num_samples = 20
num_epochs  = 10

tf.random.set_seed(0)
x = tf.random.normal((num_samples, 2), mean=0, stddev=1)
y = tf.reshape(3.145 * x[:, 0] +  6.323 * x[:, 1], (-1, 1))

square_loss = lambda: tf.reduce_mean(tf.pow(y - tf.matmul(x, w), 2))
evaluate = lambda i: print(''.join((
    f'epoch:{i+1:>3}{"":>4}weight 1: {w.numpy()[0, 0]:>7.3f}',
    f'{"":>4}weight 2: {w.numpy()[1, 0]:>7.3f}{"":>4}loss: {square_loss().numpy():>7.3f}'
)))

def train(num_epochs, optimizer, loss, weight):
    evaluate(-1)
    for i in range(0, num_epochs):
        optimizer.minimize(loss, var_list=[w])
        evaluate(i)<jupyter_output><empty_output><jupyter_text>Adaptive Gradient Descent (AdaGrad)Equation:$\mathbf{w}^{(t+1)}=\mathbf{w}^{(t)}-\frac{\eta}{\sqrt{\sum\limits_{i=1}^t\mathbf{g}^{(t)T}\mathbf{g}^{(t)}+\varepsilon}}\mathbf{g}^{(t)}$$\mathbf{g}^{(t)}=∇_\mathbf{w}L(\mathbf{w}^{(t)})$Parameters:- $\varepsilon$ is a constant that make sure the denominator is not zero (default in Tensorflow/Keras: $1e-7$).- $\eta$ is the learning rate (default in Tensorflow/Keras: $0.001$)Properties:- The magnitude of the update will always be smaller in later iterations.- The update become very inefficient in later iterations.- Vulnerable to local minimum.- Converge even with large learning rate.<jupyter_code>eta = 1
w = tf.Variable(tf.zeros((2, 1)))
opt = optimizers.Adagrad(learning_rate=eta)
train(num_epochs, opt, square_loss, w)<jupyter_output>epoch:  0    weight 1:   0.000    weight 2:   0.000    loss:  37.729
epoch:  1    weight 1:   0.999    weight 2:   0.999    loss:  23.809
epoch:  2    weight 1:   1.574    weight 2:   1.639    loss:  17.052
epoch:  3    weight 1:   1.973    weight 2:   2.126    loss:  12.899
epoch:  4    weight 1:   2.267    weight 2:   2.523    loss:  10.088
epoch:  5    weight 1:   2.492    weight 2:   2.860    loss:   8.075
epoch:  6    weight 1:   2.666    weight 2:   3.152    loss:   6.577
epoch:  7    weight 1:   2.801    weight 2:   3.410    loss:   5.430
epoch:  8    weight 1:   2.907    weight 2:   3.639    loss:   4.530
epoch:  9    weight 1:   2.989    weight 2:   3.844    loss:   3.812
epoch: 10    weight 1:   3.053    weight 2:   4.030    loss:   3.229<jupyter_text>RMSpropEquation:$\mathbf{w}^{(t+1)}=\mathbf{w}^{(t)}-\frac{\eta}{\sqrt{G^{(t)}} + \varepsilon}\mathbf{g}^{(t)}$$\mathbf{g}^{(t)}=\nabla_\mathbf{w}L(\mathbf{w}^{(t)})$$G^{(t)}=\rho G^{(t-1)}+(1-\rho)\mathbf{g}^{(t)T}\mathbf{g}^{(t)}$Parameters:- $\varepsilon$ is a constant that make sure the denominator is not zero (default in Tensorflow/Keras: $1e-7$).- $\eta$ is the learning rate (default in Tensorflow/Keras: $0.001$)- $\rho$ is the parameter that control how much the model should consider the computed gradient (sum of magnitude) in previous iteration. (default in Tensorflow/Keras $0.9$)Properties:- Vulnerable to local minimum.- Compare to AdaGrad, RMSProp use a weighted average between previous magnitude of gradient and the magnitude of gradient in current iteration. Keeping the normalization on the magnitude of gradient, but allows more efficient optimization in later iterations.<jupyter_code>eta = 1
w = tf.Variable(tf.zeros((2, 1)))
opt = optimizers.RMSprop(learning_rate=eta)
train(num_epochs, opt, square_loss, w)<jupyter_output>epoch:  0    weight 1:   0.000    weight 2:   0.000    loss:  37.729
epoch:  1    weight 1:   3.162    weight 2:   3.162    loss:   6.049
epoch:  2    weight 1:   3.368    weight 2:   4.563    loss:   1.867
epoch:  3    weight 1:   3.282    weight 2:   5.347    loss:   0.576
epoch:  4    weight 1:   3.218    weight 2:   5.800    loss:   0.165
epoch:  5    weight 1:   3.182    weight 2:   6.055    loss:   0.043
epoch:  6    weight 1:   3.163    weight 2:   6.193    loss:   0.010
epoch:  7    weight 1:   3.153    weight 2:   6.263    loss:   0.002
epoch:  8    weight 1:   3.149    weight 2:   6.297    loss:   0.000
epoch:  9    weight 1:   3.146    weight 2:   6.313    loss:   0.000
epoch: 10    weight 1:   3.146    weight 2:   6.319    loss:   0.000<jupyter_text>MomentumEquation:$\mathbf{w}^{(t+1)}=\mathbf{w}^{(t)}-\mathbf{v}^{(t)}$$\mathbf{v}^{(t)}=\gamma\mathbf{v}^{(t-1)}+\eta\nabla_\mathbf{w}L(\mathbf{w}^{(t)})$Parameters:- $\eta$ is the learning rate (default in Tensorflow/Keras: $0.01$)- $\gamma$ is the momentum coefficient controlling how much the model should consider the computed gradient before.Properties:- Implemented in `SGD` and `RMSProp` in Tensorflow and Keras.- Generalize well to most of the application.- Consider both magnitude and direction of previous gradient when update the parameters.- **Might not converge with large learning rate.**- **Less vulnerable to local minimum**.<jupyter_code>eta = 0.1
w = tf.Variable(tf.zeros((2, 1)))
opt = optimizers.SGD(learning_rate=eta, momentum=0.1)
train(num_epochs, opt, square_loss, w)<jupyter_output>epoch:  0    weight 1:   0.000    weight 2:   0.000    loss:  37.729
epoch:  1    weight 1:   0.757    weight 2:   0.817    loss:  26.427
epoch:  2    weight 1:   1.419    weight 2:   1.604    loss:  17.908
epoch:  3    weight 1:   1.920    weight 2:   2.283    loss:  12.251
epoch:  4    weight 1:   2.290    weight 2:   2.860    loss:   8.505
epoch:  5    weight 1:   2.561    weight 2:   3.351    loss:   5.987
epoch:  6    weight 1:   2.757    weight 2:   3.770    loss:   4.266
epoch:  7    weight 1:   2.898    weight 2:   4.128    loss:   3.072
epoch:  8    weight 1:   2.999    weight 2:   4.434    loss:   2.230
epoch:  9    weight 1:   3.069    weight 2:   4.696    loss:   1.631
epoch: 10    weight 1:   3.118    weight 2:   4.921    loss:   1.199<jupyter_text>Nesterov Accelerate Gradient (NAG)Equation:$\mathbf{w}^{(t+1)}=\mathbf{v}^{(t+1)}-\eta\nabla_\mathbf{v}L(\mathbf{v}^{(t)})$$\mathbf{v}^{(t+1)}=\mathbf{w}^{(t)}+\gamma(\mathbf{w}^{(t)}-\mathbf{w}^{(t-1)})$Parameters:- $\eta$ is the learning rate (default in Tensorflow/Keras: $0.01$)- $\gamma$ is the momentum coefficient controlling how much the model should consider the computed gradient before.Properties:- Implemented in `SGD` in Tensorflow and Keras.- Generalize well to most of the application.- Consider both magnitude and direction of previous gradient when update the parameters.- NAG consider the approximation for the second order derivative of the gradient. i.e. gradient of $\gamma(\mathbf{w}^{(t)}-\mathbf{w}^{(t-1)})$. Therefore it generally converges faster than Momentum.- **Might not converge with large learning rate.**- **Less vulnerable to local minimum.**<jupyter_code>eta = 0.1
w = tf.Variable(tf.zeros((2, 1)))
opt = optimizers.SGD(learning_rate=eta, momentum=0.1, nesterov=True)
train(num_epochs, opt, square_loss, w)<jupyter_output>epoch:  0    weight 1:   0.000    weight 2:   0.000    loss:  37.729
epoch:  1    weight 1:   0.833    weight 2:   0.899    loss:  25.418
epoch:  2    weight 1:   1.466    weight 2:   1.671    loss:  17.308
epoch:  3    weight 1:   1.939    weight 2:   2.329    loss:  11.958
epoch:  4    weight 1:   2.291    weight 2:   2.889    loss:   8.377
epoch:  5    weight 1:   2.551    weight 2:   3.368    loss:   5.943
epoch:  6    weight 1:   2.742    weight 2:   3.777    loss:   4.262
epoch:  7    weight 1:   2.881    weight 2:   4.128    loss:   3.086
epoch:  8    weight 1:   2.981    weight 2:   4.429    loss:   2.252
epoch:  9    weight 1:   3.053    weight 2:   4.688    loss:   1.654
epoch: 10    weight 1:   3.104    weight 2:   4.910    loss:   1.221<jupyter_text>Adam[Equation](https://towardsdatascience.com/adam-latest-trends-in-deep-learning-optimization-6be9a291375c):$\mathbf{w}^{(t+1)}=\mathbf{w}^{(t)}-\frac{\eta}{\sqrt{\hat{v}^{(t)} + \varepsilon}}\mathbf{\hat{m}}^{(t)}$$\mathbf{g}^{(t)}=\nabla_\mathbf{w}L(\mathbf{w}^{(t)})$$\mathbf{m}^{(t)}=\beta_1\mathbf{m}^{(t-1)}+(1-\beta_1)\mathbf{g}^{(t)}$$G^{(t)}=\mathbf{g}^{(t)T}\mathbf{g}^{(t)}$$v^{(t)}=\beta_2v^{(t-1)}+(1-\beta_2)G^{(t)}$Parameters:- $\eta$ is the learning rate (default in Tensorflow/Keras: $0.01$)- $\varepsilon$ is a constant that make sure the denominator is not zero (default in Tensorflow/Keras: $1e-7$).- $\beta_1$ is the parameter that control how much the model should consider the (mainly) the direction of gradient before. (default in Tensorflow/Keras $0.9$)- $\beta_1$ is the parameter that control how much the model should consider the magnitude of gradient before. (default in Tensorflow/Keras $0.999$)Properties:- Consider both magnitude and direction of previous gradient when update the parameters.- Generalize well to most of the application.- Using the exponentially moving average to adaptively adjust the learning rate.- Converge with slightly larger large learning rate.- Less vunerable to local minimum.<jupyter_code>eta = 1
w = tf.Variable(tf.zeros((2, 1)))
opt = optimizers.Adam(learning_rate=eta)
train(num_epochs, opt, square_loss, w)<jupyter_output>epoch:  0    weight 1:   0.000    weight 2:   0.000    loss:  37.729
epoch:  1    weight 1:   1.000    weight 2:   1.000    loss:  23.800
epoch:  2    weight 1:   1.976    weight 2:   1.991    loss:  13.610
epoch:  3    weight 1:   2.896    weight 2:   2.964    loss:   7.039
epoch:  4    weight 1:   3.707    weight 2:   3.907    loss:   3.652
epoch:  5    weight 1:   4.347    weight 2:   4.805    loss:   2.615
epoch:  6    weight 1:   4.767    weight 2:   5.638    loss:   2.853
epoch:  7    weight 1:   4.960    weight 2:   6.387    loss:   3.457
epoch:  8    weight 1:   4.952    weight 2:   7.034    loss:   3.919
epoch:  9    weight 1:   4.787    weight 2:   7.562    loss:   4.068
epoch: 10    weight 1:   4.504    weight 2:   7.965    loss:   3.918<jupyter_text>Adam with Weight DecayWhen training the neural network, we often add $L2$-penalty to the loss function in order to prevent overfitting:$L(y, f)=(\mathbf{y}-f(\mathbf{X}, \mathbf{w}))^2+\frac{\lambda}{2}\mathbf{w}^T\mathbf{w}=L'(\mathbf{y}, f(\mathbf{X}, \mathbf{w}))+\frac{\lambda}{2}\mathbf{w}^T\mathbf{w}$However, if we introduce $L2$-penalty in combination with Adam optimizer:$\mathbf{w}^{(t+1)}=\mathbf{w}^{(t)}-\frac{\eta}{\sqrt{\hat{y}^{(t)}+\varepsilon}}\frac{1}{1-\beta_1}(\beta_1\mathbf{m}^{(t-1)}+(1-\beta_1)(\mathbf{g}^{(t)}+\lambda\mathbf{w}^{(t)}))$If $\hat{y}^{(t)}$ is large (suggests that the magnitude of the gradient is large), the $L2$-penalty actually becomes smaller. In other words, adding $L2$-penalty only penalize the weight with smaller changes, and therefore the effect of using $L2$-penalty with Adam have limited effect. To address this, [<NAME> and <NAME>](https://arxiv.org/pdf/1711.05101.pdf) proposed a new method to decouples the choice of weight decay factor from the setting of the learning rate. To be more specific, the authors proposed:$\mathbf{w}^{(t+1)}=\mathbf{w}^{(t)}-\eta(\frac{1}{\sqrt{\hat{v}^{(t)} + \varepsilon}}\mathbf{\hat{m}}^{(t)}+\lambda\mathbf{w}^{(t)})$<jupyter_code>eta = 1
weight_decay = 0.1
w = tf.Variable(tf.zeros((2, 1)))
opt = tfa.optimizers.AdamW(weight_decay=weight_decay, learning_rate=eta)
train(num_epochs, opt, square_loss, w)<jupyter_output>epoch:  0    weight 1:   0.000    weight 2:   0.000    loss:  37.729
epoch:  1    weight 1:   1.000    weight 2:   1.000    loss:  23.799
epoch:  2    weight 1:   1.876    weight 2:   1.891    loss:  14.485
epoch:  3    weight 1:   2.616    weight 2:   2.677    loss:   8.659
epoch:  4    weight 1:   3.201    weight 2:   3.362    loss:   5.294
epoch:  5    weight 1:   3.616    weight 2:   3.948    loss:   3.474
epoch:  6    weight 1:   3.854    weight 2:   4.438    loss:   2.466
epoch:  7    weight 1:   3.930    weight 2:   4.838    loss:   1.795
epoch:  8    weight 1:   3.872    weight 2:   5.153    loss:   1.246
epoch:  9    weight 1:   3.715    weight 2:   5.389    loss:   0.784
epoch: 10    weight 1:   3.499    weight 2:   5.555    loss:   0.445
{'fraction_non_alphanumeric': 0.15380107166778298, 'fraction_numerical': 0.09804085733422639, 'mean_word_length': 3.0809702767338574, 'pattern_counts': {'":': 3, '<': 25, '<?xml version=': 0, '>': 32, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '482308', 'license': 'MIT', 'n_tokens_mistral': 4901, 'n_tokens_neox': 4313, 'n_words': 1313, 'path': 'notebooks/optimizers.ipynb', 'repo_name': 'dn070017/Neural-Networks-Tensorflow', 'size': '11959'}