<jupyter_start><jupyter_text>Gradient Descent with Momentum\begin{align}p(\tau) & =  \nabla E(w(\tau-1)) + \beta p(\tau-1) \\w(\tau) & =  w(\tau-1) - \alpha p(\tau) \end{align}When $\beta=0$, we recover gradient descent.| Symbol           | Meaning                                                        ||:-----------------|:---------------------------------------------------------------|| $$\tau$$         | Iteration number                                               || $$w$$            | Coefficients vector                                            || $$w(\tau)$$      | Coefficients vector at **$\tau$** iteration                    || $$\alpha$$       | Learning rate                                                  || $$\beta$$        | Momentum                                                       || $$\nabla$$       | Gradient function                                              || $$E(w(\tau-1))$$ | Error function with **w** coefficients at **$\tau$** iteration |<jupyter_code>%matplotlib inline
import matplotlib.pylab as plt

import time
from IPython import display
import numpy as np


#y = np.array([7.46, 6.77, 12.74, 7.11, 7.81, 8.84, 6.08, 5.39, 8.15, 6.42, 5.73])
y = np.array([8.04, 6.95, 7.58, 8.81, 8.33, 9.96, 7.24, 4.26, 10.84, 4.82, 5.68])
#y = np.array([9.14, 8.14, 8.74, 8.77, 9.26, 8.10, 6.13, 3.10, 9.13, 7.26, 4.74])
x = np.array([10., 8., 13., 9., 11., 14., 6., 4., 12., 7., 5.])
N = len(x)

# Design matrix
A = np.vstack((np.ones(N), x, x**2, x**3)).T

# Learning rate
alpha = 0.0014088
beta = 0.95
              
# Initial parameters
w = np.array([2., 1., 0.1, 0.001])
p = np.zeros(4)

# Chart
f = A.dot(w)

# loop
for epoch in range(10):
    # Error
    err = y - A.dot(w)
    
    # Total error
    E = np.sum(err ** 2) / N
    
    # Gradient
    dE = -2. * A.T.dot(err) / N
    p = dE + beta * p
    
    if epoch % 1 == 0: 
        print(epoch, ':', E)   
        
    # Perfom one descent step
    w = w - alpha * p<jupyter_output>0 : 253.620021909
1 : 5923352875.48
2 : 1.43399218111e+17
3 : 3.47157027299e+24
4 : 8.40436950711e+31
5 : 2.03462471613e+39
6 : 4.92564936844e+46
7 : 1.19245684516e+54
8 : 2.886834245e+61
9 : 6.98877447176e+68
{'fraction_non_alphanumeric': 0.20867579908675798, 'fraction_numerical': 0.13424657534246576, 'mean_word_length': 1.3867102396514162, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 4, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 12, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '486591', 'license': 'MIT', 'n_tokens_mistral': 957, 'n_tokens_neox': 772, 'n_words': 197, 'path': 'notebooks/GradientDescentWithMomentum.ipynb', 'repo_name': 'mustafagonul/machine-learning-examples', 'size': '2190'}