<jupyter_start><jupyter_text>Implementing Gradient Descent in Python Here, we will implement a simple representation of gradient descent using python. We will create an arbitrary loss function and attempt to find a local minimum value for that function. Our function will be this-f(x) = x^3-5x^2+7<jupyter_code>import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline<jupyter_output><empty_output><jupyter_text>We will first visualize this function with a set of values ranging from -1 and 3 (arbitrarily chosen to ensure steep curve)  creating the function and plotting it<jupyter_code>#Get 100000 evenly spaced numbers between -1 and 3 
#(arbitratil chosen to ensure steep curve)

function = lambda x : (x**3)-(3*(x**2))+7
x = np.linspace(-1,3,num=100000)
#Ploting the curve
plt.plot(x,function(x))
plt.show()<jupyter_output><empty_output><jupyter_text>Derivative function:  This function takes in a value of x and returns its derivative at x based on the initial function we pecified.It is shown below:<jupyter_code>def deriv(x):
    '''
    argument :
    x -> a numerical value of x
    
    returns : 
    x_deriv -> a numerical value of the derivative of x
    '''
    
    x_deriv = 3*(x**2)-(6*(x))
    return x_deriv
def step(x_new,x_prev,precision,l_r):
    '''
    Arguments:
    
    x_new -> a starting value of x that will get updated based on the learning rate
    
    x_prev -> the previous value of x that is getting updated to the new one
    
    precision -> a precision that determines the stop of the stepwise descent 
    
    l_r -> the learning rate (size of each descent step)
    
    Output:
    
    1. Prints out the latest new value of x which equates to the minimum we are looking for
    2. Prints out the the number of x values which equates to the number of gradient descent steps
    3. Plots a first graph of the function with the gradient descent path
    4. Plots a second graph of the function with a zoomed in gradient descent path in the important area
    '''
    
    x_list,y_list = [x_new],[function(x_new)]
    # keep looping until your desired precision
    while abs(x_new - x_prev)>precision:
        # change the value of x
        x_prev = x_new
        
        # get the derivetive at the old value of x
        d_x = - deriv(x_prev)
        
        # get your new value of x which will little
        # closer to the global minima
        x_new = x_prev + (l_r * d_x)
        
        # append the x_new to the x_list
        x_list.append(x_new)
        
        #append the y value at x_new in y_list
        y_list.append(function(x_new))
    
    print("Local minimum occurs at : " + str(x_new))
    print("Number of steps : " +str(len(x_list)))
    
    plt.subplot(1,2,2)
    plt.scatter(x_list,y_list,c="g")
    #plt.plot(x_list,y_list,c="g")
    plt.plot(x,function(x),c="b")
    plt.title("Gradient descent")
    plt.show()
    
    plt.subplot(1,2,1)
    plt.scatter(x_list,y_list,c="g")
    #plt.plot(x_list,y_list,c="g")
    plt.plot(x,function(x),c="b")
    plt.xlim([1.0,2.1])
    plt.title("Zoomed in Gradient descent to Key Area")
    plt.show()
    
    plt.subplot(1,2,1)
    plt.scatter(x_list,y_list,c="g")
    #plt.plot(x_list,y_list,c="g")
    plt.plot(x,function(x),c="b")
    plt.xlim([1.75,2.25])
    plt.title("Zoomed in Gradient descent to Key Area")
    plt.show()
plt.subplot(1,2,2)
plt.subplot(1,2,1)
#Implement gradient descent 
#(all the arguments are arbitrarily chosen)

step(0.5, 0, 0.001,0.05)
step(0.1, 0, 0.001,0.05)
step(2.95, 0, 0.001,0.05)<jupyter_output>Local minimum occurs at : 2.0017933729196633
Number of steps : 17<jupyter_text>Another Example Gradient descent Gradient descent on function (x**5) - (x**3) - (5*(x**2)) + 4<jupyter_code>import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
fun = lambda x : (x**5) - (x**3) - (5*(x**2)) +x + 4
x1 = np.linspace(-2,2,1000)
plt.plot(x1,fun(x1))
def derivative(x):
    x_deriv = (5*(x**4)) - (3*(x**2)) - 10*x +1
    return x_deriv
def NewStep(x_new,x_prev,precision,l_r):
    x_list,y_list = [x_new],[fun(x_new)]
    
    while abs(x_new-x_prev) > precision:
        x_prev = x_new
        d_x = -derivative(x_prev)
        x_new = x_prev + (l_r*d_x)
        
        x_list.append(x_new)
        y_list.append(fun(x_new))
    
    print("Global minima accured at : " + str(x_new))
    print("Number of step taken : " + str(len(x_list)))
    
    plt.subplot(1,2,2)
    plt.scatter(x_list,y_list,c="b")
    plt.plot(x_list,y_list,c="b")
    plt.plot(x1,fun(x1),c="g")
    plt.title("Gradient descient")
    plt.show() 
    
    plt.subplot(1,2,1)
    plt.scatter(x_list,y_list,c="b")
    plt.plot(x_list,y_list,c="b")
    plt.plot(x1,fun(x1))
    plt.title("Zoomed in Gradient descent in key area ")
    plt.xlim(0,2)
    plt.show()
NewStep(0.1,0,0.001,0.05)<jupyter_output>Global minima accured at : 1.3904511244299729
Number of step taken : 35
{'fraction_non_alphanumeric': 0.12119366626065774, 'fraction_numerical': 0.03552578156719448, 'mean_word_length': 3.1683587140439933, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 23, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '623738', 'license': 'MIT', 'n_tokens_mistral': 1892, 'n_tokens_neox': 1791, 'n_words': 499, 'path': 'Gradient-Descent.ipynb', 'repo_name': 'ManjeetRaj/Implementation-of-Gradient-Descent-in-Python', 'size': '4926'}