<jupyter_start><jupyter_text>Gradient descent visualization--- ***Author: <NAME>***This notebook accompanies the article: Deep Dive into Math Behind Deep Networks. The visualization that is created using this notebook serves to create an intuition about the gradient descent. Great thanks to MorvanZhou for the inspiration that I found here.  Imports<jupyter_code>import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D<jupyter_output><empty_output><jupyter_text>Settings<jupyter_code># learning rate
LR = 0.04
# parameters a and b of the real function
REAL_PARAMS = [1.2, 2.5]
# starting point for gradient descent
INIT_PARAMS = [-1, -1.5]
# output directory (the folder must be created on the drive)
OUTPUT_DIR = "gradient_descent"<jupyter_output><empty_output><jupyter_text>Auxiliary function<jupyter_code># precede the number with zeros, creating a thong of a certain length
def makeIndexOfLength(index, length):
    indexStr = str(index)
    return ('0' * (length - len(indexStr)) + indexStr)<jupyter_output><empty_output><jupyter_text>Performing the simulation<jupyter_code>x = np.linspace(-1, 1, 200, dtype=np.float32)

y_fun = lambda a, b: np.sin(b*np.cos(a*x))
tf_y_fun = lambda a, b: tf.sin(b*tf.cos(a*x))

noise = np.random.randn(200)/10
y = y_fun(*REAL_PARAMS) + noise

# tensorflow graph
a, b = [tf.Variable(initial_value=p, dtype=tf.float32) for p in INIT_PARAMS]
pred = tf_y_fun(a, b)
mse = tf.reduce_mean(tf.square(y-pred))
train_op = tf.train.GradientDescentOptimizer(LR).minimize(mse)

a_list, b_list, cost_list = [], [], []
with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    for t in range(180):
        a_, b_, mse_ = sess.run([a, b, mse])
        a_list.append(a_); b_list.append(b_); cost_list.append(mse_)
        result, _ = sess.run([pred, train_op])<jupyter_output><empty_output><jupyter_text>Creates visualization<jupyter_code># 3D cost figure
for angle in range(0, 180):
    plt.style.use('dark_background')
    fig = plt.figure(figsize=(8,8))
    ax = Axes3D(fig)
    a3D, b3D = np.meshgrid(np.linspace(-5, 5, 100), np.linspace(-5, 5, 100))  # parameter space
    cost3D = np.array([np.mean(np.square(y_fun(a_, b_) - y)) for a_, b_ in zip(a3D.flatten(), b3D.flatten())]).reshape(a3D.shape)
    ax.plot_surface(a3D, b3D, cost3D, rstride=1, cstride=1, cmap=plt.get_cmap('rainbow'), alpha=0.6)
    ax.scatter(a_list[0], b_list[0], zs=cost_list[0], s=300, c='r')  # initial parameter place
    ax.set_xlabel('a'); ax.set_ylabel('b')
    ax.plot(a_list[:angle], b_list[:angle], zs=cost_list[:angle], zdir='z', c='r', lw=3)    # plot 3D gradient descent
    ax.view_init(30 + (90 - angle)/5, 45 + angle*2)
    plt.savefig("./" + OUTPUT_DIR + "/" + makeIndexOfLength(angle, 3) + ".png")
    plt.close()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1450193593804998, 'fraction_numerical': 0.027455121436114043, 'mean_word_length': 5.072649572649572, 'pattern_counts': {'":': 0, '<': 22, '<?xml version=': 0, '>': 22, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '515934', 'license': 'MIT', 'n_tokens_mistral': 1126, 'n_tokens_neox': 1076, 'n_words': 252, 'path': '01_mysteries_of_neural_networks/01_mathematics_of_nn/Gradient descent visualization.ipynb', 'repo_name': 'angliu-bu/ILearnDeepLearning.py', 'size': '2848'}