<jupyter_start><jupyter_text>Batch and Stochastic TrainingThis python function illustrates two different training methods: batch and stochastic training.  For each model, we will use a regression model that predicts one model variable.We start by loading the necessary libraries and resetting the computational graph.<jupyter_code>import matplotlib.pyplot as plt
import numpy as np
import tensorflow as tf
import tensorflow.compat.v1 as tf
tf.disable_v2_behavior()
from tensorflow.python.framework import ops
ops.reset_default_graph()<jupyter_output>WARNING:tensorflow:From /home/rong/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/compat/v2_compat.py:88: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
Instructions for updating:
non-resource variables are not supported in the long term<jupyter_text>We start a computational graph session.<jupyter_code>sess = tf.Session()<jupyter_output><empty_output><jupyter_text>Stochastic Training---------------------- Generate DataThe data we will create is 100 random samples from a `Normal(mean = 1, sd = 0.1)`.  The target will be an array of size 100 filled with the constant 10.0.We also create the necessary placeholders in the graph for the data and target.  Note that we use a shape of `[1]` for stochastic training.<jupyter_code>x_vals = np.random.normal(1, 0.1, 100)
y_vals = np.repeat(10., 100)

x_data = tf.placeholder(shape=[1], dtype=tf.float32)
y_target = tf.placeholder(shape=[1], dtype=tf.float32)<jupyter_output><empty_output><jupyter_text>Model Variables and OperationsWe create the one variable in the graph, `A`.  We then create the model operation, which is just the multiplication of the input data and `A`.<jupyter_code># Create variable (one model parameter = A)
A = tf.Variable(tf.random_normal(shape=[1]))

# Add operation to graph
my_output = tf.multiply(x_data, A)<jupyter_output><empty_output><jupyter_text>Loss FunctionFor this, we choose the L2 loss.  We can easily choose the L1 loss by replacing `tf.square()` with `tf.abs()`.<jupyter_code># Add L2 loss operation to graph
loss = tf.square(my_output - y_target)<jupyter_output><empty_output><jupyter_text>Optimization and InitializationFor the optimization function, we will choose the standard Gradient Descent Algorithm with a learning rate of `0.02`.  We also add and run a variable initialization operation.<jupyter_code># Create Optimizer
my_opt = tf.train.GradientDescentOptimizer(0.02)
train_step = my_opt.minimize(loss)

# Initialize variables
init = tf.global_variables_initializer()
sess.run(init)<jupyter_output><empty_output><jupyter_text>Train ModelWe run the training step for 100 iterations and print off the value of `A` and the loss every 5 iterations.<jupyter_code>loss_stochastic = []
# Run Loop
for i in range(100):
    rand_index = np.random.choice(100)
    rand_x = [x_vals[rand_index]]
    rand_y = [y_vals[rand_index]]
    sess.run(train_step, feed_dict={x_data: rand_x, y_target: rand_y})
    if (i+1)%5==0:
        print('Step #' + str(i+1) + ' A = ' + str(sess.run(A)))
        temp_loss = sess.run(loss, feed_dict={x_data: rand_x, y_target: rand_y})
        print('Loss = ' + str(temp_loss))
        loss_stochastic.append(temp_loss)<jupyter_output>Step #5 A = [1.6898265]
Loss = [68.30719]
Step #10 A = [3.2756445]
Loss = [47.73897]
Step #15 A = [4.552496]
Loss = [29.584751]
Step #20 A = [5.5492105]
Loss = [18.734362]
Step #25 A = [6.375694]
Loss = [18.06535]
Step #30 A = [7.012703]
Loss = [6.242101]
Step #35 A = [7.4105396]
Loss = [2.6205263]
Step #40 A = [7.803639]
Loss = [1.7787511]
Step #45 A = [8.236774]
Loss = [7.2427974]
Step #50 A = [8.505573]
Loss = [0.93380576]
Step #55 A = [8.713567]
Loss = [0.02651341]
Step #60 A = [9.025083]
Loss = [5.1584845]
Step #65 A = [9.125193]
Loss = [3.6211145]
Step #70 A = [9.281811]
Loss = [0.00219164]
Step #75 A = [9.344097]
Loss = [0.00039907]
Step #80 A = [9.345924]
Loss = [0.0430604]
Step #85 A = [9.429876]
Loss = [0.00137173]
Step #90 A = [9.44043]
Loss = [3.3481634]
Step #95 A = [9.588872]
Loss = [1.669579]
Step #100 A = [9.488841]
Loss = [0.07389808]<jupyter_text>Batch Training------------------We start by resetting the computational graph<jupyter_code># Batch Training:
# Re-initialize graph
ops.reset_default_graph()
sess = tf.Session()<jupyter_output><empty_output><jupyter_text>For Batch training, we need to declare our batch size. The larger the batch size, the smoother the convergence will be towards the optimal value.  But if the batch size is too large, the optimization algorithm may get stuck in a local minimum, where a more stochastic convergence may jump out.Here, the we may change the batch size from 1 to 100 to see the effects of the batch size on the convergence plots at the end.<jupyter_code># Declare batch size
batch_size = 25<jupyter_output><empty_output><jupyter_text>Generate the DataThe data we will create is 100 random samples from a `Normal(mean = 1, sd = 0.1)`.  The target will be an array of size 100 filled with the constant 10.0.We also create the necessary placeholders in the graph for the data and target.Note that here, our placeholders have shape `[None, 1]`, where the batch size will take the place of the `None` dimension.<jupyter_code># Create data
x_vals = np.random.normal(1, 0.1, 100)
y_vals = np.repeat(10., 100)

# we change the shape of the placeholder . They are two dimensions, where the first dimension is None. We could have
# explicitly set it to 25, but we can generalize and use the None value.
x_data = tf.placeholder(shape=[None, 1], dtype=tf.float32)
y_target = tf.placeholder(shape=[None, 1], dtype=tf.float32)<jupyter_output><empty_output><jupyter_text>Model Variables and OperationsWe create the one variable in the graph, `A`.  We then create the model operation, which is just the multiplication of the input data and `A`.<jupyter_code># Create variable (one model parameter = A)
#  in order to do matric mutiplycation, we set A to 1 x 1
A = tf.Variable(tf.random_normal(shape=[1,1]))

# Add operation to graph
my_output = tf.matmul(x_data, A)<jupyter_output><empty_output><jupyter_text>Loss FunctionFor this, we choose the L2 loss.  We can easily choose the L1 loss by replacing `tf.square()` with `tf.abs()`.<jupyter_code># Add L2 loss operation to graph
loss = tf.reduce_mean(tf.square(my_output - y_target))<jupyter_output><empty_output><jupyter_text>Optimization and InitializationFor the optimization function, we will choose the standard Gradient Descent Algorithm with a learning rate of `0.02`.  We also add and run a variable initialization operation.<jupyter_code># Initialize variables
init = tf.global_variables_initializer()
sess.run(init)

# Create Optimizer
my_opt = tf.train.GradientDescentOptimizer(0.02)
train_step = my_opt.minimize(loss)<jupyter_output><empty_output><jupyter_text>Train ModelWe run the training step for 100 iterations and print off the value of `A` and the loss every 5 iterations.Note that here we select a batch of data instead of just one data point.<jupyter_code>loss_batch = []
# Run Loop
for i in range(100):
    rand_index = np.random.choice(100, size=batch_size)
    rand_x = np.transpose([x_vals[rand_index]])
    rand_y = np.transpose([y_vals[rand_index]])
    sess.run(train_step, feed_dict={x_data: rand_x, y_target: rand_y})
    if (i+1)%5==0:
        print('Step #' + str(i+1) + ' A = ' + str(sess.run(A)))
        temp_loss = sess.run(loss, feed_dict={x_data: rand_x, y_target: rand_y})
        print('Loss = ' + str(temp_loss))
        loss_batch.append(temp_loss)<jupyter_output>Step #5 A = [[1.218846]]
Loss = 76.36306
Step #10 A = [[2.842389]]
Loss = 52.22381
Step #15 A = [[4.1554513]]
Loss = 34.036705
Step #20 A = [[5.224248]]
Loss = 23.620054
Step #25 A = [[6.0980124]]
Loss = 15.185609
Step #30 A = [[6.798677]]
Loss = 10.290394
Step #35 A = [[7.372198]]
Loss = 7.4269495
Step #40 A = [[7.844726]]
Loss = 5.9302974
Step #45 A = [[8.221858]]
Loss = 2.7095554
Step #50 A = [[8.529215]]
Loss = 3.281688
Step #55 A = [[8.763024]]
Loss = 1.2678502
Step #60 A = [[8.979831]]
Loss = 1.5252069
Step #65 A = [[9.156942]]
Loss = 1.5001912
Step #70 A = [[9.310086]]
Loss = 1.3281527
Step #75 A = [[9.430096]]
Loss = 1.9603093
Step #80 A = [[9.547461]]
Loss = 1.1993431
Step #85 A = [[9.632623]]
Loss = 1.3519974
Step #90 A = [[9.698171]]
Loss = 1.6457906
Step #95 A = [[9.7291155]]
Loss = 0.96946937
Step #100 A = [[9.763433]]
Loss = 0.96356195<jupyter_text>Plot Stochastic vs Batch TrainingHere is the matplotlib code to plot the loss for each.<jupyter_code>plt.plot(range(0, 100, 5), loss_stochastic, 'b-', label='Stochastic Loss')
plt.plot(range(0, 100, 5), loss_batch, 'r--', label='Batch Loss, size=20')
plt.legend(loc='upper right', prop={'size': 11})
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12213913437570814, 'fraction_numerical': 0.09370043054611375, 'mean_word_length': 4.784403669724771, 'pattern_counts': {'":': 0, '<': 58, '<?xml version=': 0, '>': 58, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '521632', 'license': 'MIT', 'n_tokens_mistral': 3625, 'n_tokens_neox': 3069, 'n_words': 994, 'path': '02_TensorFlow_Way/06_Working_with_Batch_and_Stochastic_Training/06_batch_stochastic_training.ipynb', 'repo_name': 'Rong-creator/tensorflow_cookbook', 'size': '8826'}