<jupyter_start><jupyter_text>Using Tensorflow---This is a quick primer on setting up clean computational graphs, and how to execute them.The following commands will be covered in a practical manner.* tf.Session    * tf.Session.eval(TENSOR)    * tf.Session(graph=g)* tf.Graph* tf.placeholder* tf.constant* tf.Variable:) Yeah that's pretty much it<jupyter_code>import tensorflow as tf
tf.__version__<jupyter_output><empty_output><jupyter_text>How to use Tensorflow?Tensorflow is a library written mostly in C/C++, or a special GPU-specific language called CUDA/CuDNN, that creates a *static computation graph* with a Python API.What does this mean?It means we can write code in Python to run complex operations repetitively using Tensorflow. Major constraint: the organization has to have a static structure, meaning you have to specify __*clearly*__ where the inputs are.How does this affect the coding?Creation of tensorflow program is basically unusable by anyone outside of the creators **UNLESS** the tensors are named in an organized manner. Otherwise debugging someone else's tensorflow code will create a clinical level migraine.<jupyter_code>import numpy as np
# this is what I call pyenglish
# interjecting language with import statements<jupyter_output><empty_output><jupyter_text>Using tensorflow: Sessions and ConstantsTensorflow creates Sessions to manage the actual memory associated with various computational graphs and their executions. One can create different computational graphs with a single session, but to execute any specific part of a given computational graph, one needs a session.What does this mean?To run a Tensorflow program, we need to specify the parts, name them, and put them together––and then create a session to get the values of the different parts (tensors).<jupyter_code>"""Let's create a line!
Line form:
y = Ax + b
A - slope of 1
b - bias of -1"""

A = tf.constant([1], dtype=tf.float32)
b = tf.constant([-1],dtype=tf.float32)
x = tf.placeholder(shape=[1],dtype=tf.float32)

y = A*x + b

sess = tf.Session()
out = sess.run(y, feed_dict={x:[1]})<jupyter_output><empty_output><jupyter_text>EvaluatingIn order to evaluate the output of an operation (or to simply return the value of a tensor –– these are equivalent processes) Tensorflow sessions require input data to be passed into a `feed_dict` parameter to `sess.eval`There are two ways to pass in data as input tensors. There are two parts: _the key_, and the actual data.The trickier part by far is passing in the right keys to the input tensors.1. Pass the symbolic variable containing the tensor as a key, and the data as the value```pythoninput_tensor = tf.placeholder(...)feed_dict = {    input_tensor: input_data}```2. Pass a string corresponding the name of the tensor, and the 'input' operation. This only works if the input tensor has a name (or you know the default name). The format is "{tensor_name}:{0}" for assigning values to tensors.```pythoninput_tensor = tf.placeholder(..., name="x_input")feed_dict = {    "x_input:0": input_data}```We make sure to name our input tensors so that we can use method 2 whenever possible, that way we don't have to keep track of the symbolic instance. This sort of workaround probably won't be deprecated in future updates because this makes use of the internal representation of the computational graph. But I could be wrong. Takeaway : NAME YOUR TENSORS<jupyter_code>out = []
for i in range(100):
    # Here we 
    y_hat = sess.run(y, feed_dict = {x: [i]})
    out.append(y_hat)
import matplotlib.pyplot as plt
%matplotlib inline
plt.plot(list(range(100)),out)
plt.show()<jupyter_output><empty_output><jupyter_text>Fitting a LineFirst, some noisy data. We are going to stick with regression examples here because they are easy to visualize. But maybe we'll do n-dimensional visualization later.We'll be fitting a line using A•x + b = y_hat<jupyter_code>x = np.linspace(-2, 2, num=1000)
noise = np.random.randn(1000) * 2
slope = np.pi**2-np.exp(-1)
bias = 3.219230

ys = slope*x+noise+bias

plt.scatter(x, ys)
plt.show()
# Side note - this is a great way to look at docs for any given function
help(np.linspace)<jupyter_output>Help on function linspace in module numpy.core.function_base:

linspace(start, stop, num=50, endpoint=True, retstep=False, dtype=None)
    Return evenly spaced numbers over a specified interval.
    
    Returns `num` evenly spaced samples, calculated over the
    interval [`start`, `stop`].
    
    The endpoint of the interval can optionally be excluded.
    
    Parameters
    ----------
    start : scalar
        The starting value of the sequence.
    stop : scalar
        The end value of the sequence, unless `endpoint` is set to False.
        In that case, the sequence consists of all but the last of ``num + 1``
        evenly spaced samples, so that `stop` is excluded.  Note that the step
        size changes when `endpoint` is False.
    num : int, optional
        Number of samples to generate. Default is 50. Must be non-negative.
    endpoint : bool, optional
        If True, `stop` is the last sample. Otherwise, it is not included.
        Default is True.
    retstep : bo[...]<jupyter_text>Now we'll write in some extra Tensorflow code to model the line. However, to do so, we'll need to construct a new computational graph. This can be done by explicitly declaring a graph to use in our session. ```pythongraph = tf.Graph()  new graphwith tf.Session(graph=graph) as sess:      our new graph will be the default container for all subsequent operations    ...```<jupyter_code>with tf.Session(graph=tf.Graph()) as sess:
    # Data Input
    # ---------------------------------------------
    x_in = tf.placeholder(tf.float32, name='x_input')
    labels = tf.placeholder(tf.float32, name='y_input')
    
    # Variable Declaration
    # ---------------------------------------------
    m = tf.Variable(tf.random_normal([1]),name='slope')
    b = tf.Variable(tf.random_normal([1]),name='bias')
    
    # Operations
    # ---------------------------------------------
    y_hat = x_in*m + b
    loss = tf.losses.mean_squared_error(labels, y_hat)

    # Optimizers & Initialization
    # ---------------------------------------------
    opt = tf.train.GradientDescentOptimizer(learning_rate=1e-3).minimize(loss)
    init_op = tf.global_variables_initializer()
    
    # Execution
    # ---------------------------------------------
    sess.run(init_op)
    
    # Training
    # ---------------------------------------------
    
    #   Visual code (Loss)
    fig, ax = plt.subplots()
    handles = []
    legend_labels = []
    
    for i in range(3):
        """
        i: the training cycle we are on (called an epoch)
        For every training cycle, randomly permute the indices to the x & y data, and iteratively train on that.
        This is called Stochatistic Gradient Descent. Hence we use the Gradient Descent Optimizer.
        (Other optimizers won't work on this style of training LOL)
.        """
        indices = np.random.permutation(len(x))
        losses = []
        for k in indices:
            # The results of running each operation are returned into result = [result of opt, result of loss]
            result = sess.run([opt,loss], feed_dict={x_in: [x[k]], labels: [ys[k]]})
            
            # Store the losses in an array for visualization
            losses.append(result[1])
        
        # Some more visual code to plot the losses
        line_handle, = ax.plot(list(range(len(x))),losses)
        handles.append(line_handle)
        legend_labels.append("Epoch {}".format(i+1))
    
    #   Visual code (Loss)
    fig.suptitle("Loss vs Time")
    ax.set_xlabel("Time")
    ax.set_ylabel("Loss")
    ax.legend(handles, legend_labels)
    fig.show()
    
    # Visualizing
    # ---------------------------------------------
    print('m:\t{0:4.3f}\t\tm_hat:\t{1[0]:4.3f}'.format(slope,sess.run(m)))
    print('b:\t{0:4.3f}\t\tb_hat:\t{1[0]:4.3f}'.format(bias,sess.run(b)))
    
    out = []
    for dat in x:
        out.append(sess.run(y_hat, feed_dict={x_in:[dat]}))
    fig, ax = plt.subplots()
    ax.scatter(x, ys)
    ax.plot(x, out,color='black')
    fig.suptitle("Evaluation")
    fig.show()<jupyter_output>/Users/codetesting/anaconda/envs/Kaggle/lib/python3.6/site-packages/matplotlib/figure.py:403: UserWarning: matplotlib is currently using a non-GUI backend, so cannot show the figure
  "matplotlib is currently using a non-GUI backend, "<jupyter_text>Your turn!We'll generate 3 dimensional data. Provide some infrastructure, and you'll be expected to fill in the rest! Data (2 dimensional)<jupyter_code>x1s = np.linspace(0,10, 1000)
x2s = np.linspace(0,10, 1000)
noise = np.random.randn(1000) 
true_bias = 0.043
slope1 = 4.56004
slope2 = 3.21904
ys = x1s*slope1+ x2s*slope2+ x1s*x2s + true_bias # yes you will have to model the nonlinearity

# Best Sol - Normal Eq
A = np.ones((len(x1s)))
x1x2s = x1s*x2s
A = np.asarray([x1s, x2s, x1x2s, A]).T
# print(A[:10])
# lhs = A.T.dot(A)
# rhs = A.T.dot(ys)
# solution = np.linalg.solve(lhs,rhs)
solution, c, _,__ = np.linalg.lstsq(A,ys)
print('best solution')
print('m1:',solution[0])
print('m2:',solution[1])
print('m3:',solution[2])
print('b:',solution[3])
print(np.sum(ys-np.matmul(A,solution)))
def dont_forget():
    plt.plot([-1]*2, [1, 3], color='black')
    plt.plot([-2,-2], [-3, -0.2], color='black')
    plt.plot([-0.5,-0.5], [-3, -0.2], color='black')
    plt.plot([1,1],[0.5, 3.5],color='black')
    plt.plot([2,2],[0.34, 2.5],color='black')
    plt.plot([0.75]*2,[-3.29, -0.5],color='black')
    plt.plot([1,3], [-3]*2, color='black')

    axis = plt.gca()
    axis.xaxis.set_visible(False)
    axis.yaxis.set_visible(False)
    axis.set_ylim([-4,4])
    axis.set_xlim([-3,3.5])
    plt.show()
dont_forget()<jupyter_output><empty_output><jupyter_text>Fill in the blanks, CS61a style<jupyter_code>def format_str(true_name, true_val, pred_name, pred_val):
    return '{0}:\t{1:4.3f}\t\t{2}:\t{3:4.3f}'.format(true_name, true_val, pred_name, pred_val)
    
with tf.Session(_____=______) as sess:
    x_input1 = tf.___________(dtype=tf.float32,shape=(1,),name=_____)
    x_input2 = tf.___________(dtype=tf.float32,shape=(1,),name=_____)
    y_input = tf.___________(dtype=tf.float32,shape=(1,),name=_____)
    
    m1 = tf.________(tf.truncated_normal([1],0.4), name=___)
    m2 = tf.________(tf.truncated_normal([1],-0.1), name=___)
    m3 = tf.________(tf.truncated_normal([1],0.3), name=____)
    b_hat = tf.________(tf.truncated_normal([1]), name=_____)
    
    y_hat = __ * x_input1 + __ * x_input2 + m3 * ______ * ______ + ______
    
    loss = tf.losses.mean_squared_error(_______, _______)
    opt = tf.________._______________(learning_rate=1e-4).minimize(_______)
    
    init_op = tf.___________________________()
    sess.___________
    
    num_epochs = 20
    handles = []
    leg_lab = []
    
    fig, ax = plt.subplots()
    for i in range(num_epochs):
        indices = np.random.permutation(len(____))
        losses = []
        for k in indices:
            results = sess.run([___,loss], 
                        feed_dict=
                                 {________: [x1s[k]],
                                  ________: [x2s[k]],
                                  ________: [ys[k]]}
                        )
            losses.append(results[1])
        line_handle, = ax.plot(list(range(len(x1s))), losses)
        
        handles.append(line_handle)
        leg_lab.append("Epoch: {}".format(i+1))
        
    ax.legend(handles, leg_lab)
    ax.set_xlabel("Time")
    ax.set_ylabel("Loss")
    fig.suptitle("Losses vs Time")
    fig.show()
    
    error = []
    for x1e, x2e, ye in zip(x1s, x2s, ys):
        error.append(sess.run(loss, feed_dict={x_input1:[x1e], x_input2:[x2e], y_input:[ye]}))

    print(format_str('m1',slope1,'m_hat1',sess.run(___)[0]))
    print(format_str('m2',slope2,'m_hat2',sess.run(___)[0]))
    print(format_str('m3',1.0,'m_hat3',sess.run(___)[0]))
    print(format_str('b',true_bias,'b_hat',sess.run(___)[0]))
    print(format_str('loss',np.sum(ys-np.matmul(A,solution)),'^loss^',np.sum(error)))
# solution
# Note: a good solution can be unstable, and < 5.0
# A great solution <<1.0, but I think this just doesn't happen (even in 10 epochs) bc
# 1) I'm not recording loss every epoch.
# 2) This is not implementing batch gradient descent.
# 3) Also, the x1*x2 term is really finicky for some reason
def format_str(true_name, true_val, pred_name, pred_val):
    return '{0}:\t{1:4.3f}\t\t{2}:\t{3:4.3f}'.format(true_name, true_val, pred_name, pred_val)
    
with tf.Session(graph=tf.Graph()) as sess:
    x_input1 = tf.placeholder(dtype=tf.float32,shape=(1,),name='x_in1')
    x_input2 = tf.placeholder(dtype=tf.float32,shape=(1,),name='x_in2')
    y_input = tf.placeholder(dtype=tf.float32,shape=(1,),name='y_in')
    
    m1 = tf.Variable(tf.truncated_normal([1],0.4), name='m1')
    m2 = tf.Variable(tf.truncated_normal([1],-0.1), name='m2')
    m3 = tf.Variable(tf.truncated_normal([1],0.3), name='m3')
    b_hat = tf.Variable(tf.truncated_normal([1]), name='b_hat')
    
    y_hat = m1 * x_input1 + m2 * x_input2 + m3 * x_input1 * x_input2 + b_hat
    
    loss = tf.losses.mean_squared_error(y_input, y_hat)
    opt = tf.train.GradientDescentOptimizer(learning_rate=1e-4).minimize(loss)
    
    init_op = tf.global_variables_initializer()
    sess.run(init_op)
    
    num_epochs = 20
    handles = []
    leg_lab = []
    
    fig, ax = plt.subplots()
    for i in range(num_epochs):
        indices = np.random.permutation(len(x1s))
        losses = []
        for k in indices:
            results = sess.run([opt,loss], 
                        feed_dict=
                                 {x_input1: [x1s[k]],
                                  x_input2: [x2s[k]],
                                  y_input: [ys[k]]}
                        )
            losses.append(results[1])
        line_handle, = ax.plot(list(range(len(x1s))), losses)
        
        handles.append(line_handle)
        leg_lab.append("Epoch: {}".format(i+1))
        
    ax.legend(handles, leg_lab)
    ax.set_xlabel("Time")
    ax.set_ylabel("Loss")
    fig.suptitle("Losses vs Time")
    fig.show()
    
    error = []
    for x1e, x2e, ye in zip(x1s, x2s, ys):
        error.append(sess.run(loss, feed_dict={x_input1:[x1e], x_input2:[x2e], y_input:[ye]}))

    print(format_str('m1',slope1,'m_hat1',sess.run(m1)[0]))
    print(format_str('m2',slope2,'m_hat2',sess.run(m2)[0]))
    print(format_str('m3',1.0,'m_hat3',sess.run(m3)[0]))
    print(format_str('b',true_bias,'b_hat',sess.run(b_hat)[0]))
    print(format_str('loss',np.sum(ys-np.matmul(A,solution)),'^loss^',np.sum(error)))<jupyter_output>/Users/codetesting/anaconda/envs/Kaggle/lib/python3.6/site-packages/matplotlib/figure.py:403: UserWarning: matplotlib is currently using a non-GUI backend, so cannot show the figure
  "matplotlib is currently using a non-GUI backend, "
{'fraction_non_alphanumeric': 0.15004666044527395, 'fraction_numerical': 0.023596853752832957, 'mean_word_length': 3.084671930302205, 'pattern_counts': {'":': 1, '<': 33, '<?xml version=': 0, '>': 30, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 11, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '86761', 'license': 'MIT', 'n_tokens_mistral': 5196, 'n_tokens_neox': 4992, 'n_words': 1320, 'path': 'Machine Learning/TensorflowBasic.ipynb', 'repo_name': 'arun-ramamurthy/crash-course', 'size': '15002'}