<jupyter_start><jupyter_text>Setup<jupyter_code>import matplotlib as mpl
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import tensorflow as tf

from mpl_toolkits.mplot3d import Axes3D # needed for project='3d'
import helpers_02

%matplotlib inline<jupyter_output><empty_output><jupyter_text>Basic NumPy version of Gradient Descent To get us started, let's investigate how gradient descent works.  We'll start by implementing it in NumPy. To keep things simple, we'll use a very basic cost function $J(x)=x^2$.<jupyter_code># Cost function: x^2
def J(x):
    return x * x

# Gradient of x^2 = 2*x
def J_grad(x):
    return 2 * x

# Creates datapoints to generate a gradient slope line
def grad_graph_helper(fn, mid, error, delta):
    x = np.linspace(mid-delta, mid+delta)
    y = error - (mid - x) * fn(mid)
    return x, y

# Plots x^2 with a given gradient slope and annotations 
def make_plot(ax, x, error, grad):
    grad_x, grad_y = grad_graph_helper(J_grad, x, error, 1)
    xs = np.linspace(-10, 10, 300)
    Js = J(xs)
    
    ax.set_xlabel('x'); ax.set_ylabel('J'); ax.grid(True)

    ax.plot(xs, Js)
    ax.plot(grad_x, grad_y, c='g', linewidth=2)
    ax.scatter(x, error, c='r', marker='s', s=50, zorder=6)
    
    if grad < 0:
        text_fill, x_add, error_add = ("< 0", "right"), 2, 10
    else:
        text_fill, x_add, error_add = ("> 0", "left"), -9, 10
    ax.annotate("Grad is {}; move to the {}".format(*text_fill),
                [x, error], [x+x_add, error+error_add],
                arrowprops={'arrowstyle': '->'})

def descend(x, learning_rate, steps=50, graph_steps=[0, 5, 15, 49]):
    x = float(x)
    fig, axes = plt.subplots(ncols=1, 
                             nrows=len(graph_steps), 
                             figsize=(12, 4*len(graph_steps)))
    axes = iter(axes)
    for step in range(steps):
        error, grad = J(x), J_grad(x)
        if step in graph_steps:
            ax = next(axes)
            ax.set_title('Step: {}'.format(step))
            make_plot(ax, x, error, grad)
        x -= learning_rate * grad
    plt.tight_layout()
# Starting from the left side
descend(-9.0, 0.05)
# Starting from the right side
descend(9.0, 0.05)
# What happens if learning rate is too high
descend(1.0, 1.3, 6, [2, 3, 4, 5])<jupyter_output><empty_output><jupyter_text>Gradient Descent with TensorFlow  Create some toy dataLet's create some data to learn from.  We're going to end up fitting a line to this data using linear regression.  The following code creates scatter data that is roughly along the line $y=3x+2$.  This is in the classing point-slope form of a line $y=mx+b$, but to prepare us for more general formulations, we'll write it as $y=wx+b$ ($w$ for *weight*).<jupyter_code>x = np.random.normal(loc=1, scale=1, size=[20])
epsilon = np.random.normal(loc=0, scale=1.5, size=[20])
y = (3 * x) + 2 + epsilon
plt.scatter(x,y,c='r');<jupyter_output><empty_output><jupyter_text>Now, we can create a simple linear model that can learn to fit the previous data.  You know the drill by now.  We will:  1.  Create the model  2.  Run the model  Create Graph for the Linear Regression Model<jupyter_code>linreg_graph = tf.Graph()
with linreg_graph.as_default():
    with tf.name_scope('inputs'):
        x_placeholder = tf.placeholder(tf.float32, [None], name='x')
        y_placeholder = tf.placeholder(tf.float32, [None], name='y')
        learning_rate = tf.placeholder(tf.float32, [], name='learning_rate')

    with tf.name_scope('model'):
        w = tf.Variable(tf.truncated_normal([]), name='w')
        b = tf.Variable(tf.constant(0, dtype=tf.float32), name='b')
        y_hat = tf.multiply(w, x_placeholder) + b
    
    with tf.name_scope('loss'):
        loss = tf.reduce_mean(tf.square(y_hat - y_placeholder), name='MSE')
    
    with tf.name_scope('train'):
        opt = tf.train.GradientDescentOptimizer # long name!
        train = opt(learning_rate).minimize(loss)
    
    with tf.name_scope('global_step'):
        global_step = tf.Variable(0, trainable=False, name='global_step')
        inc_step = tf.assign_add(global_step, 1, name='inc_step')
    
    with tf.name_scope('helpers'):
        init = tf.global_variables_initializer()<jupyter_output><empty_output><jupyter_text>Run the Model We start off by initializeing the Variables:<jupyter_code>session = tf.Session(graph=linreg_graph)
session.run(init)<jupyter_output><empty_output><jupyter_text>And now we setup a simple training and testing scenario with some model data.  I've recreated the simulated data below for easy reference.  `train_dict` and `test_dict` will be fed into `run` calls.  They tie our python level data to Tensors in our graph (e.g., (e.g., `x_train` to  `x_placeholder`).<jupyter_code># "true" model:  y = true_w * x + true_b
true_w, true_b = 3,2

# create training data and make a feeder dict
x_train = np.random.uniform(-5, 5, size=[20])
epsilon = np.random.normal(loc=0, scale=1.5, size=[20])
y_train = (true_w * x_train) + true_b + epsilon
train_dict = {x_placeholder: x_train,
              y_placeholder: y_train,
              learning_rate: 0.05}

# some points to evaluate at 
# also serve to reconstruct fitted line below
x_test = np.array([-5.0, 5.0])
test_dict = {x_placeholder: x_test}<jupyter_output><empty_output><jupyter_text>And finally, the "main" running of the model.  We have an loop that performs training every pass through.  Then, at selected steps (0, 1, 4, 10, 49), we also keep track of some results of testing on our evaluation `x_test` data.  In the `y_test=` line (line 10), we *do not* fetch the value of the `train` Operation.  Thus, we *do not* update the weights of our model.  Put another way, we only train when we fetch `train`.<jupyter_code>snapshots = []
print("{:2} {:^6s}".format("i", "MSE"))
for i in range(50):    
    J, w_est, b_est, step, _ = session.run([loss, w, b, inc_step, train], 
                                           feed_dict=train_dict)
    if not i % 5:
        print("{:<2d} {:>6.3f}".format(step, J))

    if step in [0, 1, 4, 10, 49]:
        y_test = session.run(y_hat, feed_dict=test_dict)
        snapshots.append((i, y_test, w_est, b_est, J))<jupyter_output><empty_output><jupyter_text>Visualize the Progression of the Estimates<jupyter_code># Get the final W and b values
final_w_est, final_b_est = session.run([w, b])

# extract out the sequences of w/b/J from the overall snapshots
# PYTHON TRICKERY:  zip(*seq) does the "opposite" of zip()
# here, it essentially pulls out the "columns" of snapshots
_, _, w_ests, b_ests, losses = zip(*snapshots)
for i, snapshot in enumerate(snapshots):
    # extract needed components for current snapshot
    step, y_test, curr_w_est, curr_b_est, curr_loss = snapshot

    # setup outer figure
    fig = plt.figure(figsize=(16,4))
    title_fmt = 'Step: {} error: {:0.4f} w = {:0.4f} b = {:0.4f}'
    title_string = title_fmt.format(step, curr_loss, 
                                    curr_w_est, curr_b_est)
    fig.suptitle(title_string, size=20)

    # Scatter plot of data with predicted line
    ax = plt.subplot(131)
    ax.set_xlabel('x'); ax.set_ylabel('y'); ax.grid(True)
    ax.scatter(x_train, y_train, c='r')
    ax.plot(x_test, y_test) # both curr train->test and estimated line

    # 3D view of estimated w and b values against loss
    ax = plt.subplot(132, projection='3d')
    ax.set_xlabel('w'); ax.set_ylabel('b'); ax.set_zlabel('loss')
    ax.plot(w_ests, b_ests, losses)
    ax.scatter(curr_w_est, curr_b_est, curr_loss, c='r')
    
    # Overhead view of approximate error curves
    ax = plt.subplot(133)
    ax.set_xlabel('w'); ax.set_ylabel('b'); ax.grid(True)
    for i in range(1,6):
        circ = mpl.patches.Circle([final_w_est, final_b_est], 
                                  .5 * i, fill=False)
        ax.add_patch(circ)
        
    ax.plot(w_ests, b_ests)                   # the path of w/b estimates
    ax.scatter(curr_w_est, curr_b_est, c='r') # really just one point
    ax.set_aspect('equal')
# scikit learn's estimates
import sklearn.linear_model as sk_lm
model = sk_lm.LinearRegression()
model.fit(np.reshape(x_train,[20,1]), y_train)
print('sk linreg w: {:5.3f} b: {:5.3f}'.format(model.coef_[0], 
                                                model.intercept_))

# compared with our estimates
print('our tflr  w: {:5.3f} b: {:5.3f}'.format(final_w_est, 
                                                final_b_est))<jupyter_output><empty_output><jupyter_text>Examining gradients The below code gets the gradient of the `tf.square()` Operation, which we expect to return `2*input`, as the derivative of $x^2$ is $2x$<jupyter_code>grad_graph = tf.Graph()
with grad_graph.as_default():
    # A dummy variable that we can play around with to get different values
    a = tf.Variable(3.0)

    # The operation we want to get the derivative of
    # b = a**2; deriv(b) = deriv(a**2) = 2a 
    # eval @ a=3 gives deriv(b wrt a) = 6
    b = tf.square(a) 

    # Create an Optimizer
    opt = tf.train.GradientDescentOptimizer(0.05)

    # Compute the gradient of `b` with respect to `a`
    grads = opt.compute_gradients(b, [a])
    init = tf.global_variables_initializer()

    
with tf.Session(graph=grad_graph) as session:
    session.run(init)
    print(session.run(grads))<jupyter_output><empty_output><jupyter_text>Some Realistic Data We'll use the UCI Housing dataset for this example, where the target is to predict the median value of homes in neighborhoods of Boston. The below code downloads the dataset and information files.<jupyter_code>data_dir = 'data'

base_url = 'http://archive.ics.uci.edu/ml/machine-learning-databases/'
data_suf = 'housing/housing.{}'
url = base_url + data_suf

data_path = helpers_02.download(url.format('data'), data_dir)
helpers_02.download(url.format('names'), data_dir)<jupyter_output><empty_output><jupyter_text>The variables in the dataset are defined in `data/housing.names`.  You can read through that file, but we've extracted the important parts here.1. **CRIM**: per capita crime rate by town2. **ZN**: proportion of residential land zoned for lots over 25,000 sq.ft.3. **INDUS**: proportion of non-retail business acres per town4. **CHAS**: Charles River dummy variable (= 1 if tract bounds river; 0 otherwise)5. **NOX**: nitric oxides concentration (parts per 10 million)6. **RM**: average number of rooms per dwelling7. **AGE**: proportion of owner-occupied units built prior to 19408. **DIS**: weighted distances to five Boston employment centres9. **RAD**: index of accessibility to radial highways10. **TAX**: full-value property-tax rate per \$10,00011. **PTRATIO**: pupil-teacher ratio by town12. **B**: 1000(Bk - 0.63)^2 where Bk is the proportion of blacks by town13. **LSTAT**: \% lower status of the population14. **MEDV**: Median value of owner-occupied homes in $1000'sNow, we read in the data using Pandas powerful `pd.read_table` function and label the columns with manually extracted column names.<jupyter_code># manually extracted from housing.names
housing_names = ('CRIM,ZN,INDUS,CHAS,NOX,'
                 'RM,AGE,DIS,RAD,TAX,'
                 'PTRATIO,B,LSTAT,MEDV').split(',')

my_data = pd.read_table(data_path, sep='\s+', names=housing_names)

# one way to shuffle the data
# data = data.reindex(np.random.permutation(data.index))
my_data.head()<jupyter_output><empty_output><jupyter_text>Here is a quick way to make a single train/test split of the overall DataFrame.  This has some benefits over scikit_learn's `train_test_split` because our method does not shuffle the data (skl shuffles the indices and uses fancy indexing to select the train and test sets - this triggers a copy).  The result is that our method doesn't require copies which might be important with bigger datasets.<jupyter_code># why not just use sklearn shuffle split?  
# skl uses fancy indexing which triggers copies
# these are continguous blocks, so there is no copy needed
def df_train_test_blocks(df, pct):
    num_train_examples = int(len(df) * pct)
    return df.iloc[:num_train_examples], df.iloc[num_train_examples:]
my_train, my_test = df_train_test_blocks(my_data, .8)
my_train.tail()<jupyter_output><empty_output><jupyter_text>Now that we have some more realistic data and we've created train/test subsets of the data, let's do some (linear regression) modeling!  Let's pick a few features (number of rooms, age, and tax rate) and (1) look at some descriptive statistics for those features and (2) look at the pairwise interactions between those features and median value with some scatter plots.  These graphs are only for the training data!<jupyter_code>my_ftrs = my_train[['MEDV', 'AGE', 'RM', 'TAX']]
my_ftrs.describe()

import seaborn as sns
sns.pairplot(my_ftrs, x_vars=['AGE', 'RM', 'TAX'], y_vars=['MEDV']);<jupyter_output><empty_output><jupyter_text>Linear Regression Models with `scikit_learn`<jupyter_code># models:  price ~ age, price ~ rooms, price ~ tax
# scatter, regression line, r^2
# more models:  saturated (all), price ~ rooms + tax<jupyter_output><empty_output><jupyter_text>Let's make three regression models between these variables (individually) and median price.  So, our three models will be (where $y \sim x$ is read as "y is modeled on x", this notation comes from R):   1. $price \sim  age$  1. $price \sim rooms$  3. $price \sim tax$  In turn, we'll use the training data to build each of these models (using scikit_learn) and then we'll graph the fit line against the training data.  We'll also compute the $R^2$ statistic for the fit against the training data ($R^2$ is SKL's default scorer for LinearRegression).<jupyter_code>model_on = ['AGE', 'RM', 'TAX']
models = {}
fig, axes = plt.subplots(3,1)
for ax, ftr in zip(axes, model_on):
    models[ftr] = sk_lm.LinearRegression().fit(my_ftrs[[ftr]], 
                                               my_ftrs[['MEDV']])
    ax.scatter(my_ftrs[[ftr]], my_ftrs[['MEDV']], c='r')

    x_vals = np.array(ax.get_xlim()) # great trick to get min/max x
    y_vals = models[ftr].intercept_ + models[ftr].coef_[0] * x_vals
    ax.plot(x_vals, y_vals, c='b')

    train_r_squared = models[ftr].score(my_ftrs[[ftr]], 
                                       my_ftrs[['MEDV']])
    
    ax.set_xlabel(ftr); ax.set_ylabel('MEDV')
    ax.set_title("MEDV ~ {}\nTrain $R^2$ = {:5.4f}".format(ftr, train_r_squared))

plt.tight_layout()<jupyter_output><empty_output><jupyter_text>Now, typically, we don't expect single variable to fully capture complex phenomena. So, let's build two more models:    1. $price \sim age + room + tax$  2. $price \sim room + tax$  Here, adding something to the right hand side of the model means it is an additional feature/variable in our model.<jupyter_code>models = (('Full', ['AGE', 'RM', 'TAX']),
          ('Room, Tax', ['RM', 'TAX']))
target = 'MEDV'

for model_name, predictors in models:
    fit = sk_lm.LinearRegression().fit(my_ftrs[predictors],
                                       my_ftrs[target])
    r2 = fit.score(my_ftrs[predictors], my_ftrs[target])
    print("{:10} Train R^2: {:5.4f}".format(model_name, r2))<jupyter_output><empty_output><jupyter_text>Housing Data Modeling with TensorFlow The room and tax model seems to do fairly well (at least compared to all three variables).  So, let's recreate that with TensorFlow.  We are going to add one layer of abstraction to our model.  You'll notice that the scikit_learn `Linear Regression` has three steps:  1.  create it  2.  fit it  3.  predict/score with it  These are very similar to the way we will typically use a TensorFlow model.  We've had two types of *running* our models that correspond to the fit and predict (train and test) steps for scikit learn.  The way we will implement this (and simplify the code that *uses* our model) is by create a simple `class` which has three methods:  1. `__init__`:  create the TensorFlow Graph  2. `fit`:  read in training data and optimize to it  3. `predict`:  read in testing data and give our best guesses (and possibly other results)<jupyter_code>class TF_GD_LinearRegression:
    # WARNING: 
    # s instead of self is non-standard, for abbreviation purposes
    def __init__(s):  
        graph = tf.Graph()
        with graph.as_default():
            with tf.name_scope('inputs'):
                s.x_placeholder = tf.placeholder(tf.float32, 
                                                 [None, 2], name='x')
                s.y_placeholder = tf.placeholder(tf.float32, 
                                                 [None], name='y')
                s.learning_rate = tf.placeholder(tf.float32, 
                                                 [], name='learning_rate')
            with tf.name_scope('model'):
                s.w = tf.Variable(tf.truncated_normal([2, 1]), name='w')
                s.b = tf.Variable(0.0, name='b')
                s.y_hat = tf.matmul(s.x_placeholder, s.w) + s.b
            
            with tf.name_scope('loss'):
                s.loss = tf.reduce_mean(tf.square(s.y_hat - s.y_placeholder), name='MSE')
            
            with tf.name_scope('train'):
                s.train = tf.train.GradientDescentOptimizer(s.learning_rate).minimize(s.loss)
            
            s.init = tf.global_variables_initializer()
        s.session = tf.Session(graph=graph)
        s.session.run(s.init)
    
    def fit(s, train_dict):
        return s.session.run([s.loss, s.w, s.b, s.train], feed_dict=train_dict)

    def predict(s, test_dict):
        return s.session.run(s.y_hat, feed_dict=test_dict)<jupyter_output><empty_output><jupyter_text>There are at least three major advantages to structuring our code in a class:  1.  We have clearly delineated what the capabilities of the code are:  creation, fit, and prediction.  2.  Instead of "polluting" our top-level code (our main program) with many model Tensors, they are all contained within the class.  If we *must* access them in the main program, we can always access them with `object.Tensor` (there's an example coming below when we create the training and testing dictionaries).  3.  Although we don't do it here, if we needed to make many "same but different" models, we could use the same structure to create the models and passing in parameters to control the differences.  Enough talk.  Let's do this thing.<jupyter_code># read in the data (just a reminder from what was above)
print("Data file:", data_path)
print("Names:", housing_names)
my_data = pd.read_table(data_path, sep='\s+', names=housing_names)

# select out some columns of interest and then train/test split it
my_ftrs = my_data[['MEDV', 'AGE', 'RM', 'TAX']]
my_train, my_test = df_train_test_blocks(my_ftrs, .8)

# our ftrs to use for this particular model
predictors = ['RM', 'TAX']
target     = 'MEDV' # output tensor (y_placeholder) expects 1D tensor, not 2D - so no list

# feeding data to TensorFlow, might not be happy with DataFrames
# so we access the underlying NumPy data directly with .values
my_train_x = my_train[predictors].values
my_train_y = my_train[target].values
my_test_x  = my_test[predictors].values
my_test_y  = my_test[target].values
model = TF_GD_LinearRegression()

# model.x_placeholder becomes s.x_placeholder in TF_GD_LinearRegression above
train_dict = {model.x_placeholder: my_train_x,
              model.y_placeholder: my_train_y,
              model.learning_rate: 0.000001}

test_dict = {model.x_placeholder: my_test_x}
print("{:3} {:>15s}".format("i", "Training MSE"))
snapshots = []

for epoch in range(1000):
    loss, w, b, _ = model.fit(train_dict)
    if epoch % 100 == 0 or epoch < 10:
        print("{:<3d} {:>15.3f}".format(epoch, loss))
        y_test = model.predict(test_dict)
        snapshots.append((epoch, y_test, w, b, loss))
print("{:<2d} {:>15.3f}".format(epoch, loss))<jupyter_output><empty_output><jupyter_text>Let's take the final trained model and use it to make predictions on our test set.  We'll then evaluate the quality of those predictions.  For ease of comparison, we'll compute the MSE (mean squared error) and use that as our metric.  You might like to play around with adding more iterations (epochs) to the training phase above and see how many iterations it takes to match scikit_learn's performance.  As an estimate, 100,000 epochs was getting closer to scikit learn's level, but didn't match it.<jupyter_code># final trained model
import sklearn.metrics as sk_metrics
y_test = model.predict(test_dict)
tf_roomtax_mse = sk_metrics.mean_squared_error(y_test[:,0], my_test_y)

print("tensorflow gd linear regression model")
print("price ~ room + tax")
print("w: {} b: {}".format(w[:,0],b))
print("Test MSE:", tf_roomtax_mse)
# and for comparison, with the scikit learn room+tax model
skl_fit = sk_lm.LinearRegression().fit(my_train_x, my_train_y)
skl_roomtax_mse = sk_metrics.mean_squared_error(skl_fit.predict(my_test_x), my_test_y)

print("sklearn linear regression model:")
print("price ~ room + tax")
print("w: {} b: {}".format(skl_fit.coef_, skl_fit.intercept_))
print("Test MSE: {}".format(skl_roomtax_mse))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11760242792109256, 'fraction_numerical': 0.015032245827010623, 'mean_word_length': 3.605590740336318, 'pattern_counts': {'":': 0, '<': 83, '<?xml version=': 0, '>': 84, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 22, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '266084', 'license': 'MIT', 'n_tokens_mistral': 7213, 'n_tokens_neox': 6865, 'n_words': 2202, 'path': 'Intel-TF101-Class2/Week2_Machine_Learning_Fundamentals_with_Tensorflow_HW.ipynb', 'repo_name': 'anishmo99/intel-TensorFlow-Deep-Learning', 'size': '21088'}