<jupyter_start><jupyter_text>Tensorflow Exercises: Linear Regression  Scope: Use Tensorflow 2.0 and pandas to build a simple linear regression model and then test the results on a hold-out set.<jupyter_code>import pandas as pd
import numpy as np
import tensorflow as tf
from sklearn.model_selection import train_test_split<jupyter_output><empty_output><jupyter_text>Simple OptimizationTo "warm-up", I'll use the tensorflow framework to solve a simple optimization problem. This will be done with the analytic gradient, and the autodifferentiation procedure which is standard in Tensorflow.First, define the function we will be using which is the Beale function:$$ f(x,y) = (1.5-x+xy)^2 + (2.25-x+xy^2)^2 + (2.625-x+xy^3)^2$$ The gradient is:$$  \nabla f(x,y) =  \begin{bmatrix}2 x (y^6 + y^4 - 2 y^3 - y^2 - 2 y + 3) + 5.25 y^3 + 4.5 y^2 + 3 y - 12.75 \\ 6 x (x (y^5 + 0.666667 y^3 - y^2 - 0.333333 y - 0.333333) + 2.625 y^2 + 1.5 y + 0.5)\end{bmatrix}  $$<jupyter_code>def f_beale(x):
    return (tf.convert_to_tensor( (1.5-x[0]+x[0]*x[1])**2 + (2.25-x[0]+x[0]*x[1]**2)**2 + (2.625-x[0]+x[0]*x[1]**3)**2) )

def grad_beal(x):
    dx = 2*x[0]*(x[1]**6 + x[1]**4 - 2*x[1]**3 - x[1]**2-2*x[1] + 3) + 5.25*x[1]**3+4.5*x[1]**2+3*x[1]-12.75
    dy = 6*x[0]*(x[0]*(x[1]**5 + 2.0/3.0 * x[1]**3-x[1]**2-1.0/3.0 * x[1]-1.0/3.0) + 2.625*x[1]**2+1.5*x[1]+.5)
    return(tf.convert_to_tensor([dx,dy]))

def AD_beale(x):
    x = tf.Variable(x,dtype = tf.float32) 
    with tf.GradientTape(persistent = True) as dv:
        temp_beale = f_beale(x)
    dx = dv.gradient(temp_beale, x)
    return(dx)<jupyter_output><empty_output><jupyter_text>We can see below that the results end up being the same for the derivatives:<jupyter_code>grad_beal([1,1]) == AD_beale([1,1])<jupyter_output><empty_output><jupyter_text>From here, we will attemp to solve the problem using stochastic gradient descent:<jupyter_code>opt = tf.keras.optimizers.SGD(learning_rate = .01)
#opt = tf.keras.optimizers.Adam(learning_rate = .01)
x_0 = tf.Variable([1.0,1.0],dtype = tf.float32)
func_for_opt = lambda: f_beale(x_0)

for epoch in range(1500):
    opt.minimize(func_for_opt, [x_0]).numpy()

x_0
x_vals = tf.Variable([1.0,2.0,3.0],name = "x_vals")
x_vals_two = tf.Variable([2.0,3.0,4.0],name = "x_vals_two")
with tf.GradientTape() as tape:
    y = x_vals[0]**3+x_vals[2]*x_vals[1]**2 + 2*x_vals[2]+100
    z_one = x_vals[1]*(4*(y) + x_vals[0]**2)    
    y_two = x_vals_two[0]**3+x_vals_two[2]*x_vals_two[1]**2 + 2*x_vals_two[2]+100
    z_two = x_vals_two[1]*(4*(y_two) + x_vals_two[0]**2)
    z = z_one + z_two
    
a, b = tape.gradient(z , [z,x_vals_two]) 
print(b)<jupyter_output>tf.Tensor([156. 900. 132.], shape=(3,), dtype=float32)<jupyter_text>Constrained Optimization We next solve a constrained optimization problem which is a constrained version of the Rosenbrock banana function:$$ \min_{x,y} f(x,y)  =  (1-x)^2 + 100(y-x^2)$$Subject to:$$ (x-1)^3 - y + 1 <=0 $$and:$$ x+y-2<=0 $$<jupyter_code>def f_rosenbrock(vals):
    return(tf.convert_to_tensor(  (1-vals[0])**2 + 100*(vals[1]-vals[0]**2)  ) )

def const1_rosenbrock(vals):    
    return(tf.convert_to_tensor( 1.0*((vals[0]-vals[1])**3 - vals[1] + 1)   ))

def const2_rosenbrock(vals):    
    return(tf.convert_to_tensor( 1.0*(vals[0]+vals[1]-2)   ))

def rosenbrock_lagrange(vals):
    return(  f_rosenbrock(vals)+const1_rosenbrock(vals)+const2_rosenbrock(vals)   )


f_rosenbrock([2.0,1.0])
opt = tf.keras.optimizers.SGD(learning_rate = 1e-7,momentum = .9)
#opt = tf.keras.optimizers.Adam(learning_rate = 5e-6)
x_0 = tf.Variable([.1,.1 ],dtype = tf.float32)
func_for_opt = lambda: tf.abs(f_rosenbrock(x_0))


#for epoch in range(1000):
i=1
while (func_for_opt().numpy()>1e-4)&(i < 60000):
#    print(i)
    if i%1000 == 0:
        print("At iteration " + str(i) +  " current results are " + str(x_0.numpy() )+ " with function value of "+str( func_for_opt().numpy())  + ".")
    opt.minimize(func_for_opt, [x_0]).numpy()
    i+=1

opt = tf.keras.optimizers.SGD(learning_rate = 1e-8,momentum = .9)
#opt = tf.keras.optimizers.Adam(learning_rate = 1e-6)
x_0 = tf.Variable([1.1,1.1  ],dtype = tf.float32)
func_for_opt = lambda: tf.abs(rosenbrock_lagrange(x_0))


#for epoch in range(1000):
i=1
while (func_for_opt().numpy()>1e-6)&(i < 100000):
#    print(i)
    if i%5000 == 0:
        print("At iteration " + str(i) +  " current results are " + str(x_0.numpy() )+ " with function value of "+str( func_for_opt().numpy())  + ".")
    opt.minimize(func_for_opt, [x_0]).numpy()
    i+=1

print("Final results at iteration " + str(i) +  " where the final results are " + str(x_0.numpy() )+ " with function value of "+str( func_for_opt().numpy())  + ".")<jupyter_output>At iteration 5000 current results are [1.0583223 1.119438 ] with function value of 0.0006868541.
At iteration 10000 current results are [1.0583223 1.1194379] with function value of 0.000674814.
At iteration 15000 current results are [1.0583256 1.1194365] with function value of 0.00016772747.
At iteration 20000 current results are [1.058328  1.1194353] with function value of 0.00078471005.
At iteration 25000 current results are [1.0583247 1.1194369] with function value of 6.9633126e-05.
At iteration 30000 current results are [1.0583297 1.1194346] with function value of 0.0012120008.
At iteration 35000 current results are [1.0583258 1.1194364] with function value of 0.00020335615.<jupyter_text>Load Data/Format Data The results above show the basic steps of working through a mathematical optimization problem. At this point, we pivot to the use of the mathematical optimization framework to solve a statistical problem. Simple linear regression uses the mean squared error function to find a linear relationship between a set of features and an outcome. This is defined as:$$ MSE(\beta) = \frac{\sum_{i=1}^N (y_i - \bar{\beta}X_i )^2 }{N}$$Here, we step through a simple example of linear regression using tensorflow and an implementation of the MSE function.The functions are defined in the code below:<jupyter_code>def MSE(y_val ,x_val,weights):
    output = tf.tensordot(X, weights, axes=1 ) 
    mse_val =  tf.reduce_mean( tf.square( output - y_val ) )
    return(mse_val)<jupyter_output><empty_output><jupyter_text>For this numerical example, we'll use the cars data-set and fit a few different variables to predict the gas mileage. First step: Load the data into the environment and form the design matrix:<jupyter_code>data = pd.read_csv( 'cars.csv' )
continuous_features = data[ [ "Identification.Year","Engine Information.Engine Statistics.Horsepower","Engine Information.Engine Statistics.Torque"] ].values / 100 
X = np.concatenate( [ continuous_features ] , axis=1 )
X = np.append(np.ones((X.shape[0],1) ) , X, axis=1)
Y = data[ [ 'Fuel Information.City mpg' ] ].values

# Perform basic subset selection in the code:
train_features , test_features ,train_labels, test_labels = train_test_split( X , Y , test_size=0.2 )
# Training data.
X = tf.Variable( train_features , dtype=tf.float32 )
Y = tf.Variable( train_labels , dtype=tf.float32 )                                                         
# Testing data
test_X = tf.Variable( test_features , dtype=tf.float32 ) 
test_Y = tf.Variable( test_labels , dtype=tf.float32 ) 
num_features = X.shape[1]
# Define the coefficientst that we'll be starting with:
#weights = tf.Variable(tf.random.normal((num_features,1)))
weights = tf.Variable(tf.ones(4))

def MSE(y_val ,x_val,weights):
    output = tf.tensordot(X, weights, axes=1 )
    mse_val =  tf.reduce_mean( tf.square( output - y_val ) )
    return(mse_val)

#mse_opt = tf.keras.optimizers.SGD(.001)
mse_opt = tf.keras.optimizers.Adam(.001)
weight_vals = weights
def temp_mse(weight_vals):
    return(MSE(Y,X,weight_vals))
func_for_opt = lambda: tf.abs(temp_mse(weight_vals))
mse_opt.minimize(func_for_opt, [weight_vals]).numpy()

i=1
while (temp_mse(weight_vals).numpy()>1e-2)&(i < 1000):
#    print(i)
    if i%100 == 0:
        print("At iteration " + str(i) +  " function value is "+str( func_for_opt().numpy())  + ".")
    mse_opt.minimize(func_for_opt, [weight_vals]).numpy()    
    i+=1
epochs_plot = list()
loss_plot = list()


# Define lagrangian here by taking the sum of the two functions.
def reg_lagrange( Y , y_pred_1, y_pred_2 ):
    return tf.add(tf.reduce_mean( tf.square( y_pred_1 - Y ) ) ,  tf.reduce_mean( tf.square( y_pred_2 - Y ))) 


for i in range( num_epochs ) :    
    epoch_loss = list()
    for b in range( int(num_samples/batch_size) ):
        x_batch , y_batch = iterator.get_next()
#output = h( x_batch , weights , bias ) 
        output_1 = h( test_X_one , weights_one , bias_1 )
        output_2 = h( test_X_one , weights_one , bias_2 )
        loss = epoch_loss.append( reg_lagrange( Y=y_batch , y_pred_1 = output_1, y_pred_2 = output_2 ).numpy() )
    
        output = tf.Variable( output, dtype=tf.float32 ) 
        y_batch = tf.Variable( y_batch, dtype=tf.float32 ) 
        with tf.GradientTape() as tape:
            mse_val = tf.reshape( tf.reduce_mean( tf.square( output - y_batch ) )  , [ 1 , 1 ] )   
    output_1
    
        with tf.GradientTape() as tape:
            ret_val = reg_lagrange( Y=y_batch , y_pred_1 = output_1, y_pred_2 = output_2 )
    
        dJ_dH = tape.gradient(mse_val,output)
        #dJ_dH = mean_squared_error_deriv( y_batch , output)
        dH_dW = x_batch
        dJ_dW = tf.reduce_mean( dJ_dH * dH_dW )
        dJ_dB = tf.reduce_mean( dJ_dH )
    
        weights -= ( learning_rate * dJ_dW )
        bias -= ( learning_rate * dJ_dB ) 
    loss = np.array( epoch_loss ).mean()
    epochs_plot.append( i + 1 )
    loss_plot.append( loss ) 
continuous_features = data[ [ "Identification.Year","Engine Information.Engine Statistics.Horsepower"] ].values / 100 
#categorical_research_features = data[ [ 'Research' ] ].values 
step1_vars = np.concatenate( [ continuous_features ] , axis=1 )
step2_vars = np.concatenate( [ data[ ["Engine Information.Engine Statistics.Torque"] ] ], axis=1 )
dep_var = data[ [ 'Fuel Information.City mpg' ] ].values
test_X_one = tf.constant( step1_vars , dtype=tf.float32 ) 

num_epochs = 10
num_samples = test_X_one.shape[0]
batch_size = 50
learning_rate = 0.001


test_X_two = tf.constant( step2_vars , dtype=tf.float32 ) 
test_Y = tf.constant( dep_var , dtype=tf.float32 )

dataset = tf.data.Dataset.from_tensor_slices(( test_X_one , test_Y )) 
dataset = dataset.shuffle( 500 ).repeat( num_epochs ).batch( batch_size )
iterator = dataset.__iter__()
num_features = test_X_one.shape[1]
weights = tf.random.normal( ( num_features , 1 ) )
bias = 0

epochs_plot = list()
loss_plot = list()

for i in range( num_epochs ) :    
    epoch_loss = list()
    for b in range( int(num_samples/batch_size) ):
        x_batch , y_batch = iterator.get_next()
   
        output = h( x_batch , weights , bias ) 
        loss = epoch_loss.append( mean_squared_error( y_batch , output ).numpy() )
    
        output = tf.Variable( output, dtype=tf.float32 ) 
        y_batch = tf.Variable( y_batch, dtype=tf.float32 ) 
        with tf.GradientTape() as tape:
            mse_val = tf.reshape( tf.reduce_mean( tf.square( output - y_batch ) )  , [ 1 , 1 ] )   
    
    
        dJ_dH = tape.gradient(mse_val,output)
        #dJ_dH = mean_squared_error_deriv( y_batch , output)
        dH_dW = x_batch
        dJ_dW = tf.reduce_mean( dJ_dH * dH_dW )
        dJ_dB = tf.reduce_mean( dJ_dH )
    
        weights -= ( learning_rate * dJ_dW )
        bias -= ( learning_rate * dJ_dB ) 
    loss = np.array( epoch_loss ).mean()
    epochs_plot.append( i + 1 )
    loss_plot.append( loss ) 
    
#    print( 'Loss is {}'.format( loss ) ) 

#Get predictions and then use them to create residuals.
preds = bias +  weights[0]*step1_vars[:,0]+  weights[1]*step1_vars[:,1]     
res_vals = preds-dep_var[:,0]        
hold = pd.DataFrame(step2_vars)
hold["res"] = np.array(res_vals).tolist()
test_X_two = tf.constant( np.concatenate( [ hold ], axis=1 ) , dtype=tf.float32 ) 
 
dataset_two = tf.data.Dataset.from_tensor_slices(( test_X_two , test_Y )) 
dataset_two = dataset_two.shuffle( 500 ).repeat( num_epochs ).batch( batch_size )
iterator = dataset_two.__iter__()
num_features = test_X_two.shape[1]
weights = tf.random.normal( ( num_features , 1 ) )
bias = 0
        
for i in range( num_epochs ) :    
    epoch_loss = list()
    for b in range( int(num_samples/batch_size) ):
        x_batch , y_batch = iterator.get_next()
   
        output = h( x_batch , weights , bias ) 
        loss = epoch_loss.append( mean_squared_error( y_batch , output ).numpy() )
    
        output = tf.Variable( output, dtype=tf.float32 ) 
        y_batch = tf.Variable( y_batch, dtype=tf.float32 ) 
        with tf.GradientTape() as tape:
            mse_val = tf.reshape( tf.reduce_mean( tf.square( output - y_batch ) )  , [ 1 , 1 ] )   
    
    
        dJ_dH = tape.gradient(mse_val,output)
        #dJ_dH = mean_squared_error_deriv( y_batch , output)
        dH_dW = x_batch
        dJ_dW = tf.reduce_mean( dJ_dH * dH_dW )
        dJ_dB = tf.reduce_mean( dJ_dH )
    
        weights -= ( learning_rate * dJ_dW )
        bias -= ( learning_rate * dJ_dB )      
        
        
        
    loss = np.array( epoch_loss ).mean()
    epochs_plot.append( i + 1 )
    loss_plot.append( loss ) 
    print( 'Loss is {}'.format( loss ) )<jupyter_output><empty_output><jupyter_text>The version below solves the problem as a lagrange optimization problem:$$ min(\hat{y_2}  - y)^2 + \lambda(\hat{y_1}  - \beta X)^2$$<jupyter_code>def reg_lagrange_problem( Y , y_one_pred, y_two_pred ):
    return tf.reduce_mean( tf.square( y_pred - Y ) ) + 
preds = bias +  weights[0]*step1_vars[:,0]+  weights[1]*step1_vars[:,1] 
res_vals = preds-dep_var[:,0]
res_vals = np.array(res_vals)
#res_vals.resize([5076,0])
res_vals
np.array(res_vals,step2_vars[:,0])
step2_vars[:,0].
type(data)
np.concatenate( [ data[ ["Engine Information.Engine Statistics.Torque"] ] ], axis=1 )
hold = pd.DataFrame(step2_vars)
hold["res"] = res_vals.tolist()
np.concatenate( [ hold ], axis=1 )
np.concatenate((step2_vars, res_vals), 1)
type(res_vals)
type(step2_vars)
np.hstack(res_vals,step2_vars)
def run_reg(indeps,deps,num_epochs = 10,batch_size = 50, learning_rate = .001):
    num_samples = indeps.shape[0]
    indeps = tf.constant( indeps , dtype=tf.float32 ) 
    deps = tf.constant( deps , dtype=tf.float32 ) 
    
    dataset = tf.data.Dataset.from_tensor_slices(( indeps , deps )) 
    dataset = dataset.shuffle( 500 ).repeat( num_epochs ).batch( batch_size )
    iterator = dataset.__iter__()
    num_features = indeps.shape[1]
    weights = tf.random.normal( ( num_features , 1 ) )
    bias = 0
    epochs_plot = list()
    loss_plot = list()
 

pred_vals = bias +  weights[0]*step1_vars[:,0]+  weights[1]*step1_vars[:,1] - dep_var
pred_vals - 
dep_var
bias
bias<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14332861857614332, 'fraction_numerical': 0.04963965784333536, 'mean_word_length': 3.6647816525290606, 'pattern_counts': {'":': 0, '<': 36, '<?xml version=': 0, '>': 34, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '485375', 'license': 'MIT', 'n_tokens_mistral': 6087, 'n_tokens_neox': 5671, 'n_words': 1256, 'path': 'Tensorflow Exercises Linear Regression.ipynb', 'repo_name': 'poc1673/ML-for-Networks', 'size': '14847'}