<jupyter_start><jupyter_text>Create an L-layer neural network (classification)Since the goal is to learn the details of neural networks, this model is implemented using numpy matrix operations. We also have added l2 and dropout regularization options. Model parameters* X = $n_x$ by m array where training vectors are in columns* Y = 1 by m array of true class values* $n_x$ = number of predictors* m = number of training examples* L = number of layers* n = list containing number of neurons per layer, n[0] = $n_x$* Wi = n[i] by n[i-1] array of model parameters for layer i* bi = n[i] by i array of bias parameters for layer i* dWi = n[i] by n[i-1] array of partial derivatives of the cost function with respect to the corresponding parameters of Wi* dbi = n[i] by i array of partial derivatives of the cost function with respect to the corresponding bias parameters* regularization = None or 'l2'* lambd = regularization parameter for l2<jupyter_code>import numpy as np
# activation function for output vector
def sigmoid(z):
    return 1/(1+np.exp(-z))
# activation function for hidden layers
def relu(z):
    return np.maximum(0, z)
# derivative of the relu function. We keep the shape of the input array for vectorization purposes. Strictly
# speaking, the total derivative should be a (much larger) diagonal matrix
def der_relu(z):
    return (z>0).astype(int)
def predict(X, parameters):
    '''Predict binary class value given input X and model parameters. The activation function
    for each hidden layer is a ReLU and the output activation function is a sigmoind. Since this 
    function is only used for model predictions, no intermediate values of the model are cached
    or returned to the user.
    
    Input:
        X = n_x by m array of feature vectors
        parameters = dictionary of parameters Wi and bi for the neural network
    
    Output:
        A = 1 by m array of predicted class values.'''
    
    # Calculate the number of layers in the network
    L = len(parameters)//2
    
    # hidden layer computations
    A = X
    for i in range(1, L):
        A = relu(np.dot(parameters['W'+str(i)], A) + parameters['b'+str(i)])
        
    # output layer
    A = sigmoid(np.dot(parameters['W'+str(L)], A) + parameters['b'+str(L)])
    
    # predict class
    A = (A >= 0.5)
    
    return A.astype(int)
def initialize(n):
    '''Initialize model parameters using the input specification. We use He initialization since we have ReLU
    activation functions.
    
    Input:
        n = length L+1 list where n[i] is the number of neurons in layer i
        
    Output:
        parameters = dictionary containing initialized model parameters Wi and bi'''
    
    parameters = dict()
    for i in range(1, len(n)):
        parameters['W'+str(i)] = np.random.randn(n[i], n[i-1])*(2/np.sqrt(n[i-1]))
        parameters['b'+str(i)] = np.zeros((n[i], 1))
        
    return parameters
def propogate(X, Y, parameters, regularization=None):
    '''Perform forward and backward propogation.
    
    Input:
        X = n_x by m array containing training examples
        Y = 1 by m array containing true binary class value for training examples
        parameters = dictionary of model parameters Wi and bi
        
    Output:
        gradient = dictionary of gradient arrays dWi and dbi corresponding to model parameters Wi and bi'''
    
    m = X.shape[1]    # infer the number of training examples from X
    L = len(parameters)//2    # infer the number of layers from the number of parameters
    sum_vec = np.full((m, 1), 1)    # vector used in gradient calculations
    cache = dict()    # cache for results of intermediate steps
    
    # forward propogation
    
    # hidden layer computations
    A = X    # A stores the activation of the previous layer, initializes to training vectors
    cache['A0'] = A
    for i in range(1, L):
        Z = np.dot(parameters['W'+str(i)], A) + parameters['b'+str(i)]    # linear step
        A = relu(Z)    # activation
        cache['Z'+str(i)] = Z    # store intermediate values for use in gradient calcululation
        cache['A'+str(i)] = A
        
    # output layer
    Z = np.dot(parameters['W'+str(L)], A) + parameters['b'+str(L)]    # linear step
    A = sigmoid(Z)    # activation
    cache['Z'+str(L)] = Z    # store intermediate values for use in gradient calculation
    cache['A'+str(L)] = A
    
    #backward propogation
    gradient = dict()
    
    # output layer
    dA = (1/m)*(A-Y)    # tracking matrix for chain rule
    gradient['dW'+str(L)] = np.dot(dA, cache['A'+str(L-1)].T)    # dWL
    gradient['db'+str(L)] = np.dot(dA, sum_vec)    # dbL
    
    # hidden layers
    for i in reversed(range(1, L)):
        dA = np.dot(dA.T, parameters['W'+str(i+1)])*der_relu(cache['Z'+str(i)].T)    # update dA. See page 3 of notes for derivation/proof
        dA = dA.T
        gradient['dW'+str(i)] = np.dot(dA, cache['A'+str(i-1)].T)    # dWi
        gradient['db'+str(i)] = np.dot(dA, sum_vec)    # dbi
    
    return gradient
def cost(Yhat, Y, eps=0.000000001, regularization=None, lambd=0, parameters=None):
    '''Calculate cost of prediction Yhat. The small value eps is used to prevent log(0) error.
    
    Input:
        Yhat = 1 by m array of class probabilities
        Y = 1 by m array of true binary class
        eps = small value to prevent log(0) error
        regularization = None or l2
        lambd = l2 regularization constant
        parameters = dictionary of model parameters
        
    Output:
        cost = cost of the given prediction Yhat'''
    
    m = Yhat.shape[1]    # infer the number of training examples
    
    if regularization == 'l2':
        L = len(parameters)//2   # infer the number of layers
        l2_sum = 0   # sum squares of parameters
        for i in range(1, L+1):
            l2_sum += np.sum(parameters['W'+str(i)]**2)
        return -(1/m)*np.sum(Y*np.log(Yhat + eps)+(1-Y)*np.log(1-Yhat + eps)) + (lambd/(2*m))*l2_sum
    else:
        return -(1/m)*np.sum(Y*np.log(Yhat + eps)+(1-Y)*np.log(1-Yhat + eps))
def fit(X, Y, parameters, learning_rate=0.01, iterations=2000, regularization=None, lambd=0, print_cost=False):
    
    L = len(parameters)//2
    m = X.shape[0]
    
    for i in range(iterations):
        gradient = propogate(X, Y, parameters, regularization=regularization)
        
        # print cost every 1000 iterations
        if (i%1000 == 0) and (print_cost == True):
            current_cost = cost(predict(X, parameters), Y, regularization=regularization, lambd=lambd, parameters=parameters)
            print('Cost after {} iterations: {:.12f}'.format(i, current_cost))
        
        # update parameters
        if regularization == 'l2':
            for i in range(1, L+1):
                parameters['W'+str(i)] = (1-lambd/m)*parameters['W'+str(i)] - learning_rate*gradient['dW'+str(i)]
                parameters['b'+str(i)] = parameters['b'+str(i)] - learning_rate*gradient['db'+str(i)]
        else:
            for i in range(1, L+1):
                parameters['W'+str(i)] = parameters['W'+str(i)] - learning_rate*gradient['dW'+str(i)]
                parameters['b'+str(i)] = parameters['b'+str(i)] - learning_rate*gradient['db'+str(i)]
            
    return parameters
def L_layer_model(X_train, Y_train, X_test, Y_test, n, learning_rate=0.01, iterations=2000, regularization=None, lambd=0, print_cost=False):
    
    # initialize model parameters
    parameters = initialize(n)
    
    # fit model
    parameters = fit(X_train, Y_train, parameters, learning_rate=learning_rate, iterations=iterations, regularization=regularization, lambd=lambd, print_cost=print_cost)
    
    train_predictions = predict(X_train, parameters)
    train_accuracy = 100-np.average(np.abs(Y_train-train_predictions))*100
    
    test_predictions = predict(X_test, parameters)
    test_accuracy = 100-np.average(np.abs(Y_test-test_predictions))*100
    
    print('Training set accuracy: {:.4f}%'.format(train_accuracy))
    print('Test set accuracy: {:.4f}%'.format(test_accuracy))
    
    return parameters<jupyter_output><empty_output><jupyter_text>Test the functions with a random input<jupyter_code># Test
X = np.random.randn(4,5)
Y = np.array([[1, 1, 0, 0, 1]])
n = [4, 6, 5, 4, 3, 7, 2, 1]
parameters = initialize(n)
relu(X)
sigmoid(X)
predict(X, parameters)
predict?
gradient = propogate(X, Y, parameters)
parameters['b4'] - 0.001*gradient['db4']
parameters = fit(X, Y, parameters, regularization='l2', lambd=0.01)
Yhat = predict(X, parameters)
cost(Yhat, Y, regularization='l2', lambd=0.01, parameters=parameters)
parameters = L_layer_model(X, Y, X, Y, n, learning_rate=0.5, iterations=20000, regularization='l2', lambd=0.1)
parameters['W1']<jupyter_output><empty_output><jupyter_text>Build an 'or' networkTest the L layer model by building a simple network to represent an or gate.<jupyter_code>X = np.array([[1, 1, 0, 0], [1, 0, 1, 0]])
Y = np.array([[1, 1, 1, 0]])
n = [2, 4, 1]
# model with single hidden layer of four neurons
parameters = L_layer_model(X, Y, X, Y, n, learning_rate=0.9, iterations=2000, regularization='l2', lambd=0.1, print_cost=True)
# add a second hidden layer
n = [2, 4, 2, 1]
parameters = L_layer_model(X, Y, X, Y, n, learning_rate=0.1, iterations=20000, print_cost=False)
# Simple networks are best in this situation
n = [2, 2, 1]
parameters = L_layer_model(X, Y, X, Y, n, learning_rate=0.1, iterations=20000, regularization='l2', lambd=0.01)
predict(np.array([[1], [0]]), parameters)<jupyter_output><empty_output><jupyter_text>After running the above tests successfully, it appears that the L layer model is functional.  Test the L layer model on interview data setWe import an interview data set to investigate the performance of an L layer neural network on a larger data set with more predictors.<jupyter_code>from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
import pandas as pd
train_location = '' # insert file path here

train_df = pd.read_csv(train_location)
# Clean data. Details can be found in other notebook
train_df = train_df.dropna()

# Correct days so that all are spelled out
train_df['x35'] = train_df['x35'].map(lambda x: 'wednesday' if x=='wed' else 'thursday' if (x=='thur' or x=='thurday') else 'friday' if x=='fri' else x)

# Correct sept. to Sept and Dev to Dec in column x68
train_df['x68'] = train_df['x68'].map(lambda x: 'Jan' if x=='January' else 'Sept' if x=='sept.' else 'Dec' if x=='Dev' else x)

# Transform columns x34, x35, x68, and x93 to dummy variables
train_df = pd.get_dummies(train_df, columns=['x34', 'x35', 'x68', 'x93'])

# Transform columns x41 and x45 to floats
train_df['x41'] = train_df['x41'].map(lambda x: x.lstrip('$'))
train_df['x41'] = pd.to_numeric(train_df['x41'])

train_df['x45'] = train_df['x45'].map(lambda x: x.rstrip('%'))
train_df['x45'] = pd.to_numeric(train_df['x45'])

# Take the first 1000 samples for some quick tests
train_df = train_df.iloc[:1000,:]
train_df.shape<jupyter_output><empty_output><jupyter_text>Note that there are 127 data columns in the data set. The column labeled 'y' contains the true binary class of each training example.<jupyter_code># split data into train/test sets
X_train, X_test, Y_train, Y_test = train_test_split(train_df.drop('y', axis=1), (train_df['y']), test_size=0.2, random_state=42)

# scale data using the standard scaler in sklearn
scaler = StandardScaler()

X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)

# Get numpy columns for Y
Y_train = Y_train.values.reshape(len(Y_train),1)
Y_test = Y_test.values.reshape(len(Y_test),1)
X_train.shape<jupyter_output><empty_output><jupyter_text>Run model with one hidden layer of three neurons.<jupyter_code># note that samples are in rows of train_df, so the transposes are fed into the models
params = L_layer_model(X_train.T, Y_train.T, X_test.T, Y_test.T, n=[126, 3, 1], learning_rate=0.1, iterations=30000, regularization='l2', lambd=0.01, print_cost=True)<jupyter_output>Cost after 0 iterations: 17.174500831276
Cost after 1000 iterations: 0.725473019850
Cost after 2000 iterations: 0.621933756145
Cost after 3000 iterations: 0.596062268569
Cost after 4000 iterations: 0.596077053197
Cost after 5000 iterations: 0.596083724827
Cost after 6000 iterations: 0.596086945799
Cost after 7000 iterations: 0.596087305082
Cost after 8000 iterations: 0.596086729450
Cost after 9000 iterations: 0.440663387838
Cost after 10000 iterations: 0.440663100800
Cost after 11000 iterations: 0.414758287677
Cost after 12000 iterations: 0.414757619184
Cost after 13000 iterations: 0.388852851078
Cost after 14000 iterations: 0.337044893018
Cost after 15000 iterations: 0.311142009118
Cost after 16000 iterations: 0.311142667390
Cost after 17000 iterations: 0.311142842948
Cost after 18000 iterations: 0.311142726611
Cost after 19000 iterations: 0.311142472852
Cost after 20000 iterations: 0.311142135066
Cost after 21000 iterations: 0.311141810185
Cost after 22000 iterations: 0.311141524736
[...]<jupyter_text>Results* Training accuracy = 98.5%* Test accuracy = 84.5%We appear to be overfitting the model to the data. We can try increasing the regularization to prevent this problem.<jupyter_code>params = L_layer_model(X_train.T, Y_train.T, X_test.T, Y_test.T, n=[126, 3, 1], learning_rate=0.1, iterations=30000, regularization='l2', lambd=0.5, print_cost=True)<jupyter_output>Cost after 0 iterations: 5.470268891816
Cost after 1000 iterations: 1.271012331030
Cost after 2000 iterations: 1.141639926927
Cost after 3000 iterations: 0.960348981523
Cost after 4000 iterations: 1.012160190017
Cost after 5000 iterations: 1.012166777467
Cost after 6000 iterations: 1.012167700083
Cost after 7000 iterations: 1.038072339403
Cost after 8000 iterations: 1.038065651807
Cost after 9000 iterations: 1.038066926702
Cost after 10000 iterations: 1.038067147211
Cost after 11000 iterations: 1.038067082628
Cost after 12000 iterations: 1.038067374484
Cost after 13000 iterations: 1.038067305621
Cost after 14000 iterations: 1.038067353607
Cost after 15000 iterations: 1.038067585201
Cost after 16000 iterations: 1.038067236351
Cost after 17000 iterations: 1.038067399130
Cost after 18000 iterations: 1.038067140730
Cost after 19000 iterations: 1.038067292885
Cost after 20000 iterations: 1.038067564015
Cost after 21000 iterations: 1.038067381729
Cost after 22000 iterations: 1.038067547840
C[...]<jupyter_text>Results indicate that increasing the value of lambd reduces the amount of variance.<jupyter_code>params = L_layer_model(X_train.T, Y_train.T, X_test.T, Y_test.T, n=[126, 3, 1], learning_rate=0.1, iterations=30000, regularization='l2', lambd=0.6, print_cost=True)<jupyter_output>Cost after 0 iterations: 8.218089347854
Cost after 1000 iterations: 1.478261382085
Cost after 2000 iterations: 1.581859493972
Cost after 3000 iterations: 1.581828718192
Cost after 4000 iterations: 1.581813130745
Cost after 5000 iterations: 1.581816937729
Cost after 6000 iterations: 1.581815329419
Cost after 7000 iterations: 1.581811210604
Cost after 8000 iterations: 1.581811317183
Cost after 9000 iterations: 1.581811568895
Cost after 10000 iterations: 1.581811607957
Cost after 11000 iterations: 1.555907457306
Cost after 12000 iterations: 1.555907716240
Cost after 13000 iterations: 1.581812039888
Cost after 14000 iterations: 1.581811839860
Cost after 15000 iterations: 1.555907672412
Cost after 16000 iterations: 1.555907825443
Cost after 17000 iterations: 1.555907793317
Cost after 18000 iterations: 1.555907786853
Cost after 19000 iterations: 1.555907804617
Cost after 20000 iterations: 1.581811992176
Cost after 21000 iterations: 1.555907859592
Cost after 22000 iterations: 1.581811751734
C[...]<jupyter_text>Results indicate that increasing the value of lambd further does not necessarily improve performance any more.We can try models with more hidden layers/neurons.<jupyter_code>params = L_layer_model(X_train.T, Y_train.T, X_test.T, Y_test.T, n=[126, 6, 3, 1], learning_rate=0.1, iterations=30000)
params = L_layer_model(X_train.T, Y_train.T, X_test.T, Y_test.T, n=[126, 63, 3, 1], learning_rate=0.1, iterations=30000)
params = L_layer_model(X_train.T, Y_train.T, X_test.T, Y_test.T, n=[126, 63, 3, 1], learning_rate=0.1, iterations=30000, regularization='l2', lambd=0.05)<jupyter_output>Training set accuracy: 100.0000%
Test set accuracy: 89.5000%<jupyter_text>Let's compare the neural networks to simple logistic regression.<jupyter_code>params = L_layer_model(X_train.T, Y_train.T, X_test.T, Y_test.T, n=[126, 1], learning_rate=0.01, iterations=30000)
params = L_layer_model(X_train.T, Y_train.T, X_test.T, Y_test.T, n=[126, 1], learning_rate=0.1, iterations=30000)
params = L_layer_model(X_train.T, Y_train.T, X_test.T, Y_test.T, n=[126, 1], learning_rate=1, iterations=30000)
params = L_layer_model(X_train.T, Y_train.T, X_test.T, Y_test.T, n=[126, 1], learning_rate=0.005, iterations=30000)<jupyter_output>Training set accuracy: 91.6250%
Test set accuracy: 90.5000%<jupyter_text>These results indicate that neural networks don't have performance gains over simple logistic regression when the amount of available data is small. Increase the amount of data to 10000 samples to evaluate performance.<jupyter_code>train_location = '/Users/connorodell/Documents/Data_Science/learning/exercise_03_train.csv'

train_df = pd.read_csv(train_location)
# Clean data. Details can be found in other notebook
train_df = train_df.dropna()

# Correct days so that all are spelled out
train_df['x35'] = train_df['x35'].map(lambda x: 'wednesday' if x=='wed' else 'thursday' if (x=='thur' or x=='thurday') else 'friday' if x=='fri' else x)

# Correct sept. to Sept and Dev to Dec in column x68
train_df['x68'] = train_df['x68'].map(lambda x: 'Jan' if x=='January' else 'Sept' if x=='sept.' else 'Dec' if x=='Dev' else x)

# Transform columns x34, x35, x68, and x93 to dummy variables
train_df = pd.get_dummies(train_df, columns=['x34', 'x35', 'x68', 'x93'])

# Transform columns x41 and x45 to floats
train_df['x41'] = train_df['x41'].map(lambda x: x.lstrip('$'))
train_df['x41'] = pd.to_numeric(train_df['x41'])

train_df['x45'] = train_df['x45'].map(lambda x: x.rstrip('%'))
train_df['x45'] = pd.to_numeric(train_df['x45'])

train_df = train_df.iloc[:10000,:]
train_df.shape
# split data into train/test sets
X_train, X_test, Y_train, Y_test = train_test_split(train_df.drop('y', axis=1), (train_df['y']), test_size=0.2, random_state=42)

# scale data using the standard scaler in sklearn
scaler = StandardScaler()

X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)

# Get numpy columns for Y
Y_train = Y_train.values.reshape(len(Y_train),1)
Y_test = Y_test.values.reshape(len(Y_test),1)
X_train.shape
params = L_layer_model(X_train.T, Y_train.T, X_test.T, Y_test.T, n=[126, 63, 3, 1], learning_rate=0.1, iterations=30000, regularization='l2', lambd=0.01)
params = L_layer_model(X_train.T, Y_train.T, X_test.T, Y_test.T, n=[126, 63, 3, 1], learning_rate=0.1, iterations=200000)<jupyter_output>Model accuracy: 97.2000%<jupyter_text>With more data, test accuracy has improved on the order of 10%.We can see if the logistic regression model has similary performance gains with more data.<jupyter_code>params = L_layer_model(X_train.T, Y_train.T, X_test.T, Y_test.T, n=[126, 1], learning_rate=0.1, iterations=30000)<jupyter_output>Training set accuracy: 89.0625%
Test set accuracy: 89.1500%<jupyter_text>Results indicate that more data does not improve the logistic regression model. We can interpret this to mean that the model has too much bias. Now we test the models with the full data set.<jupyter_code>train_location = '/Users/connorodell/Documents/Data_Science/learning/exercise_03_train.csv'

train_df = pd.read_csv(train_location)
# Clean data. Details can be found in other notebook
train_df = train_df.dropna()

# Correct days so that all are spelled out
train_df['x35'] = train_df['x35'].map(lambda x: 'wednesday' if x=='wed' else 'thursday' if (x=='thur' or x=='thurday') else 'friday' if x=='fri' else x)

# Correct sept. to Sept and Dev to Dec in column x68
train_df['x68'] = train_df['x68'].map(lambda x: 'Jan' if x=='January' else 'Sept' if x=='sept.' else 'Dec' if x=='Dev' else x)

# Transform columns x34, x35, x68, and x93 to dummy variables
train_df = pd.get_dummies(train_df, columns=['x34', 'x35', 'x68', 'x93'])

# Transform columns x41 and x45 to floats
train_df['x41'] = train_df['x41'].map(lambda x: x.lstrip('$'))
train_df['x41'] = pd.to_numeric(train_df['x41'])

train_df['x45'] = train_df['x45'].map(lambda x: x.rstrip('%'))
train_df['x45'] = pd.to_numeric(train_df['x45'])

#train_df = train_df.iloc[:10000,:]
#train_df.shape
# split data into train/test sets
X_train, X_test, Y_train, Y_test = train_test_split(train_df.drop('y', axis=1), (train_df['y']), test_size=0.2, random_state=42)

# scale data using the standard scaler in sklearn
scaler = StandardScaler()

X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)

# Get numpy columns for Y
Y_train = Y_train.values.reshape(len(Y_train),1)
Y_test = Y_test.values.reshape(len(Y_test),1)<jupyter_output><empty_output><jupyter_text>Test various hyperparameters to evaluate relative performance.<jupyter_code>params = L_layer_model(X_train.T, Y_train.T, X_test.T, Y_test.T, n=[126, 63, 3, 1], learning_rate=0.1, iterations=2000, regularization='l2', lambd=0.01, print_cost=True)
params = L_layer_model(X_train.T, Y_train.T, X_test.T, Y_test.T, n=[126, 63, 6, 1], learning_rate=0.1, iterations=2000, print_cost=True)
params = L_layer_model(X_train.T, Y_train.T, X_test.T, Y_test.T, n=[126, 63, 6, 1], learning_rate=0.1, iterations=30000, regularization='l2', lambd=0.01)
params = L_layer_model(X_train.T, Y_train.T, X_test.T, Y_test.T, n=[126, 63, 10, 10, 1], learning_rate=0.1, iterations=30000, regularization='l2', lambd=0.01)
params = L_layer_model(X_train.T, Y_train.T, X_test.T, Y_test.T, n=[126, 63, 6, 1], learning_rate=0.1, iterations=30000, regularization='l2', lambd=0.05)
params = L_layer_model(X_train.T, Y_train.T, X_test.T, Y_test.T, n=[126, 63, 6, 1], learning_rate=0.1, iterations=30000, regularization='l2', lambd=0.05)
params = L_layer_model(X_train.T, Y_train.T, X_test.T, Y_test.T, n=[126, 63, 6, 1], learning_rate=0.1, iterations=30000, regularization='l2', lambd=0.05)
params = L_layer_model(X_train.T, Y_train.T, X_test.T, Y_test.T, n=[126, 63, 6, 1], learning_rate=0.1, iterations=30000, regularization='l2', lambd=0.06)
params = L_layer_model(X_train.T, Y_train.T, X_test.T, Y_test.T, n=[126, 63, 6, 1], learning_rate=0.1, iterations=30000, regularization='l2', lambd=0.07)
params = L_layer_model(X_train.T, Y_train.T, X_test.T, Y_test.T, n=[126, 63, 6, 1], learning_rate=0.1, iterations=30000, regularization='l2', lambd=0.08)
params = L_layer_model(X_train.T, Y_train.T, X_test.T, Y_test.T, n=[126, 63, 6, 1], learning_rate=0.1, iterations=30000, regularization='l2', lambd=0.09)
params = L_layer_model(X_train.T, Y_train.T, X_test.T, Y_test.T, n=[126, 63, 6, 1], learning_rate=0.1, iterations=30000, regularization='l2', lambd=0.1)<jupyter_output>Cost after 0 iterations: 11.098705361369
Cost after 1000 iterations: 0.654147662794
Cost after 2000 iterations: 0.175100131228
Cost after 3000 iterations: 0.136776328702
Cost after 4000 iterations: 0.134133307839
Cost after 5000 iterations: 0.129508021327
Cost after 6000 iterations: 0.129508021327
Cost after 7000 iterations: 0.128847266111
Cost after 8000 iterations: 0.129508021327
Cost after 9000 iterations: 0.128186510895
Cost after 10000 iterations: 0.127525755679
Cost after 11000 iterations: 0.127525755679
Cost after 12000 iterations: 0.127525755679
Cost after 13000 iterations: 0.126204245247
Cost after 14000 iterations: 0.126204245247
Cost after 15000 iterations: 0.125543490031
Cost after 16000 iterations: 0.124882734815
Cost after 17000 iterations: 0.125543490031
Cost after 18000 iterations: 0.125543490031
Cost after 19000 iterations: 0.124882734815
Cost after 20000 iterations: 0.125543490031
Cost after 21000 iterations: 0.124882734815
Cost after 22000 iterations: 0.126204245247
[...]<jupyter_text>Compare to simple linear regression on the full data set.<jupyter_code>params = L_layer_model(X_train.T, Y_train.T, X_test.T, Y_test.T, n=[126, 1], learning_rate=0.1, iterations=30000)<jupyter_output>Training set accuracy: 89.0285%
Test set accuracy: 89.1468%
{'fraction_non_alphanumeric': 0.1200328070535165, 'fraction_numerical': 0.10133278654910806, 'mean_word_length': 4.700327255726975, 'pattern_counts': {'":': 0, '<': 52, '<?xml version=': 0, '>': 54, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '490636', 'license': 'MIT', 'n_tokens_mistral': 10213, 'n_tokens_neox': 8338, 'n_words': 2497, 'path': 'DeepLearning/L_layer_network.ipynb', 'repo_name': 'connorodell/learning', 'size': '24385'}