<jupyter_start><jupyter_text>Logistic Regression Using Gradient DescentGoal: Implement a simple logistic regression model using gradient descent and forward/backward propogation ideas from neural networks. The basic idea follows from the Deep Learning course by deeplearning.ai, but the propogation functions have been reorganized by myself to match concepts from vector calculus more closely. In particular, the total derivative as a matrix and the chain rule as matrix multiplication.<jupyter_code>import numpy as np
import time<jupyter_output><empty_output><jupyter_text>Before implementing the model, I want to answer the following**Question**: Is it faster to use a sum or a dot product to find the sum of the elements in a vector?<jupyter_code>v = np.random.rand(1000,1)
u = np.full((1,1000), 1)

tic = time.time()
total = np.dot(u, v)
toc = time.time()
print(total[0,0])
print('Dot product sum: {} ms'.format((toc-tic)*1000))

tic = time.time()
total = np.sum(v, axis=0)
toc = time.time()
print(total[0])
print('Numpy sum: {} ms'.format((toc-tic)*1000))<jupyter_output>505.70772442755737
Dot product sum: 0.0820159912109375 ms
505.70772442755737
Numpy sum: 0.09202957153320312 ms<jupyter_text>After running the above cell several times, it appears that the dot product is usually faster. Sometimes the np.sum method is faster, so let's find an average.<jupyter_code>iterations = 10000

dot_total = 0
sum_total = 0

for i in range(iterations):
    v = np.random.rand(1000,1)
    u = np.full((1,1000), 1)
    
    tic = time.time()
    total = np.dot(u, v)
    toc = time.time()
    dot_total += toc-tic

    tic = time.time()
    total = np.sum(v, axis=0)
    toc = time.time()
    sum_total += toc-tic
    
print('Dot product elapsed time: {} ms'.format(1000*dot_total))
print('Numpy sum elapsed time: {} ms'.format(1000*sum_total))<jupyter_output>Dot product elapsed time: 29.576539993286133 ms
Numpy sum elapsed time: 58.959245681762695 ms<jupyter_text>Based on the above test, it appears that the dot product is, on average, about twice as fast as the numpy sum.I will use a dot product in my backwards propogation algorithm.<jupyter_code>def sigmoid(x):
    '''Sigmoid/logistic function.
    
        x = numpy array or float'''
    
    return 1/(1+np.exp(-x))
def logloss(y, yhat):
    '''Loss function for logistic regression.
    
        y = true (binary) class
        yhat = predicted class probability'''
    
    return -(y*np.log(yhat)+(1-y)*np.log(1-yhat))
def propogate(X, Y, w, b):
    '''Implement forward and backward propogation for simple logistic regression.
    
        X = n_x by m numpy array containg training data where n_x is the number of features and m is the number of samples
        Y = m by 1 numpy array containing true sample classes
        w = m by 1 numpy array of model parameters
        b = model bias (float)'''
    
    m = X.shape[1]    # number of training samples
    n_x = X.shape[0]    # number of features
    
    w.reshape(n_x,1)    # force w to be a column vector
    
    # Forward propogation
    A = sigmoid(np.dot(X.T,w)+b)
    
    # Backward propogation
    dw = (1/m)*np.dot(np.full((1,m),1), (A-Y)*X.T)
    db = (1/m)*np.sum(A-Y)
    
    # Cost for current parameter values
    cost = (1/m)*np.sum(logloss(Y, A))
    
    gradients = dict({'dw': dw, 'db': db})
    
    return gradients, cost<jupyter_output><empty_output><jupyter_text>Create a fitting function to loop through forward and backward propogation to fit model parameters via gradient descent.<jupyter_code>def fit(X, Y, w, b, iterations=2000, learning_rate=0.01):
    '''Implement gradient descent fitting procedure.
    
        X = n_x by m array with samples in columns
        Y = 1 by m array of true binary class values
        w = n_x by 1 array of model parameters
        b = model bias'''
    
    n_x = X.shape[0]    # number of features
    m = X.shape[1]    # number of training samples
    
    # gradient descent loop
    for i in range(iterations):
        gradient, cost = propogate(X, Y, w, b)
        dw = gradient['dw']
        db = gradient['db']
        w = w - learning_rate*dw.reshape(n_x,1)
        b = b - learning_rate*db
    
    parameters = dict({'w': w, 'b': b})
    return parameters<jupyter_output><empty_output><jupyter_text>Function to initialize model parameters to zero.<jupyter_code>def initialize(X):
    '''Initialize logistic regression parameters to zero.
    
        X = n_x by m array where n_x is the number of model features
        
    Returns a numpy array w of zeros in the shape n_x by 1 and a scaler b=0.'''
    
    n_x = X.shape[0]
    w = np.zeros((n_x,1))
    b = 0
    
    return w, b<jupyter_output><empty_output><jupyter_text>Function to predict binary class values using logistic regression model.<jupyter_code>def class_val(x):
    '''Find the most likely class given the input probability x.'''
    if x <= 0.5:
        return 0
    else:
        return 1

# use numpy vectorization to apply class_val to an array
vclass_val = np.vectorize(class_val)
def predict(X, w, b):
    '''Predict class using logistic regression model.
    
        X = array with n_x rows, samples in columns'''
    
    A = sigmoid(np.dot(X.T, w)+b)
    
    return vclass_val(A)<jupyter_output><empty_output><jupyter_text>Merge functions into one that both trains and tests a logistic model.<jupyter_code>def LogModel(X_train, Y_train, X_test, Y_test, iterations=2000, learning_rate=0.01):
    '''Build logistic regression model using training data and test on provided
    testing data.
    
        X_train = n_x by m array with feature vectors in columns
        Y_train = m by 1 array of true binary class for each training sample
        X_test = n_x by any number array with feature vectors in columns
        Y_test = column vector of true binary class for each test sample
        iterations = positive integer, number of iterations for gradient descent
        learning_rate = positive float, learning rate used for gradient descent
        
    Prints percentage of correct predictions on testing data using the model fit by gradient descent.
    Returns model parameters.'''
    
    w, b = initialize(X_train)
    
    parameters = fit(X_train, Y_train, w, b, iterations=iterations, learning_rate=learning_rate)
    w = parameters['w']
    b = parameters['b']
    
    predictions = predict(X_test, w, b)
    
    model_accuracy = 100-np.average(np.abs(Y_test-predictions))*100
    
    print('Model accuracy: {:.4f}%'.format(model_accuracy))
    
    return parameters<jupyter_output><empty_output><jupyter_text>Small test to see if the function works.<jupyter_code>X_train, Y_train, X_test, Y_test = np.array([[1.,2.,-1.],[3.,4.,-3.2]]), np.array([[1],[0],[1]]), np.array([[1.,2.,-1.],[3.,4.,-3.2]]), np.array([[1],[0],[1]])

LogModel(X_train, Y_train, X_test, Y_test)<jupyter_output>Model accuracy: 100.0000%<jupyter_text>Now we import a larger data set from an insurance company project to test the model.<jupyter_code>from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
import pandas as pd
train_location = '/Users/connorodell/Documents/Data_Science/learning/exercise_03_train.csv'

train_df = pd.read_csv(train_location)
# Clean data. Details can be found in other notebook
train_df = train_df.dropna()

# Correct days so that all are spelled out
train_df['x35'] = train_df['x35'].map(lambda x: 'wednesday' if x=='wed' else 'thursday' if (x=='thur' or x=='thurday') else 'friday' if x=='fri' else x)

# Correct sept. to Sept and Dev to Dec in column x68
train_df['x68'] = train_df['x68'].map(lambda x: 'Jan' if x=='January' else 'Sept' if x=='sept.' else 'Dec' if x=='Dev' else x)

# Transform columns x34, x35, x68, and x93 to dummy variables
train_df = pd.get_dummies(train_df, columns=['x34', 'x35', 'x68', 'x93'])

# Transform columns x41 and x45 to floats
train_df['x41'] = train_df['x41'].map(lambda x: x.lstrip('$'))
train_df['x41'] = pd.to_numeric(train_df['x41'])

train_df['x45'] = train_df['x45'].map(lambda x: x.rstrip('%'))
train_df['x45'] = pd.to_numeric(train_df['x45'])
# split data into train/test sets
X_train, X_test, Y_train, Y_test = train_test_split(train_df.drop('y', axis=1), train_df['y'], test_size=0.2, random_state=42)

# scale data using the standard scaler in sklearn
scaler = StandardScaler()

X_train = scaler.fit_transform(X_train)
X_test = scaler.fit_transform(X_test)

# Get numpy columns for Y
Y_train = Y_train.values.reshape(len(Y_train),1)
Y_test = Y_test.values.reshape(len(Y_test),1)
# run new model
# note that samples are in rows of X_train and X_test, so the transposes are fed into the models
params = LogModel(X_train.T, Y_train, X_test.T, Y_test)
params<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1114783996352445, 'fraction_numerical': 0.03248603670352217, 'mean_word_length': 3.732470334412082, 'pattern_counts': {'":': 0, '<': 39, '<?xml version=': 0, '>': 38, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '348460', 'license': 'MIT', 'n_tokens_mistral': 3104, 'n_tokens_neox': 2878, 'n_words': 942, 'path': 'DeepLearning/logistic_gradient_descent.ipynb', 'repo_name': 'connorodell/learning', 'size': '8773'}