<jupyter_start><jupyter_text>Logistic Regression - SKLearn, TensorFlow, Keras, PyTorch Author: <NAME>Date: September 23, 2019Updated: September 24, 2019 The goal of this notebook is to compute a multi-class Logistic Regression using different frameworks. The frameworks that will be investigated are:- Sklearn- Statsmodel- TensorFlow 1.0- Keras- PyTorch- TensorFlow 2.0 Equations pertaining to logistic regression:$$ z = X^T W + b$$$$ \hat{y} = \sigma(X^T W + b)$$$$ \sigma(z) = \frac{1}{1+e^{-z}}$$ For multi-classification problem, the softmax function will be used for the activation function in the output layer.$$Softmax(x_i)=\frac{e^{x_i}}{\sum_j e^{x_j}}$$ The loss function that will be used will be the cross-entropy loss.$$H(y, \hat{y}) = - \sum_i y_i log(\hat{y_i})$$<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

from sklearn.datasets import load_iris
from sklearn.metrics import accuracy_score<jupyter_output><empty_output><jupyter_text>The numpy array is forced to have dtype np.float32 so that pytorch will accept these values.<jupyter_code>X = np.array(load_iris().data, dtype=np.float32)
y = np.array(load_iris().target, dtype=np.float32)<jupyter_output><empty_output><jupyter_text>Scaling the data so that it helps with convergence when running gradient descent in TensorFlow, Keras and Pytorch.<jupyter_code>from sklearn.preprocessing import StandardScaler

X_scaler = StandardScaler()
X_scaled = X_scaler.fit_transform(X)<jupyter_output><empty_output><jupyter_text>The input variables<jupyter_code>load_iris().feature_names<jupyter_output><empty_output><jupyter_text>The output variables<jupyter_code>load_iris().target_names<jupyter_output><empty_output><jupyter_text>Standardized Settings for TensorFlow, Keras and PyTorch<jupyter_code>input_dim = 4        # takes variable 'x' 4 input variables, due to 4 columns
output_dim = 3       # takes variable 'y' 3 output variables, due to 3 output categorizations
learning_rate = 0.01
epochs = 1000<jupyter_output><empty_output><jupyter_text>Sklearn Included a large regularization value C to offset the L2 regularization that is default in SKLearn<jupyter_code>from sklearn.linear_model import LogisticRegression
log_model = LogisticRegression(C=1e8, max_iter=epochs, solver='lbfgs', multi_class='multinomial', n_jobs=-1, verbose=0)
log_model.fit(X, y.squeeze())
probas = log_model.predict_log_proba(X)
predicted = np.argmax(probas, 1)
print("Accuracy Score:", accuracy_score(y, predicted))<jupyter_output>Accuracy Score: 0.9866666666666667<jupyter_text>Statsmodel The parameters method and maxiter are tuneable in the fit argument.<jupyter_code>import statsmodels.api as sm # For constants
from statsmodels.discrete.discrete_model import MNLogit

X_c = sm.add_constant(X)
log_model_sm = sm.MNLogit(y, X_c).fit(method='lbfgs', maxiter=epochs) #Fit the model
predicted = np.argmax(log_model_sm.predict(X_c), 1)
print("Accuracy Score:", accuracy_score(y, predicted))<jupyter_output>Accuracy Score: 0.9866666666666667<jupyter_text>TensorFlow 1.0 In TensorFlow there is no loss function specifically for cross-entropy. However there is a loss function that combines both the Softmax and the cross-entropy loss function.There are two path ways to implement this loss function. However, this will influence how the placeholder and cost function will be chosen.- With One-hot    - Placeholder: Ensure the shape to be output of dimension `[None, output_dim]`. output_dim being the number of classes        - `y_tf = tf.placeholder(tf.float32, shape=[None, output_dim], name='y')`    - feed_dict: Ensure that you pass data as a one-hot with dtype float32. Use `pd.getdummies()` to one-hot        - `y_onehot = pd.get_dummies(y.squeeze())`        - `y_onehot = np.array(y_onehot, dtype='float32')`    - `tf.losses.softmax_cross_entropy()`, this uses the function below    - `tf.nn.softmax_cross_entropy_with_logits_v2()` - must wrap this with `tf.reduce_mean()`- Without one-hot    - Placeholder: Ensure the shape to be output of dimension `[None, 1]`        - `y_tf = tf.placeholder(tf.float32, shape=[None, 1], name='y')`    - feed_dict: Ensure that you pass a 1-D array with dtype int32        - `y_sparse = np.array(y, dtype='int32').reshape(-1,1)`    - Use the loss functions with sparse in the name.        - `tf.losses.sparse_softmax_cross_entropy()`        - `tf.nn.sparse_softmax_cross_entropy_with_logits()` - must wrap this with `tf.reduce_mean()`<jupyter_code>import tensorflow as tf
print('Version:', tf.__version__)<jupyter_output>Version: 1.14.0<jupyter_text>The following will demonstrate the version with one-hot. Need to one-hot the y labels for `tf.losses.softmax_cross_entropy` to work correctly.<jupyter_code>y_onehot = pd.get_dummies(y.squeeze())
y_onehot = np.array(y_onehot, dtype='float32')<jupyter_output><empty_output><jupyter_text>Creating the placeholder for the data.<jupyter_code>X_tf = tf.placeholder(tf.float32, shape=[None, input_dim], name='X')
y_tf = tf.placeholder(tf.float32, shape=[None, output_dim], name='y')<jupyter_output><empty_output><jupyter_text>Create variables for W and b. Initializing W with Xavier/Glorot Initialization and b with zeros.<jupyter_code>W_tf = tf.Variable(tf.glorot_uniform_initializer()((input_dim, output_dim)), dtype=tf.float32, name='W')
b_tf = tf.Variable(tf.zeros_initializer()((output_dim)), dtype=tf.float32, name='b')<jupyter_output>WARNING:tensorflow:From C:\Users\yauha\Anaconda3\envs\tf\lib\site-packages\tensorflow\python\ops\init_ops.py:1251: calling VarianceScaling.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor<jupyter_text>Define the y predicted equation. Note: There is need to wrap the equation with `tf.nn.softmax()` because the loss function calculates the softmax. Without the wrapper is the logit, and this is passed into the loss function.<jupyter_code>#WX + b
y_pred_tf = tf.add(tf.matmul(X_tf, W_tf), b_tf)

#The cost function
cost_tf = tf.losses.softmax_cross_entropy(onehot_labels=y_onehot, logits=y_pred_tf)

#The optimizer
optimizer_tf = tf.train.AdamOptimizer(learning_rate).minimize(cost_tf)

#Global Variables Initilizer -- Mandatory
init = tf.global_variables_initializer()<jupyter_output>WARNING:tensorflow:From C:\Users\yauha\Anaconda3\envs\tf\lib\site-packages\tensorflow\python\ops\losses\losses_impl.py:121: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where<jupyter_text>Run the computation graph. For the output, run an argmax function `tf.argmax()` on the output results to get the categories.<jupyter_code>#The Tensorflow Session
with tf.Session() as sess:
    sess.run(init)
    
    for epoch in range(epochs):
        
        feed_dict = feed_dict={X_tf: X_scaled, y_tf: y_onehot}
        
        #Run the optimizer at every epoch
        sess.run(optimizer_tf, feed_dict)
            
        if (epoch + 1) % 50 == 0:
            #Calculate the cost at every epoch
            print("Epoch", (epoch + 1), ": cost =", cost_tf.eval(feed_dict)) 
    
    # Store values
    predicted = sess.run(tf.argmax(y_pred_tf, 1), feed_dict)
accuracy_score(y, predicted)<jupyter_output><empty_output><jupyter_text>Keras In Keras the loss function<jupyter_code>import tensorflow as tf
from tensorflow.keras import Sequential
from tensorflow.keras.layers import Dense
print(tf.keras.__version__)
model = Sequential([
    Dense(output_dim, activation='softmax', input_shape=(input_dim,)),
])

optimizer = tf.keras.optimizers.Adam(learning_rate)

model.compile(loss='sparse_categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])

model.summary()
history = model.fit(x=X_scaled, y=y, epochs=epochs, verbose=0)
acc = model.evaluate(X_scaled, y, verbose=0)[1]
print('Accuracy:', acc)<jupyter_output>Accuracy: 0.98<jupyter_text>PyTorch<jupyter_code>import torch
class LogisticRegressionModel_PyT(torch.nn.Module):
    def __init__(self, input_dim, output_dim):
        super(LogisticRegressionModel_PyT, self).__init__()
        self.W = torch.nn.Parameter(torch.nn.init.xavier_uniform_(torch.empty(input_dim, output_dim)))
        self.b = torch.nn.Parameter(torch.zeros(output_dim))

        
    def forward(self, x):
        out = torch.matmul(x, self.W) + self.b
        return out            

class LogisticRegressionModel_PyT_Layers(torch.nn.Module):
    def __init__(self, input_dim, output_dim):
        super(LogisticRegressionModel_PyT_Layers, self).__init__()
        self.linear = torch.nn.Linear(input_dim, output_dim)
        
    def forward(self, x):
        out = self.linear(x)
        return out            
model = LogisticRegressionModel_PyT(input_dim, output_dim)

if torch.cuda.is_available():
    model.cuda()
    
criterion = torch.nn.CrossEntropyLoss() #This calculates both nn.LogSoftmax and nn.NLLLoss
optimiser = torch.optim.Adam(model.parameters(), lr = learning_rate)
#Convert to Variable

if torch.cuda.is_available():
    X_data = torch.from_numpy(X_scaled).cuda()
    y_data = torch.tensor(y.squeeze(), dtype=torch.long).cuda()
else:
    X_data = torch.from_numpy(X_scaled)
    y_data = torch.tensor(y.squeeze(), dtype=torch.long)

    
for epoch in range(epochs):
    # Clear gradient buffers because we don't want any gradient from previous epoch to carry forward, 
    # dont want to cummulate gradients
    optimiser.zero_grad()

    # Get output from the model, given the inputs
    y_pred = model(X_data)
    # Get loss for the predicted output
    loss = criterion(y_pred, y_data)
    # Get gradients w.r.t to parameters
    loss.backward()

    # Update parameters
    optimiser.step()
    if (epoch + 1) % 50 == 0:
        print('epoch {}, loss {}'.format(epoch + 1, loss.item()))    
    
with torch.no_grad():
    if torch.cuda.is_available():
        predicted = model(torch.from_numpy(X_scaled).cuda())
        _, predicted = torch.max(predicted.data, 1)
        predicted = predicted.cpu().data.numpy()
    else:
        predicted = model(torch.from_numpy(X_scaled))
        _, predicted = torch.max(predicted.data, 1)
        predicted = predicted.data.numpy()
accuracy_score(y, predicted)<jupyter_output><empty_output><jupyter_text>Note: The environment has been switched at this point of the notebook  TensorFlow 2.0<jupyter_code>import tensorflow as tf
print('Version:', tf.__version__)
print('Version:', tf.keras.__version__)<jupyter_output>Version: 2.0.0-rc2
Version: 2.2.4-tf<jupyter_text>Creating the model.Differences from TensorFlow 1.0:- `tf.placeholders()`, `tf.Session()` has been removed- `tf.glorot_uniform_initializer()` is gone, and has become `tf.keras.initializers.GlorotUniform()`- Loss functions are streamlined. Must have a softmax layer and use a Cross-Entropy loss function. There are no loss function that calculates both the softmax and cross-entropy anymore.    - However, the `tf.nn` version of the loss functions still exist. To use this loss function, please wrap it with `tf.reduce_mean()` and modify your code accordingly. For example, remove the softmax layer and pass in the logits directly.Alternatively, the model class can utilize the layers in Keras. This is analogous to PyTorch.<jupyter_code>class LogisticRegressionModel_TF2(tf.keras.Model):
    def __init__(self, input_dim, output_dim):
        super(LogisticRegressionModel_TF2, self).__init__()
        self.W = tf.Variable(tf.keras.initializers.GlorotUniform()((input_dim, output_dim)), dtype=tf.float32, name='W')
        self.b = tf.Variable(tf.zeros_initializer()((output_dim)), dtype=tf.float32, name='b')
    
    #In PyTorch the function will be 'forward' instead.
    def call(self, x):
        x = tf.matmul(x, self.W) + self.b
        out = tf.keras.activations.softmax(x)
        return out
    
class LogisticRegressionModel_TF2_Layers(tf.keras.Model):
    def __init__(self, input_dim, output_dim):
        super(LogisticRegressionModel_TF2_Layers, self).__init__()
        self.d1 = tf.keras.layers.Dense(output_dim, activation='softmax', input_dim=(input_dim,))
        
    def call(self, x):
        x = self.d1(x)
        return x
model = LogisticRegressionModel_TF2(input_dim, output_dim)
@tf.function
def compute_loss(model, inputs, targets):
    predictions = model(inputs)
    return criterion(targets, predictions)

@tf.function
def grad(model, inputs, targets):
    with tf.GradientTape() as tape:
        predictions = model(inputs)
        loss_value = criterion(targets, predictions)
        
    #This calculates teh gradient
    return tape.gradient(loss_value, model.trainable_variables)

criterion = tf.keras.losses.SparseCategoricalCrossentropy()
optimizer = tf.keras.optimizers.Adam(learning_rate)
for epoch in range(epochs):
    
    #Backward propagation
    grads = grad(model, X_scaled, y)
    
    #update parameters
    optimizer.apply_gradients(zip(grads, model.trainable_variables))
    
    if(epoch+1) % 50 == 0:
        template = 'Epoch {}, Loss: {}'
        print(template.format(epoch+1, compute_loss(model, X_scaled, y)))
predicted = tf.argmax(model(X_scaled), 1)
accuracy_score(y, predicted)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1085949177877429, 'fraction_numerical': 0.012705530642750373, 'mean_word_length': 4.755268817204301, 'pattern_counts': {'":': 1, '<': 68, '<?xml version=': 0, '>': 68, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '399184', 'license': 'MIT', 'n_tokens_mistral': 4551, 'n_tokens_neox': 4375, 'n_words': 1114, 'path': 'Notebooks/TensorFlow-Keras-Pytorch/02 Logistic Regression.ipynb', 'repo_name': 'vrif/portfolio', 'size': '13383'}