<jupyter_start><jupyter_text>Logistic Regression Classifier - NumPy ImplementationIn this tutorial, we will implement the logistic regression classifier ourselves using the NumPy library, just to illustrate how simple it is. If you are not a hardcore programmer, jump to the next demo ([Logistic Regression - Keras](logistic_regression_keras.ipynb)). In future demos, we will profit from the Keras API, which will simplify the implementation of more complex classifiers. The goals of this tutorial are:  - Illustrate how to implement, train and deploy a simple linear classifier (logistic regression), which is the basis of complex (deep) neural networks.  - One-hot-encoding label representation.  - Function minimization using the gradient descent technique.   - Display the classifier decision boundaries Importing Python Libraries<jupyter_code>%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
import sys
from IPython.display import display

## Adding dataset_utils to the system path so we can import it
MY_UTILS_PATH = '../Modules/'
if not MY_UTILS_PATH in sys.path:
    sys.path.append(MY_UTILS_PATH)
import dataset_utils<jupyter_output><empty_output><jupyter_text>Loading and Displaying 2D Dataset<jupyter_code>#Generating a synthetic dataset with four classes
data = dataset_utils.corners(N = 300,scale = 0.5,gapwidth = 3,cornerwidth = 4)

# Separating features (X) and labels (Y)
X = data[:,:-1] # Features
Y = data[:,-1].astype(int) # Labels

colors = np.array(['r','y','b','g'])
#Displaying the dataset
plt.scatter(X[:,0],X[:,1],c=Y)

plt.xlabel("x")
plt.ylabel("y")
plt.grid()
plt.title("Four Corners")
plt.show()<jupyter_output><empty_output><jupyter_text>Data Normalization<jupyter_code># Range Normalization - normalizing between 0 and 1
X_c = X - X.min(axis=0,keepdims=True)
X_c /= X_c.max(axis=0,keepdims=True)

plt.scatter(X_c[:, 0], X_c[:, 1], c=colors[Y])
plt.title("Normalized data")
plt.xlabel("x")
plt.ylabel("y")
plt.grid()
plt.show()<jupyter_output><empty_output><jupyter_text>One-hot encoding<jupyter_code>def oneHotIt(Y,k):
    n = Y.size # number of samples
    i = np.arange(n)
    Y_oh = np.zeros(shape=(n,k))
    Y_oh[i,Y] = 1
    return Y_oh

k = np.unique(Y).size # Number of classes
Y_oh = oneHotIt(Y,k)
# Displaying the 5 first elemnts
print('Y[:5]:')
print(Y[:5])
print('\nY_oh[:5]=')
print(Y_oh[:5])<jupyter_output>Y[:5]:
[0 0 0 0 0]

Y_oh[:5]=
[[1. 0. 0. 0.]
 [1. 0. 0. 0.]
 [1. 0. 0. 0.]
 [1. 0. 0. 0.]
 [1. 0. 0. 0.]]<jupyter_text>Extra column with 1s in X to account for the model bias<jupyter_code>X_bias = np.hstack([np.ones((X_c.shape[0],1)),X_c]) # Including cplumn with 1s to account
                                                    # for the model bias
print('X_bias[:5]=\n')
print(X_bias[:5])<jupyter_output>X_bias[:5]=

[[1.         0.75194354 0.73289023]
 [1.         0.74828653 0.74691814]
 [1.         0.90044195 0.72485022]
 [1.         0.9315348  0.71695659]
 [1.         0.82360487 0.74953027]]<jupyter_text>softmax FunctionThe softmax function converts score values into probabilities. It is defined by the following equation, where each line of Z is a vector of scores. The number of scores is equal to the number of classes in the problem.$$ Softmax(\boldsymbol{Z}) = \frac{e^{\boldsymbol{Z}}}{\sum_{j=0}^{k-1}e^{z_{i,j}}} $$<jupyter_code>def softmax(Z):
    EZ = np.exp(Z)
    S = EZ / EZ.sum(axis=1,keepdims=True) 
    return S<jupyter_output><empty_output><jupyter_text>Logistic Regression Training* $\boldsymbol{(X,Y)}$ -> Labeled data* $\boldsymbol{W}$ -> Parameters to be learned* Softmax -> converts our score vector ($\boldsymbol{XW}$) into a probability vector ($\hat{Y_{prob}}$)* One-hot encoding -> converts our probability vector to a one-hot-encoding representation ($\hat{Y_{OH}}$)* $J$ -> Loss function that compares our true labels $Y_{OH}$ with our predicted (estimated) labels $\hat{Y_{OH}}$ a ser minimizada* $\Delta_WJ$ -> Gradient vector that will be used to update the weights $\boldsymbol{W}$ using the gradient descent technique ![](../figures/regressaologistica_fluxo.jpg)  J: Cross-entropy Loss FunctionIn our example we use the cross-entropy as our loss function($J$). Given a set of labeled samples ($\boldsymbol{X}$,$\boldsymbol{Y}$), we want to find the weights $\boldsymbol{W}$ that minimize $J$:$$ J_{\boldsymbol{X,Y}_{TRUE}}(\boldsymbol{W}) = -\frac{1}{n} \sum \boldsymbol{Y}_{TRUE} \odot \log{\boldsymbol{S}(\boldsymbol{XW})}$$where $\odot$ is the point-by-point matrix multiplication. Gradient Descent Minimization The loss function is minimized using the gradient descent method. The gradient of $J$ is given by:$$ \boldsymbol{\Delta J(W)} = - \frac{1}{n} \boldsymbol{X}^T (\boldsymbol{Y}_{TRUE} - \boldsymbol{S(XW)}) $$ The weights are updated after each *batch* using the follow equation:$$ \boldsymbol{W} = \boldsymbol{W} - \eta \boldsymbol{\Delta J(W)} $$ $\eta$ is called the learning rate. In the following program, $J$ is represented by the variable *loss* and  $\Delta J$ is denoted by the variable *Grad*. The weights are initialized with 0s.<jupyter_code>n = X_bias.shape[0] # number of training examples
m = X_bias.shape[1] # number of attributes (features + 1(bias))
W = np.zeros([m,k]) # Weights matrix

iterations = 10000
learningRate = 0.15

losses = [] # store loss values during each training iteration

for i in range(iterations):
    scores = X_bias.dot(W)
    S = softmax(scores)
    loss = (-1. / n) * np.sum(Y_oh * np.log(S))
    Grad = (-1. / n) * (X_bias.T).dot(Y_oh - S) 
    W = W - (learningRate * Grad)
    losses.append(loss)
print 'final W:\n',W
print 'Final loss:',loss

plt.plot(losses)
plt.xlabel("Iteration")
plt.ylabel("Loss")
plt.grid()
plt.show()<jupyter_output>final W:
[[-6.5354209   7.52631585 -0.45955096 -0.53134399]
 [ 7.05154698 -7.31287012  9.43408014 -9.17275699]
 [ 7.04370825 -7.30453003 -9.20540229  9.46622407]]
Final loss: 0.02768107982713436<jupyter_text>Logistic Regression PredictionOnce the values of $\boldsymbol{W}$ were learned during training, prediction is performed by the following operations:* Softmax(XW) -> computes probabilities for each class* Argmax -> selects class with highest probabilityThe predict function receives as input the following arguments:- X: 2D matrix (n,m): n samples by m features - samples- W: 2D matrix (m,k): m features e k classes - weightsThe output of the function is:- Y_hat: 2D matrix (n,1): n samples x 1 - predicted labels<jupyter_code>def predict(X,W):
    S = softmax(X.dot(W))
    # selects the class (label) with highest probability
    Y_hat = np.argmax(S,axis=1)
    return Y_hat<jupyter_output><empty_output><jupyter_text>Accuracy MetricAccuracy is defined as the ratio between the number of correct predictions and the total number of  samples ($n$).$$ Acc(Y,\hat{Y}) = \frac{1}{n} \sum_{i=0}^{n-1} 1(Y_i = \hat{Y}_i) $$<jupyter_code>Y_hat = predict(X_bias,W)

accuracy = np.sum(Y_hat == Y)/np.float(n)

print 'Accuracy: ', accuracy<jupyter_output>Accuracy:  1.0<jupyter_text>Logistic Regression Decision Boundary<jupyter_code>h = .02  # step size in the mesh
slack = 0.1
# Calcula a grade para o espaço de atributos

x_min, x_max = X_c.min(axis=0) - slack, X_c.max(axis=0) + slack
xx, yy = np.meshgrid(np.arange(x_min[0], x_max[0], h), np.arange(x_min[1], x_max[1], h))
X_grid = np.c_[xx.ravel(), yy.ravel()]
X_grid = np.hstack([np.ones((X_grid.shape[0],1)),X_grid]) # bias column

# Prediction of all samples in our grid
Z = predict(X_grid, W)

# Display decision boundary
Z = Z.reshape(xx.shape)
plt.pcolormesh(xx, yy, Z, cmap=plt.cm.Paired)

# Display training samples
plt.scatter(X_bias[:, 1], X_bias[:, 2], c=colors[Y], edgecolors='k', cmap=plt.cm.Paired)
plt.xlabel('xcoord')
plt.ylabel('ycoord')
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14267368691408758, 'fraction_numerical': 0.04135096956465905, 'mean_word_length': 4.606911447084233, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 45, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '322525', 'license': 'MIT', 'n_tokens_mistral': 2932, 'n_tokens_neox': 2704, 'n_words': 810, 'path': 'JNotebooks/logistic_regression_numpy.ipynb', 'repo_name': 'rmsouza01/ML101', 'size': '7787'}