<jupyter_start><jupyter_text>Stochastic Gradient DescentThe following example was initially designed as an old classroom experiment, where one might need to fit a logistic regression with online learning where some of the weights are fixed. Example implementation and usage Logistic Regression with SGDThe sigmoid function is the main structure for the logistic regression. It is bounded by $(0,1)$ and is written as: $$\sigma(x) = \frac{1}{(1 + e^{x})}$$The cost function for the logistic regression is the log-loss or cross-entropy. It is given by: $$J(\Theta) = - \frac{1}{m} \sum_{i=1}^{m}\left(y^{(i)} \cdot \log \hat{y}^{(i)} + (1 - y^{(i)})\cdot\log(1 - \hat{y}^{(i)})\right)$$To minimize the cost value at each iteration, one might use gradient descent, such as:$$\Theta_{j} := \Theta_{j} - \alpha \frac{\partial}{\partial \Theta_{j}}J(\Theta)$$The derivative term results in:$$\frac{\partial J(\Theta)}{\partial \Theta_{j}} = ... = \frac{1}{m}\sum_{i=1}^{m}\left(\hat{y}^{(i)} - y^{(i)}\right)x_{j}^{(i)}$$Which leads to the following algorithm, used to estimate and update the all the $\Theta_{j}$ parameters (weights).$$\Theta_{j} := \Theta_{j} - \frac{\alpha}{m} \sum_{i=1}^{m}\left(\hat{y}^{(i)} - y^{(i)}\right)x_{j}^{(i)}$$Because the weights are updated at each step, to fix some of the weights one might need to indicate the indices referring to those weights and overwrite at each iteration the updated weights with the fixed values.The code shown below was designed only to illustrate the idea and does not represent an optimized script! Use carefully...  Import Libs<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.datasets import load_breast_cancer, load_wine
from sklearn.model_selection import KFold, train_test_split
from sklearn import metrics
import sys
%matplotlib inline
sys.path.append('../src/')
import warnings
warnings.filterwarnings("ignore")<jupyter_output><empty_output><jupyter_text>Load data<jupyter_code>data = load_breast_cancer()<jupyter_output><empty_output><jupyter_text>Prepare data  Features and target data<jupyter_code>X = data['data']
y = data['target']
print(X.shape, y.shape)
print(y.mean())<jupyter_output>(569, 30) (569,)
0.6274165202108963<jupyter_text>Split train and test dataThe approach shown below is only for illustration purposes. It is a naive approach. If we want to have an optimal model development we should use something like early stopping or a KFold approach.<jupyter_code>from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=101)

scaler = StandardScaler()

X_train_preproc = scaler.fit_transform(X_train)
X_test_preproc = scaler.transform(X_test)
from sgd import LogisticRegressionModel<jupyter_output><empty_output><jupyter_text>Instantiate model<jupyter_code>fixed_weights = [1.9, 0.5, 0.3] # the convergence of the method is sensitive to these values
fixed_mask = [0, 2, 4]

lr = LogisticRegressionModel(n_iter=1000,
                             stochastic=True, 
                             fixed_weights=fixed_weights,
                             fixed_mask=fixed_mask)
lr.fixed_mask
lr.fixed_weights<jupyter_output><empty_output><jupyter_text>Random initial guess for weight vector<jupyter_code>theta0 = np.random.randn(X.shape[1])
theta0 = theta0.reshape(-1,1)
theta0[:20]<jupyter_output><empty_output><jupyter_text>Train model<jupyter_code>lr.fit(X_train_preproc, y_train, theta0)<jupyter_output><empty_output><jupyter_text>Verify coefficients<jupyter_code>for i in [0, 1, 100, 500, 999]:
    print(f'Iteration number {i + 1}:') 
    print(lr.coefs_[i])
    print()<jupyter_output>Iteration number 1:
[[ 1.90000000e+00 -1.48668990e+00  5.00000000e-01 -8.10044904e-01
   3.00000000e-01 -8.59070736e-01  1.90052034e-01  1.55937034e+00
   9.16763732e-02  2.98757420e-01  1.10762515e+00 -1.66709623e+00
  -6.84318283e-01  9.07523636e-01  8.58936939e-01  1.02553133e+00
  -4.84466924e-01 -1.67950253e+00 -1.79680819e+00 -4.60214641e-01
   3.32886634e+00 -1.09370198e+00 -1.60725309e+00  2.88200134e+00
  -1.50208677e-01 -4.29160957e-01 -7.67011747e-01 -8.69694372e-02
   1.11144160e+00 -1.02788644e-03]]

Iteration number 2:
[[ 1.90000000e+00 -1.48769645e+00  5.00000000e-01 -8.16222924e-01
   3.00000000e-01 -8.62836398e-01  1.85452299e-01  1.55350217e+00
   8.99805214e-02  3.00421066e-01  1.10330423e+00 -1.66484931e+00
  -6.88411538e-01  9.03069109e-01  8.60893410e-01  1.02468775e+00
  -4.85166537e-01 -1.68168221e+00 -1.79547261e+00 -4.59395128e-01
   3.32224531e+00 -1.09506662e+00 -1.61378532e+00  2.87583079e+00
  -1.52836998e-01 -4.32799676e-01 -7.71168238e-01 -9.28736772e-02[...]<jupyter_text>Optimum weights<jupyter_code>lr.coefs_opt_st[:20]<jupyter_output><empty_output><jupyter_text>Predict on test dataThe first 20 results are shown below.<jupyter_code>lr.predict(X_test_preproc)[:20]
fig, ax = plt.subplots(1, 3, figsize=(16,4))

ax = ax.ravel()

sns.distplot(lr.predict(X_test_preproc), hist=False, rug=True, ax=ax[0])
sns.distplot(lr.predict(X_train_preproc), hist=False, rug=True, ax=ax[0])
ax[0].legend(['Test', 'Train'], fontsize=14)

sns.distplot(lr.predict(X_test_preproc), hist=False, rug=True, ax=ax[1])
ax[1].legend(['Test'], fontsize=14)

sns.distplot(lr.predict(X_train_preproc), hist=False, rug=True, ax=ax[2])
ax[2].legend(['Train'], fontsize=14)

for i in range(3):
    ax[i].set_xlabel('Event probability', fontsize=12)<jupyter_output><empty_output><jupyter_text>Costs<jupyter_code>for i in [0, 1, 100, 500, 999]:
    print(f'Iteration number {i}: {lr.costs_[i]}')<jupyter_output>Iteration number 0: inf
Iteration number 1: inf
Iteration number 100: 179.68593865677246
Iteration number 500: 31.755878137172992
Iteration number 999: 13.490944335628013<jupyter_text>Compare costs<jupyter_code>fixed_weights = [1.9, 0.5, 0.3] #the convergence of the method is sensitive to these values
fixed_mask = [0, 2, 4]

stochastic = [True, False]
iterations = 1000
all_costs = {}

for i, s in enumerate(stochastic):

    lr = LogisticRegressionModel(n_iter=iterations,
                                 stochastic=s, 
                                 fixed_weights=fixed_weights,
                                 fixed_mask=fixed_mask)

    lr.fit(X_train_preproc, y_train, theta0)

    all_costs[i] = lr.costs_
fig, ax = plt.subplots(1, 2, figsize=(16,6))

ax = ax.ravel()

x_ = np.arange(0, iterations, 1)

ax[0].plot(x_, all_costs[0].values())
ax[1].plot(x_, all_costs[1].values())
ax[0].set_xlabel("Iterations")
ax[1].set_xlabel("Iterations")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13630916716596764, 'fraction_numerical': 0.1262732174955063, 'mean_word_length': 4.8467600700525395, 'pattern_counts': {'":': 0, '<': 46, '<?xml version=': 0, '>': 46, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 6, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '453817', 'license': 'MIT', 'n_tokens_mistral': 3001, 'n_tokens_neox': 2496, 'n_words': 551, 'path': 'notebooks/stochastic-gradient-descent.ipynb', 'repo_name': 'tdosreis/stochastic-gradient-descent', 'size': '6676'}