<jupyter_start><jupyter_text>Simple Classifier / Logistic Regression This notebook will demosntrate a simple logistic regression model predicting whether a house is ```low-priced``` or ```expensive```. Similar to our linear model in ```1_linear_regression.ipynb```, we feed features from the HousingPrice dataset into the classifier model. However, now, we expect our model to output a score that determines in which category the considered house is. ![classifierTeaser](images/classifierTeaser.png) $ $ Let $\mathbf{X} \in \mathbb{R}^{N\times (D+1)}$ denote our data with $N$ samples and $D$ feature dimensions. Our targets, the binary labels, are given by $\mathbf{y} \in \mathbb{R}^{N\times 1}$. We want to estimate them with a simple classifier of the form$$ \mathbf{y}  = \sigma \left( \mathbf{X} \mathbf{w} \right), $$ $ $ where $\mathbf{w}\in \mathbb{R}^{(D+1) \times 1}$ is the weight of our classifier. The sigmoid function $\sigma: \mathbb{R} \to [0, 1]$, defined by $$ \sigma(t) = \frac{1}{1+\mathrm{exp}(-t)}, $$is used to squeeze the ouputs of the linear layer into the range $[0, 1]$. This provides us with a probabilistic interpretation of the ouput of the neural network and we can compute the label predictions by rounding the output.<jupyter_code>from exercise_code.data.csv_dataset import CSVDataset
from exercise_code.data.csv_dataset import FeatureSelectorAndNormalizationTransform
from exercise_code.data.dataloader import DataLoader

import matplotlib.pyplot as plt
import numpy as np
import os
import pandas as pd
import seaborn as sns


pd.options.mode.chained_assignment = None  # default='warn'

%matplotlib inline
%load_ext autoreload
%autoreload 2<jupyter_output><empty_output><jupyter_text>1. Load your data We apply the same dataloading and preprocessing steps as in the notebook ```1_linear_regression.ipynb```.<jupyter_code>target_column = 'SalePrice'
i2dl_exercises_path = os.path.dirname(os.path.abspath(os.getcwd()))
root_path = os.path.join(i2dl_exercises_path, "datasets", 'housing')
housing_file_path = os.path.join(root_path, "housing_train.csv")
download_url = 'https://cdn3.vision.in.tum.de/~dl4cv/housing_train.zip'

# Always make sure this line was run at least once before trying to
# access the data manually, as the data is downloaded in the 
# constructor of CSVDataset.
train_dataset = CSVDataset(target_column=target_column, root=root_path, download_url=download_url, mode="train")<jupyter_output><empty_output><jupyter_text>For the data transformations, compute min, max and mean for each feature column. We perform the same transformation on the training, validation, and test data.<jupyter_code>df = train_dataset.df
# Select only 2 features to keep plus the target column.
#selected_columns = ['OverallQual', 'GrLivArea', target_column]
selected_columns = ['GrLivArea', target_column]
mn, mx, mean = df.min(), df.max(), df.mean()

column_stats = {}
for column in selected_columns:
    crt_col_stats = {'min' : mn[column],
                     'max' : mx[column],
                     'mean': mean[column]}
    column_stats[column] = crt_col_stats    

transform = FeatureSelectorAndNormalizationTransform(column_stats, target_column)

def rescale(data, key = "SalePrice", column_stats = column_stats):
    """ Rescales input series y"""
    mx = column_stats[key]["max"]
    mn = column_stats[key]["min"]

    return data * (mx - mn) + mn

# Always make sure this line was run at least once before trying to
# access the data manually, as the data is downloaded in the 
# constructor of CSVDataset.
train_dataset = CSVDataset(mode="train", target_column=target_column, root=root_path, download_url=download_url, transform=transform)
val_dataset = CSVDataset(mode="val", target_column=target_column, root=root_path, download_url=download_url, transform=transform)
test_dataset = CSVDataset(mode="test", target_column=target_column, root=root_path, download_url=download_url, transform=transform)


print("Number of training samples:", len(train_dataset))
print("Number of validation samples:", len(val_dataset))
print("Number of test samples:", len(test_dataset))
# load training data into a matrix of shape (N, D), same for targets resulting in the shape (N, 1)
X_train = [train_dataset[i]['features'] for i in range((len(train_dataset)))]
X_train = np.stack(X_train, axis=0)
y_train = [train_dataset[i]['target'] for i in range((len(train_dataset)))]
y_train = np.stack(y_train, axis=0)
print("train data shape:", X_train.shape)
print("train targets shape:", y_train.shape)

# load validation data
X_val = [val_dataset[i]['features'] for i in range((len(val_dataset)))]
X_val = np.stack(X_val, axis=0)
y_val = [val_dataset[i]['target'] for i in range((len(val_dataset)))]
y_val = np.stack(y_val, axis=0)
print("val data shape:", X_val.shape)
print("val targets shape:", y_val.shape)

# load train data
X_test = [test_dataset[i]['features'] for i in range((len(test_dataset)))]
X_test = np.stack(X_test, axis=0)
y_test = [test_dataset[i]['target'] for i in range((len(test_dataset)))]
y_test = np.stack(y_test, axis=0)
print("test data shape:", X_val.shape)
print("test targets shape:", y_val.shape)


# 0 encodes small prices, 1 encodes large prices.<jupyter_output>train data shape: (876, 1)
train targets shape: (876, 1)
val data shape: (292, 1)
val targets shape: (292, 1)
test data shape: (292, 1)
test targets shape: (292, 1)<jupyter_text>In the following, we model the regression task as a binary classification problem in the categories ```low-priced```and ```expensive``` by labeling the 30% of the houses that are sold with the lowest price with ```0``` and, accordingly, the 30% of the houses with the highest price with ```1```.<jupyter_code>from exercise_code.networks.utils import binarize
y_all = np.concatenate([y_train, y_val, y_test])
thirty_percentile = np.percentile(y_all, 30)
seventy_percentile = np.percentile(y_all, 70)

# Prepare the labels for classification.
X_train, y_train = binarize(X_train, y_train, thirty_percentile, seventy_percentile )
X_val, y_val   = binarize(X_val, y_val, thirty_percentile, seventy_percentile)
X_test, y_test  = binarize(X_test, y_test, thirty_percentile, seventy_percentile)<jupyter_output><empty_output><jupyter_text>1. Set up a classfier model We define a simple classifier in ```exercise_code/networks/classifier.py```. Implement the forward pass in method ```forward()``` and the backward pass in ```backward()``` in the Network class ```Classifier```. This time, you also need to implement the function ```sigmoid()```.<jupyter_code>from exercise_code.networks.classifier import Classifier

model = Classifier(num_features=1)
model.initialize_weights()

y_out, _ = model(X_train)

# plot the prediction
plt.scatter(X_train, y_train)
plt.plot(X_train, y_out, color='r')<jupyter_output><empty_output><jupyter_text>2. Implement the Loss Function: Binary Cross EntropyIn this part, you will implement a binary cross entropy (BCE) loss function. Open the file `exercise_code/networks/loss.py` and implement the forward and backward pass of BCE loss into the `forward` and `backward` function.Remember the BCE loss function is:$$ bce = -\hat y log(y) - (1- \hat y) log(1-y)$$$ $ where $y$ is the output of your model, and $\hat y$ is the ground truth of the data<jupyter_code>from exercise_code.networks.loss import BCE

bce_loss = BCE()<jupyter_output><empty_output><jupyter_text>Forward and Backward CheckOnce you have finished implementation of BCE loss class, you can run the following code to check whether your forward result and backward gradient are correct. You should expect your relative error to be lower than 1e-8.Here we will use a numeric gradient check to debug the backward pass:$$ \frac {df(x)}{dx} = \frac{f(x+h) - f(x-h)}{2h} $$where $h$ is a very small number, in practice approximately 1e-5 or so.<jupyter_code>from exercise_code.tests.loss_tests import *
print (BCETest(bce_loss)())<jupyter_output>BCEForwardTest passed.
BCEBackwardTest passed.
Congratulations you have passed all the unit tests!!! Tests passed: 2/2
(0, 2)<jupyter_text>3. Run SolverYou have successfully implement a solver in the last task, now we will use that solver to solve this logistic regression problem.<jupyter_code>from exercise_code.solver import Solver
from exercise_code.networks.utils import test_accuracy
from exercise_code.networks.classifier import Classifier


# Select the number of features, you want your task to train on.
# Feel free to play with the sizes.
num_features = 1

# initialize model and weights
model = Classifier(num_features=num_features)
model.initialize_weights()

y_out, _ = model(X_test)

accuracy = test_accuracy(y_out, y_test)
print("Accuracy BEFORE training {:.1f}%".format(accuracy*100))


if np.shape(X_val)[1]==1:
    plt.scatter(X_val, y_val, label = "Ground Truth")
    inds = X_test.flatten().argsort(0)
    plt.plot(X_test[inds], y_out[inds], color='r', label = "Prediction")
    plt.legend()
    plt.show()

data = {'X_train': X_train, 'y_train': y_train,
        'X_val': X_val, 'y_val': y_val}

# We are going to use the BCE loss for this task.
loss = BCE()

# Please use these hyperparmeter as we also use them later in the evaluation
learning_rate = 1e-1
epochs = 25000

# Setup for the actual solver that's going to do the job of training
# the model on the given data. set 'verbose=True' to see real time 
# progress of the training.
solver = Solver(model, 
                data, 
                loss,
                learning_rate, 
                verbose=True, 
                print_every = 1000)

# Train the model, and look at the results.
solver.train(epochs)
plt.plot(solver.val_loss_history, label = "Validation Loss")
plt.plot(solver.train_loss_history, label = "Train Loss")
plt.xlabel("Epoch")
plt.ylabel("Loss")
plt.legend() 
plt.show() 

# Test final performance
y_out, _ = model(X_test)

accuracy = test_accuracy(y_out, y_test)
print("Accuracy AFTER training {:.1f}%".format(accuracy*100))

if np.shape(X_test)[1]==1:

    plt.scatter(X_test, y_test, label = "Ground Truth")
    inds = X_test.argsort(0).flatten()
    plt.plot(X_test[inds], y_out[inds], color='r', label = "Prediction")
    plt.legend()
    plt.show()<jupyter_output>Accuracy BEFORE training 15.8%<jupyter_text>Save your BCELoss, Classifier and Solver for SubmissionSimply save your objects using the following cell. This will save them to a pickle file `models/logistic_regression.p`.<jupyter_code>from exercise_code.tests import save_pickle

save_pickle(
    data_dict={
        
        "BCE_class": BCE,
        "Classifier_class": Classifier,
        "Solver_class": Solver
    },
    file_name="logistic_regression.p"
)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11292885855847434, 'fraction_numerical': 0.009909320370197252, 'mean_word_length': 5.289241622574956, 'pattern_counts': {'":': 3, '<': 34, '<?xml version=': 0, '>': 34, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 11, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '256304', 'license': 'MIT', 'n_tokens_mistral': 3536, 'n_tokens_neox': 3397, 'n_words': 1101, 'path': 'exercise_04/2_logistic_regression.ipynb', 'repo_name': 'buzem/i2dl', 'size': '10697'}