<jupyter_start><jupyter_text>Overview**This tutorial will focus on _Logistic Regression_**We will include some additional topics, including: * Handling categorical attributes* Logistic regression* Training for maximum likelihood* Evaluation of classification models**The lecture relies on the the following proficiencies and tools:*** [Python](https://www.python.org/) programming* Vector computations via [the numpy module](https://numpy.org/)* Data handling using [the pandas module](https://pandas.pydata.org/)* Plotting using [matplotlib](https://matplotlib.org/)* Training and using Machine Learning model via [scikit-learn](https://scikit-learn.org/)You will need them only if you plan to handle these tasks yourself  Categorical Attributes  Categorical Attributes**Let's switch to a different dataset (a toy one)*** We want to train a model to choose whether to go out and play* ...Based on weather conditions  Loading the Data**The dataset is in the `weather.csv` file from the `data` folder**<jupyter_code>!ls data
import pandas as pd
data = pd.read_csv('data/weather.csv', sep=',')
data.head()<jupyter_output><empty_output><jupyter_text>* Several attributes _do not have a numeric value_* Instead, their value is discrete with no clear ordering, i.e. _categorical_**We need a _numeric encoding_ to handle this data with linear models**  Encoding Binary Attributes**Binary attributes can be encoded with the values 0 and 1**This is the case for the columns "windy" and "play"* First, we tell pandas that the columns have a categorical type<jupyter_code>windy = data['windy'].astype('category')
play = data['play'].astype('category')
windy.head()<jupyter_output><empty_output><jupyter_text>* Categorical data is still _displayed as a string_* ...But internally it is _encoded as an integer_  Encoding Binary Attributes**Next, we replace the values with their integer code**We will store the results in a copy of the original table<jupyter_code>data2 = data.copy() # We prepare a cop for the numeric encodings
data2['windy'] = windy.cat.codes
data2['play'] = play.cat.codes
data2.head()<jupyter_output><empty_output><jupyter_text>* Now it is apparent that "windy" and "play" have become numbers  Encoding Discrete Attributes**We could use the same approach for discrete attribute in general**E.g. for the attribute "outlook" in our table* That would yield a numeric _integer_ encoding* ...Which implies an ordering among the values (e.g. rainy < overcast < sunny)* When no such ranking exists, this is a bad idea**In these cases, it is better to adopt a one-hot encoding*** We introduce a column for each value $v_k$ of the attribute $xj$* The column contains a 1 iff $x_j = v_k$, and 0 otherwiseFor example, "sunny | sunny | overcast" becomes:rainy | overcast | sunny:----:|:--------:|:-----:  0   |    0     |   1  0   |    0     |   1  0   |    1     |   0  Encoding Discrete Attributes**We can obtain a one-hot encoding in pandas via the `get_dummies` method**<jupyter_code>data2 = pd.get_dummies(data2)
data2.head()<jupyter_output><empty_output><jupyter_text>* The method by default processes all columns with categorical or object type* strings in csv files are often parsed as "object" columns* In truth, `get_dummies` can also handle the special case of binary variables* ...But I wanted to show you how to obtain an integer encoding, too :-)  Logistic Regression  Logistic Regression**Our goal is to predict the value of "play", i.e. _a categorical attribute_**We say that we are dealing with _a classification problem_ and:* We compute the output of the linear model as usual:$$g(x; w) = \sum_{j=1} w_j x_j + w_0$$ * ...But then we feed it to a _logistic function_:$$f(x; w) = \frac{1}{1 + e^{-g(x; w)}}$$  Logistic Regression**The logistic function is a type of _sigmoid_ function**<jupyter_code>import numpy as np
from matplotlib import pyplot as plt
x = np.linspace(-10, 10, 100)
plt.figure(figsize=(9, 3))
plt.plot(x, 1 / (1 + np.exp(-x)))
plt.grid(':')<jupyter_output><empty_output><jupyter_text>* Due to its use, this approach is known as _logistic regression_  Logistic Regression**Why using the logistic function?*** We can view the model output as a probability distribution* Specifically, as _the probability of the class being "1"_**With this convention, the target can also be interpreted as a probability**<jupyter_code>data2['play'].head()<jupyter_output><empty_output><jupyter_text>We view:* $y_i = 0$ as "the probability of the class being 1 is equal to 0"* $y_i = 1$ as "the probability of the class being 1 is equal to 1"  Likelihood Function**We now pretend that our model $f(x; w)$ is _the source of our data_**E.g. if $f(x; w) = 0.7$* We will generate a 1 with 70% chance* We will generate a 0 with 30% chance**Why is this interesting?**Now we can measure the chance that the model makes the right guess:* When we have $y_i = 1$, i.e. the label is 1  - We will guess right with a $f(x; w)$ probability, wrong with $1-f(x; w)$* When we have $y_i = 0$, i.e. the label is 0  - We will guess right with a $1-f(x; w)$ probability, wrong with $f(x; w)$  Likelihood Function**In summary*** When we have $y_i = 1$, we want $f(x; w)$ to be _high_* When we have $y_i = 0$, we want $f(x; w)$ to be _low_**If we repeat for all examples (assuming statistical independence)...**We get the _the probability of generating our data_.* For all the examples where the class is 1, we get:$$\prod_{y_i=1} f(x_i; w)$$* For all the examples where the class is 0, we get:$$\prod_{y_i=0} (1-f(x_i; w))$$  Likelihood Function**We both classes, we take yet another product**$$L(w) = \prod_{y_i=1} f(x_i; w) \prod_{y_i=0} (1-f(x_i; w))$$* The is sort of a probability, but is _associated to our model_, not to the data itself* ...And it also _depends on the parameters $w$_To make the distinction clearer, we call it a _likelihood function_**We want to train a model that is a _likely source for our data_**This means that we can choose the weights by solving:$$\text{argmax}_w \log L(w)$$* I.e. to _maximize the likelihood_ of the data* This often done via Gradient Descent  Maximum Likelihood Estimation> **MLE is very important in many Machine Learning approaches*** It provides a _mathematical foundation_ for the training process* It applies to linear regression, too!* ...Since the MSE can be interpreted in terms of likelihoodIt even works when targets/labels are absent* I.e. for _unsupervised learning_ scenarios  Using Logistic Regression**Using Logistic Regression in scikit-learn is actually easy**We begin by splitting input/output data as usual:<jupyter_code>cols_in = [c for c in data2.columns if c != 'play']

X = data2[cols_in]
y = data2['play']
y.head() # We have a table here, but a vector would also work<jupyter_output><empty_output><jupyter_text>Then the training and test set:<jupyter_code>from sklearn.model_selection import train_test_split
X_tr, X_ts, y_tr, y_ts = train_test_split(X, y, test_size=0.34, random_state=0)<jupyter_output><empty_output><jupyter_text>Using Logistic Regression**Then, we build a `LogisticRegression` model**<jupyter_code>from sklearn.linear_model import LogisticRegression

m = LogisticRegression()<jupyter_output><empty_output><jupyter_text>...And we call the `fit` method as usual:<jupyter_code>m.fit(X_tr, y_tr);<jupyter_output><empty_output><jupyter_text>Finally, we can obtain out predictions:<jupyter_code>y_pred_tr = m.predict(X_tr)
y_pred_ts = m.predict(X_ts)<jupyter_output><empty_output><jupyter_text>A Better Look at the Predictions**By default, the prediction is the class with the largest probability**<jupyter_code>y_pred_tr<jupyter_output><empty_output><jupyter_text>* If we are interested in the raw probability values...* ...We can call the `predict_proba` method:<jupyter_code>y_prob_tr = m.predict_proba(X_tr)
y_prob_tr[:5]<jupyter_output><empty_output><jupyter_text>* Scikit-learn gives us the predicted probability of both classes* Hence, we get two separate columns  Evaluation**We can evaluate the results using metrics**There are four basic metrics for binary classification:* Number of _True Positives_, i.e. $\mathit{TP} = \sum_{y_i=1} \tilde{f}(x_i; w)$* Number of _True Negatives_, i.e. $\mathit{TN} = \sum_{y_i=0} (1 - \tilde{f}(x_i; w))$* Number of _False Positives_, i.e. $\mathit{FP} = \sum_{y_i=0} \tilde{f}(x_i; w)$* Number of _False Negatives_, i.e. $\mathit{FN} = \sum_{y_i=1} (1 - \tilde{f}(x_i; w))$In all cases $\tilde{f}(x_i; w)$ is the most probable class for the example $x_i$  Evaluation**From these we can derive a few more complex metrics**The model (binary) _accuracy_ is defined as:$$\mathit{ACC} = \frac{TP + TN}{m}$$* I.e. the fraction of examples that is _correctly classified_* The accuracy ranges over the interval $[0, 1]$<jupyter_code>from sklearn.metrics import accuracy_score

print(f'Accuracy on the training set: {accuracy_score(y_tr, y_pred_tr):.3}')
print(f'Accuracy on the test set: {accuracy_score(y_ts, y_pred_ts):.3}')<jupyter_output>Accuracy on the training set: 0.778
Accuracy on the test set: 0.8<jupyter_text>Evaluation**...Or we can plot all basic metrics via a _confusion matrix_**Here's the one for the training set:<jupyter_code>from sklearn.metrics import plot_confusion_matrix
plot_confusion_matrix(m, X_tr, y_tr, display_labels=play.cat.categories, cmap='Blues');<jupyter_output><empty_output><jupyter_text>Evaluation**...Or we can plot all basic metrics via a _confusion matrix_**...And here for the test set:<jupyter_code>from sklearn.metrics import plot_confusion_matrix
plot_confusion_matrix(m, X_ts, y_ts, display_labels=play.cat.categories, cmap='Blues');<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1102803738317757, 'fraction_numerical': 0.00944963655244029, 'mean_word_length': 5.433533734134937, 'pattern_counts': {'":': 0, '<': 66, '<?xml version=': 0, '>': 65, 'https://': 5, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '622585', 'license': 'MIT', 'n_tokens_mistral': 3122, 'n_tokens_neox': 2998, 'n_words': 1182, 'path': '4. Logistic Regression.ipynb', 'repo_name': 'lompabo/linear-models-in-ml', 'size': '9630'}