<jupyter_start><jupyter_text>Learning Unit 2 - Making predictions 101 - Example<jupyter_code># Imports, feel free to ignore 
import pandas as pd 

from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import accuracy_score

import matplotlib.pyplot as plt
% matplotlib inline 

from utils import plot_boundaries, get_ying_yang, scatterplot, plot_tree

# In the odd chance that you have problems with graphviz, use the following instead: 
# from utils_safety import plot_boundaries, get_ying_yang, scatterplot<jupyter_output><empty_output><jupyter_text>1 - Binary Classification  You might have noticed something about the exercise and examples: we are interested in _"whether a mushroom is poisonous, or not"_, and _"Whether a Titanic passenger survived, or not"_. This type of problem, where you are trying to predict which of 2 classes an observation belongs to, is called [Binary Classification](https://en.wikipedia.org/wiki/Binary_classification). To demonstrate this, we will be using an extremely simple dataset:<jupyter_code>simple = get_ying_yang()   # often df is used as shorthand for dataframe<jupyter_output><empty_output><jupyter_text>It just has 2 features, `a`, and `b`, and we will be trying to predict `c`.<jupyter_code>simple.head(2)<jupyter_output><empty_output><jupyter_text>A bit of nomenclature: > **"target"**: that which we are trying to predict (in this case `c`). We often represent the target as `y`> **"features"**: that which we use to predict the target (in this case `a` and `b` are both features). We represent the features as `X`  > The **training set** is the data that will be used to train the model. Both `X_train` and `y_train` are part of the training set. > The **test set** is the data which is set aside to evaluate the model. It should always be different from the training set (we'll break that rule here to illustrate). As before, `X_test` and `y_test` will, repectively, be the features and target of the test set.  ![title](https://github.com/PedroGFonseca/how_to_learn_ds_meetup/raw/e553355800341008d524217912fe4ac0f2e07257/images/test_train.png) In our case, we will call `a` and `b` features, and say that `c` is the target we are trying to predict<jupyter_code>X_train = simple[['a', 'b']]  # <-- the double brackets are because we are passing a list 
y_train = simple['c']<jupyter_output><empty_output><jupyter_text>Because we are only in 2 dimentions, we can even "cheat" and look at the data with a quick scatterplot:<jupyter_code>scatterplot(simple, 'a', 'b', 'c', small=True)<jupyter_output><empty_output><jupyter_text>Pretty simple, our classifier just needs to figure out the following: > the area to the "left" is 1  > the area to the "right" is 0    3 - Working with scikit classifiers  There are many ways to do binary classification, but here we will focus on just 3 algorithms:- [Logistic Regression](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) - [Decision Tree Classifier](http://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html)- [KNN Classifier ("K nearest neighbours")](http://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.htmlsklearn.neighbors.KNeighborsClassifier) To make predictions, we will be using the [scikit-learn library](http://scikit-learn.org/stable/index.html). It is generally the standart for getting started with Machine Learning in Python, and a number of production systems use it for its simplicity and excellent community.  The API is extremely simple. When you have some features, and a target, you can fit a model by doing the following:  > `model.fit(features, target)` So what does it mean to `fit` (also known as training) a model? Essentially, you are showing it some training data (some features, and a target), and allowing it to set its parameters (there are exceptions, but we'll cover this later) so that it can predict new observations.<jupyter_code>scatterplot(simple, 'a', 'b', 'c', small=True)<jupyter_output><empty_output><jupyter_text>The easiest classifier to think about might be the [Logistic Regression](scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html).   Using `fit`, `predict`, and `predict_proba` To ilustrate the use of the scikit API, we will use the Logistic Regression. We start by instanciating a Logistic regression. Nothing much happens at this stage:   Instantiating a model:<jupyter_code>logit = LogisticRegression() # just instanciating!<jupyter_output><empty_output><jupyter_text>Fitting a model:  Now we `fit` the model, passing it the `X_train`, and the `y_train`<jupyter_code>logit.fit(X=X_train, 
          y=y_train)<jupyter_output><empty_output><jupyter_text>The logit is now fit, meaning that it has its parameters set, and is ready to make new predictions  Making predictions  We can now make predictions, using the `predict` method *Note: remember, we should never predict the training set, we're doing that here just to illustrate*<jupyter_code>predictions = logit.predict(X_train)  # notice that we dont' need the target now<jupyter_output><empty_output><jupyter_text>What do these predictions look like?<jupyter_code>print(predictions)<jupyter_output>[0 1 1 1 1 1 0 1 0 1 0 1 1 0 0 0 0 0 0 1 0 1 0 1 0 1 0 1 1 1 1 1 1 1 0 0 0
 0 0 1 1 1 0 0 1 0 0 0 0 1 1 1 0 0 0 0 0 0 0 0 0 0 1 1 1 1 0 1 0 1 0 0 0 1
 1 1 0 1 1 1 0 0 1 0 1 0 1 1 1 1 1 0 0 1 0 1 1 1 0 0]<jupyter_text>It can also predict probabilities, instead of predicting just 0 or 1:<jupyter_code>proba_predictions = logit.predict_proba(X_train) # nomenclature becomes "predict_proba" 

print(proba_predictions[0:10])  # printing only the first 10 predictions<jupyter_output>[[ 0.91649855  0.08350145]
 [ 0.06755059  0.93244941]
 [ 0.22440009  0.77559991]
 [ 0.49094284  0.50905716]
 [ 0.08910541  0.91089459]
 [ 0.29532752  0.70467248]
 [ 0.53795556  0.46204444]
 [ 0.04749218  0.95250782]
 [ 0.73058308  0.26941692]
 [ 0.24125754  0.75874246]]<jupyter_text>You will notice that each prediction is composed of two values, which add to one. That's because they are the _"probability of being 0"_ and the _"probability of being 1"_ To simply get the probability of being 1, we can do the following:<jupyter_code>proba_1 = proba_predictions[:,1]   # the syntax is a bit ugly, but it essentially means "all entries, item 1"<jupyter_output><empty_output><jupyter_text>Let's observe the first 10 again:<jupyter_code>proba_1[0:10]<jupyter_output><empty_output><jupyter_text>So essentially if you threashold these at 0.5, you get the predictions.   Observing predictions  So how well did our classifier do? Let's use a quick utility function, `plot_boundaries` to get an idea. Naturally this will only work in 2 dimentions.<jupyter_code>plot_boundaries(X_train, y_train, logit)<jupyter_output><empty_output><jupyter_text>Not particularly great. Too linear.   Decision Tree Classifier  Trees are slightly more powerful models than the logistic regressions, because they aren't forced to be linear. Let's see how a tree predicts our dataset:<jupyter_code>dt = DecisionTreeClassifier(max_depth=2)
dt.fit(X_train, y_train)
plot_boundaries(X_train, y_train, dt)<jupyter_output><empty_output><jupyter_text>Slightly smarter. What if we change the hyper parameters?<jupyter_code>dt = DecisionTreeClassifier(max_depth=6, min_samples_split=10)
dt.fit(X_train, y_train)
plot_boundaries(X_train, y_train, dt)<jupyter_output><empty_output><jupyter_text>KNN Classifier  K-nearest-neighbours rely on the idea that an observation can be predited by its `K` closest neighbours:<jupyter_code>knn = KNeighborsClassifier(n_neighbors=5)
knn.fit(X_train, y_train)
plot_boundaries(X_train, y_train, knn)<jupyter_output><empty_output><jupyter_text>What if we make the `n_neighors` (number of neighbours) smaller?<jupyter_code>knn = KNeighborsClassifier(n_neighbors=1)
knn.fit(X_train, y_train)
plot_boundaries(X_train, y_train, knn)<jupyter_output><empty_output><jupyter_text>Here we observe something interesting. We got more of our training set right, but we can tell by looking at it that something is wrong. How can we avoid this?   3 - Overfitting and train-test-split  Alright, so up to now we've been predicting data the model has already seen, but what happens on data it doesn't know?  A quick reminder of how train and test sets work: ![title](https://github.com/PedroGFonseca/how_to_learn_ds_meetup/raw/e553355800341008d524217912fe4ac0f2e07257/images/test_train.png) Let's break our dataset into training and test sets:<jupyter_code># Reminder:
# X_train: features to train on 
# y_train: targets (actual outcomes) to train on 
# X_test: features to test on 
# y_test: targets (actual outcomes) to test on 

X_train, X_test, y_train, y_test = train_test_split(simple[['a', 'b']], 
                                                    simple['c'], 
                                                    test_size=0.33, 
                                                    random_state=42)<jupyter_output><empty_output><jupyter_text>Let's go back to our KNN, and train on part of the dataset, and observe how it predicts the other part:<jupyter_code>knn = KNeighborsClassifier(n_neighbors=1)
knn.fit(X_train, y_train)
plot_boundaries(X_test, y_test, knn)<jupyter_output><empty_output><jupyter_text>Now we start to notice that there is some overfit. That red "entrance" was clearly something it saw on the training set, and learned "too much" from it. Let's increase the `n_neigbors` again:<jupyter_code>knn = KNeighborsClassifier(n_neighbors=5)
knn.fit(X_train, y_train)
plot_boundaries(X_test, y_test, knn)<jupyter_output><empty_output><jupyter_text>Much better! We can fight overfitting with sensible setting of hyper parameters.   4 - Evaluating our results  Hopefully you now have an intuition as to why it is important to always train on your traninig set, and test on your test set. One trivial (but often too basic) way to evaluate your model is using accuracy (literally percentage of times you get the answer right):<jupyter_code># toy example 
predicted = [1, 0, 1]
true = [1, 0, 0]
print(accuracy_score(y_true=true, y_pred=predicted))<jupyter_output>0.666666666667<jupyter_text>Let's evaluate how well each of our classifiers does on unseen data:   Logit<jupyter_code>model = LogisticRegression()
model.fit(X_train, y_train)
predictions = model.predict(X_test)
accuracy = accuracy_score(y_true=y_test, y_pred=predictions)
print('Accuracy for %s was %0.2f' % ('Logistic Regression', accuracy))<jupyter_output>Accuracy for Logistic Regression was 0.85<jupyter_text>The pattern is always the same, so we might as well make this a function:   Decision Tree Classifier<jupyter_code>model = DecisionTreeClassifier(max_depth=5, min_samples_split=5)
model.fit(X_train, y_train)
predictions = model.predict(X_test)
accuracy = accuracy_score(y_true=y_test, y_pred=predictions)
print('Accuracy for %s was %0.2f' % ('Decision Tree Classifier', accuracy))<jupyter_output>Accuracy for Decision Tree Classifier was 0.88<jupyter_text>KNN<jupyter_code>model = KNeighborsClassifier(n_neighbors=5)
model.fit(X_train, y_train)
predictions = model.predict(X_test)
accuracy = accuracy_score(y_true=y_test, y_pred=predictions)
print('Accuracy for %s was %0.2f' % ('K nearest neighbours', accuracy))<jupyter_output>Accuracy for K nearest neighbours was 0.97<jupyter_text>5 - Using this on our mushrooms  We will start by importing our "`prepared_mushrooms`":<jupyter_code>data = pd.read_csv('data/prepared_mushrooms.csv')<jupyter_output><empty_output><jupyter_text>Just a quick reminder of what the data looks like:<jupyter_code>data.head(3)<jupyter_output><empty_output><jupyter_text>How many rows and columns?<jupyter_code>print('Our mushroom dataset has %0.0f rows and %0.0f columns'  % (data.shape[0], data.shape[1]))
target = 'is_poisonous'

features = list(data.columns)
features.remove(target)<jupyter_output><empty_output><jupyter_text>Note: those who are comfortable with python should use a list comprehension, which is much more elegant:      `features = [column for column in data.columns if column != target]`<jupyter_code>X_train, X_test, y_train, y_test = train_test_split(data[features], 
                                                    data[target], 
                                                    test_size=0.33, 
                                                    random_state=42)<jupyter_output><empty_output><jupyter_text>What is the size of our new datasets?<jupyter_code>print('X_train shape:', X_train.shape)
print('y_train shape:', y_train.shape)
print('X_test shape:', X_test.shape)
print('y_test shape:', y_test.shape)<jupyter_output>X_train shape: (5443, 28)
y_train shape: (5443,)
X_test shape: (2681, 28)
y_test shape: (2681,)<jupyter_text>Let's fit some models!  Logistic Regression:<jupyter_code>logit = LogisticRegression()
logit.fit(X_train, y_train)
predictions = logit.predict(X_test)
accuracy = accuracy_score(y_true=y_test, y_pred=predictions)
print('Accuracy for %s was %0.4f' % ('Logistic Regression', accuracy))<jupyter_output>Accuracy for Logistic Regression was 0.9705<jupyter_text>K Nearest Neighbours:<jupyter_code>knn = KNeighborsClassifier(n_neighbors=2)
knn.fit(X_train, y_train)
predictions = knn.predict(X_test)
accuracy = accuracy_score(y_true=y_test, y_pred=predictions)
print('Accuracy for %s was %0.4f' % ('K nearest neighbours', accuracy))<jupyter_output>Accuracy for K nearest neighbours was 0.9623<jupyter_text>Decision Tree Classifier:<jupyter_code>decision_tree_classifier = DecisionTreeClassifier(max_depth=3, min_samples_leaf=100)
decision_tree_classifier.fit(X_train, y_train)
predictions = decision_tree_classifier.predict(X_test)
accuracy = accuracy_score(y_true=y_test, y_pred=predictions)
print('Accuracy for %s was %0.4f' % ('Decision Tree Classifier', accuracy))<jupyter_output>Accuracy for Decision Tree Classifier was 0.9482<jupyter_text>Obviously we can and should try to tune the hyper parameters (`max depth`, `min_samples_leaf`, etc), but we'll do that in the exercise. You might have noticed that mushroons are relatively easy to predict.   Bonus bit: interpreting models  One easy model to interpret (generally good to get first glances at datasets) is the Decision Tree Classifier:> "Left" means True     > The "value" is [_"how many were in fact 0"_, _"how many were in fact 1"_]  > The "gini" is a measure of "impurity". If the node has only 1 type, the gini is 0  > The "samples" is simply how many are in that node<jupyter_code>plot_tree(tree_classifier=decision_tree_classifier, 
          X_train=X_train, 
          class_names=['Safe', 'Poisonous'])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08811254711900915, 'fraction_numerical': 0.03325255788906839, 'mean_word_length': 5.368195456493785, 'pattern_counts': {'":': 1, '<': 128, '<?xml version=': 0, '>': 138, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 27, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '34', 'dir': 'jupyter-structured-clean-dedup', 'id': '424914', 'license': 'MIT', 'n_tokens_mistral': 5218, 'n_tokens_neox': 4739, 'n_words': 1588, 'path': 'learning_unit_2/Learning Unit 2 (Making predictions 101) - Example .ipynb', 'repo_name': 'LDSSA/data_science_101_tutorial', 'size': '14856'}