<jupyter_start><jupyter_text>Supervised Learning: Random ForestsSupervised learning is a machine learning problem where you can train a model to use features to predict a target variable based on prior examples.  This contrasts with unsupervised learning (eg. clustering), in which the data contains many features but no apparent target variable.To get a taste for supervised learning, we'll build a random forest model to predict secondary school student behavior using each student's attributes.  0. Setup<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

from sklearn import preprocessing
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import confusion_matrix<jupyter_output><empty_output><jupyter_text>1. What is supervised learning?In machine learning, you have a problem that needs to be solved, data related to the problem, and a computer.  In supervised learning, you're feeding data to the computer and training it to output the correct answer for a given set of inputs.> For example, let's say you love eating at Jin Ramen but hate waiting for a seat.  Let's also suppose that you have a dataset of 500 visits to Jin Ramen that contains information about the wait time for a table, the time of arrival, day of week, number of people in the party, etc.  You could train a supervised learning model on this dataset to predict wait time based on the other variables.  In the future, whenever you want ramen, you can input the values you know (the "features") into this model to predict wait time (the "target"), then decide whether to visit Jin Ramen or not.Supervised learning usually involves regression and classification tasks:- Regression: the target variable is continuous (eg. predicting the minutes you'll wait for a table)- Classification: the target variable is a discrete value (eg. your waiting limit is 10 mins, so you train the model to output "0" if the wait time will be over 10 mins, and "1" if the wait time will be 10 mins or under)  2. Introduction to random forestRandom forest is a machine learning technique used for classification and regression tasks.  Random forest is an "ensemble" method because one model is actually composed of many decision trees, each of which produces an output; these outputs are then averaged to produce the random forest model's final prediction. 2a. What's a decision tree?A decision tree is a model that predicts the target value by running the inputs through a tree structure.  Decision trees can perform regression and classification tasks, so they're pretty flexible when it comes to machine learning tasks.This image is a simple decision tree for predicting whether a Titanic passenger "survived" or "died" (those are the target values in this classification task):![decision tree](https://upload.wikimedia.org/wikipedia/commons/f/f3/CART_tree_titanic_survivors.png) 2b. Lots of decision trees = ForestA random forest model is composed of many (you decide how many) decision trees.  The idea is that one decision tree may not be accurate, but an entire forest of independent trees will provide higher accuracy because noise is canceled out and signal is strengthened.When creating each tree, the random forest algorithm takes a sample of the data (a bootstrap sample), and produces the best possible decision tree for that sample.  When building the tree, at every node/split, a random sample of features is used.  All this sample-taking allows the forest of trees to identify data features that are truly important, not arbitrary.Once the entire forest is created and ready to make predictions, the random forest model runs the input down every tree, and outputs the average or majority decision of its trees.> Pros:- High accuracy- Fast to train- Can handle many input variables, which relieves you of having to choose a subset of all variables before training the model- Can work with numeric and categorical variables> Cons:- Hard to interpret- Prone to overfitting on noisy datasets- Cannot deal with features or target values that do not exist in the training data  3. Random forest exampleWe're going to predict student alcohol consumption using a dataset about Portuguese students.  Take a moment to skim over the variables [here](http://archive.ics.uci.edu/ml/datasets/STUDENT+ALCOHOL+CONSUMPTION).For the training the random forest model, we're going to use the built-in functions of `scikit-learn`.  Documentation for its random forest classifier is [here](http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html).<jupyter_code># Load in data from the datasets folder
student_port = pd.read_csv('datasets/student-por.csv', sep = ';')
# Take a look at the first 5 rows
student_port.head()
# Get the dataset's dimensions
student_port.shape<jupyter_output><empty_output><jupyter_text>Let's try to predict alcohol consumption based on all or most of other variables (a classification problem).First, we need to perform some data cleaning (highly important for any data task!).  Off the bat, we can see that some of the columns don't have the correct data type.  We want to make sure every column is correctly stored as a numeric or categorical data type.<jupyter_code># What's the current data type of each variable?
student_port.info()
# Most variables should be categorical, so let's list out the variables that are numeric and/or nominal
vars_num = ['age', 'Medu', 'Fedu', 'traveltime', 'studytime', 'failures', 
            'famrel', 'freetime', 'goout', 'Dalc', 'Walc', 'health', 'absences', 'G1', 'G2', 'G3']
# Let's put all the rf data into a new df, rf_data
rf_data = student_port[vars_num].copy()
# But need to convert every categorical value into a number
# Label encoder converts to 0, 1, ... for alphabetically-sorted string values
for col in student_port.columns:
    if col not in vars_num:
        le = preprocessing.LabelEncoder()
        rf_data.loc[:, col] = le.fit_transform(student_port.loc[:, col])
        rf_data.loc[:,col] = rf_data.loc[:, col].astype('category')
# If we want to know the string value corresponding to a numerical category in X, go back to student_port
sorted(student_port['address'].unique())
# How many students in each category of workday alcohol consumption?
student_port['Dalc'].value_counts()
# How many students in each category of weekend alcohol consumption?
student_port['Walc'].value_counts()<jupyter_output><empty_output><jupyter_text>Five categories of student alcohol consumption may be too many to work with (especially since most students fall on the low end of consumption).  Since it's more interesting to divide students into light drinkers and heavy drinkers, let's create two categories of alcohol usage: 1-2 and 3-5.<jupyter_code># Map out the old level to the new level
# 0 will be "light drinkers", 1 will be "heavy drinkers"
alc_mapping = {1: 0, 2: 0, 3: 1, 4: 1, 5:1}
# Create new columns for the mapped data
rf_data['Dalc_mapped'] = rf_data['Dalc'].map(alc_mapping)
rf_data['Walc_mapped'] = rf_data['Walc'].map(alc_mapping)<jupyter_output><empty_output><jupyter_text>Modeling time!Supervised learning involves these general steps:1. Split the data into training and test datasets.  Usually the split is anywhere from 60%/40% to 80%/20%.2. Train a model on the training set.3. Apply the model on the test set and compare the predicted results against the actual values to evaluate the model's performance.    - As needed, iterate on the model to improve its performance (beware of overfitting!)    - Look into the model to understand what it's doing, and gain some insight on your dataset.  a) Decide on the data you want, and split into training and test sets<jupyter_code># Specify the columns you want as features and as the target
features = rf_data.columns
features = features.drop(['G1', 'G2', 'Mjob', 'Fjob', 'Dalc', 'Walc', 'Dalc_mapped', 'Walc_mapped'])
target = 'Walc_mapped'
# This is the full dataset of predictor variables
X = rf_data[features]
# This is the column of target (to be predicted) variables
y = rf_data[target]
# Split into training (70%) and test (30%) sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.3)
X_train.shape, X_test.shape, y_train.shape, y_test.shape<jupyter_output><empty_output><jupyter_text>b) Train the random forest model<jupyter_code># Initialize the random forest model with your desired parameters
rf_model = RandomForestClassifier(n_estimators = 100, random_state = 2017)
rf_model = rf_model.fit(X_train, y_train)<jupyter_output><empty_output><jupyter_text>c) Evaluate the model How well does the model perform?<jupyter_code># Model accuracy: the number of correct predictions divided by the number of predictions
rf_model.score(X_test, y_test)
# Apply the model on the test data to produce predictions
y_pred = rf_model.predict(X_test)
# A confusion matrix helps you evaluate the predictions (rows) against the actual values (columns)
confusion_matrix(y_test, y_pred)
# This is a pretty visualization of the confusion matrix
# From http://scikit-learn.org/stable/auto_examples/model_selection/plot_confusion_matrix.html
import itertools
def plot_confusion_matrix(cm, classes, title='Confusion matrix', cmap=plt.cm.Blues):
    plt.imshow(cm, interpolation='nearest', cmap=cmap)
    plt.title(title)
    plt.colorbar()
    tick_marks = np.arange(len(classes))
    plt.xticks(tick_marks, classes, rotation=45)
    plt.yticks(tick_marks, classes)

    thresh = cm.max() / 2.
    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
        plt.text(j, i, cm[i, j],
                 horizontalalignment="center",
                 color="white" if cm[i, j] > thresh else "black")

    plt.tight_layout()
    plt.ylabel('True label')
    plt.xlabel('Predicted label')
plot_confusion_matrix(confusion_matrix(y_test, y_pred), classes = sorted(y.unique()))<jupyter_output><empty_output><jupyter_text>Which features did the model deem most "important" for predicting the target variable?*Note:* Random forest models are considered "black box" models because they're hard to interpret.  If our model only had one decision tree instead of an entire forest of them, then we could examine the tree and see the exact criteria used to produce a prediction for any input.<jupyter_code># Show feature importances, highest first
pd.DataFrame({'feature': X.columns, 'importance': rf_model.feature_importances_}).sort_values(
    by = 'importance', ascending = False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07277882797731569, 'fraction_numerical': 0.006710775047258979, 'mean_word_length': 5.084531339850489, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 37, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '177620', 'license': 'MIT', 'n_tokens_mistral': 2951, 'n_tokens_neox': 2807, 'n_words': 1359, 'path': '4. Supervised Learning.ipynb', 'repo_name': 'cerrno/intro-to-datascience', 'size': '10580'}