<jupyter_start><jupyter_text>StackingEnsemble learning helps improve machine learning results by combining several base models. This approach often leads to better predictive performance compared to a single base model alone. Ensemble learning has been proved to be very effective and used extensively in many machine learning competitions, such as the Netflix prize competition and Kaggle.Like bagging (Random Forest) and boosting (Gradient Boosted Trees) covered in the last lab, stacked generalization (stacking) is another ensemble learning method, which introduces a different way of combining multiple base models to achieve better model prediction. The idea behind stacking is fairly simple: instead of using trivial functions (such as hard voting) to aggregate the predictions of the base learners, we train a model to perform this aggregation, called meta learner or blender. This idea is captured in the following diagram.In this lab assignment, you will implement the stacking method and apply it on several machine learning benchmark datasets.<jupyter_code># import packages
%matplotlib inline
import numpy as np
import matplotlib
import matplotlib.pyplot as plt
from matplotlib.colors import ListedColormap
import seaborn as sns
import pandas as pd
from sklearn.datasets import make_moons
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.preprocessing import PolynomialFeatures
from sklearn.neighbors import KNeighborsClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC
from sklearn.svm import SVR
from sklearn.tree import DecisionTreeClassifier
from sklearn.tree import DecisionTreeRegressor
from sklearn.metrics import accuracy_score
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import GridSearchCV

# make this notebook's output stable across runs
np.random.seed(0)
# helper functions used in this lab
def plot_decision_boundary(clf, X, y, axes=[-1.5, 2.5, -1, 1.5], alpha=0.5, contour=True):
    """
    Plot the decision boundary of a learnt classifier
    """
    x1s = np.linspace(axes[0], axes[1], 100)
    x2s = np.linspace(axes[2], axes[3], 100)
    x1, x2 = np.meshgrid(x1s, x2s)
    X_new = np.c_[x1.ravel(), x2.ravel()]
    y_pred = clf.predict(X_new).reshape(x1.shape)
    custom_cmap = ListedColormap(['#fafab0','#9898ff','#a0faa0'])
    plt.contourf(x1, x2, y_pred, alpha=0.3, cmap=custom_cmap)
    if contour:
        custom_cmap2 = ListedColormap(['#7d7d58','#4c4c7f','#507d50'])
        plt.contour(x1, x2, y_pred, cmap=custom_cmap2, alpha=1)
    plt.plot(X[:, 0][y==0], X[:, 1][y==0], "yo", alpha=alpha)
    plt.plot(X[:, 0][y==1], X[:, 1][y==1], "bs", alpha=alpha)
    plt.axis(axes)
    plt.xlabel(r"$x_1$", fontsize=18)
    plt.ylabel(r"$x_2$", fontsize=18, rotation=0)
    
def evaluate_model(model, X, y):
    # make prediction and evaluate the model performance on test data
    z_prob = model.predict_proba(X)[:,1]

    print("model ROC AUC: {}".format(roc_auc_score(y, z_prob)))<jupyter_output><empty_output><jupyter_text>Synthetic Moon DatasetIn this lab, we will classify a moon shaped synthetic dataset with two features (_x1_ and _x2_) and two classes (0 or 1) with some added noise. We use this synthetic dataset because it's easier to help demonstrate the pros and cons of different SVM models covered in this lab.<jupyter_code># load the moon train and test data from CSV files
train = pd.read_csv("moon_train.csv")
test = pd.read_csv("moon_test.csv")

train_x = train.iloc[:,0:2]
train_y = train.iloc[:,2]

test_x = test.iloc[:,0:2]
test_y = test.iloc[:,2]

print("Number of train data: {}".format(len(train_y)))
print("Number of test data: {}".format(len(test_y)))
# plot the train dataset
plt.scatter(train_x.x1, train_x.x2, s=40, c=train_y, cmap=plt.cm.Spectral)
plt.title('Scatter plot of the train dataset')
plt.xlabel('x1')
plt.ylabel('x2')
# plot the test dataset
plt.scatter(test_x.x1, test_x.x2, s=40, c=test_y, cmap=plt.cm.Spectral)
plt.title('Scatter plot of the test dataset')
plt.xlabel('x1')
plt.ylabel('x2')<jupyter_output><empty_output><jupyter_text>Stacking Implementation**Coding Part 1: train three base learners separately and evaluate their performance on the test set. We later compare our own implementation of stacking against the performance of these base learners in terms of AUC.**   K nearest neighbors- Train a K nearest neighbors model- Plot the decision boundary- Evaluate the model performance on the test set.<jupyter_code># TODO: train a KNN model using grid search with cross validation with the following hyper-parameters
parameters = {
    "n_neighbors": [2, 4, 8]
}
knn_model = # TODO: train a KNN model
knn_model.fit(train_x, train_y)
# plot the decision boundary
plot_decision_boundary(knn_model, train_x.values, train_y.values)
# make prediction and evaluate the model performance on test data
evaluate_model(knn_model, test_x, test_y)<jupyter_output><empty_output><jupyter_text>Linear SVM- Train a linear SVM model- Plot the decision boundary- Evaluate the model performance on the test set.<jupyter_code># TODO: train a linear SVM model using grid search with cross validation with the following hyper-parameters
parameters = {
    "C": [0.01, 0.1, 1, 10, 100]
}
linear_svc_model = # TODO: train a linear SVM model
linear_svc_model.fit(train_x, train_y)
# plot the decision boundary
plot_decision_boundary(linear_svc_model, train_x.values, train_y.values)
# make prediction and evaluate the model performance on test data
evaluate_model(linear_svc_model, test_x, test_y)<jupyter_output><empty_output><jupyter_text>Decision Tree- Train a decision tree- Plot the decision boundary- Evaluate the model performance on the test set.<jupyter_code># TODO: train a decision tree model using grid search with cross validation with the following hyper-parameters
parameters = {
    'criterion': ['gini', 'entropy'], 
    'max_depth': [1, 2, 4, 8], 
    'min_samples_split': [0.01, 0.05, 0.1, 0.2, 0.4, 0.8]
}

dt_model = # TODO: train a decision tree model
dt_model.fit(train_x, train_y)
# plot the decision boundary
plot_decision_boundary(dt_model, train_x.values, train_y.values)
# make prediction and evaluate the model performance on test data
evaluate_model(dt_model, test_x, test_y)<jupyter_output><empty_output><jupyter_text>Stacking Implementation**Coding Part 2: implement the stacking algorithm for a binary classification task.**   Two-layer Stacking modelGiven a two-layer stacking architecture with $N$ base learners in the first cell of this notebook, the algorithm works as follows: - Step 1: the training set is split into two subsets: _Fold 1_ and _Fold 2_.- Step 2: the first subset _Fold 1_ is used to train the $N$ base models in the first layer.- Step 3: the first layer base learners are used to make predictions on the second set.- Step 4: for each instance in the second set _Fold 2_, there are $N$ predicted values, one from each base learner. Now, we create a new training set using these predicted values as input features together with the original label as target. - Step 5: train the meta learner (or blender) on this new training dataset so it learns to predict the target value based on the predictions from the base learner in the first layer.Now, given a new instance in the test set, we first use the base learners in the first layer to make predictions, and then use these predictions to feed to the meta learner to get the final prediction.   Multiple-layer StackingWe can actually extend the idea of stacking to more than 2 layers. For example in a 3-layer stacking framework, we can split the training dataset into 3-fold where the first fold is used to train the base learners in the first layers, the second fold is used to train the learners in the middle layer, and the last fold is used to train the final blender. Same as the 2-layer stacking, learners in the middle and last layer use the predictions from learners from the previous layer. Once training is done, we can make a prediction for a new instance by going through each layer sequentially as before. In this exercise, you will implement a two-layer stacking algorithm with the following components:- Three base learners in the first layer, including Logistic Regression, Support Vector Machine, and Decision Tree. - A logistic regression model as the meta learner or blender.<jupyter_code>class MyStacking:
    """
    Stacking
    """
    def __init__(self):
        self.model_knn = None
        self.model_svc = None
        self.model_dt = None
        self.blender = None

    def train(self, x, y):
        """
        TODO: train the stacking model with {KNN, SVM, decision tree} as the base learners 
              and a logistic regression model as the blender
        """
        #--------------------------------------------
        # TODO: random split the training set x into two folds
        #--------------------------------------------
        idx = np.random.choice(x.index, size=len(x)//2, replace=False)
        x_fold1 = # TODO
        x_fold2 = # TODO
        y_fold1 = # TODO
        y_fold2 = # TODO
        
        #--------------------------------------------
        # TODO: learn the base learners using data in fold 1
        #--------------------------------------------
        # learn the KNN base learner
        print("learn the K nearest neighbor base learner...")
        parameters = {
            "n_neighbors": [2, 4, 8]
        }
        self.model_knn = # TODO
        self.model_knn.fit(x_fold1, y_fold1)

        # learn the SVM base learner
        print("learn the support vector machine base learner...")
        parameters = {
            "C": [0.01, 0.1, 1, 10, 100]
        }
        self.model_svc = # TODO
        self.model_svc.fit(x_fold1, y_fold1)

        # learn the decision tree base learner
        print("learn the decision tree base learner...")
        parameters = {
            'criterion': ['gini', 'entropy'], 
            'max_depth': [1, 2, 4, 8], 
            'min_samples_split': [0.01, 0.05, 0.1, 0.2, 0.4, 0.8]
        }
        self.model_dt = # TODO
        self.model_dt.fit(x_fold1, y_fold1)

        #--------------------------------------------
        # TODO: learn the blender using predictions on fold2 from the base learners as features
        #--------------------------------------------
        # make predictions using the base learns to generate the training data for the blender
        x = # TODO
        y = # TODO
                
        print("learn the blender...")
        parameters = {
            "penalty": ["l1", "l2"], 
            "C": [0.001, 0.01, 0.1, 1, 10]
        }
        self.blender = # TODO
        self.blender.fit(x, y)

    def predict_proba(self, x):
        """
        TODO: Make predictions (probabilities) with the trained stacking model.
        """
        if self.blender == None:
            sys.exit("The model has not been trained yet. Please call train() first. Exiting...")

        # predictions from the base learner 
        base_model_predictions = # TODO
        
        # predictions from the blender
        probabilities = # TODO
        return probabilities

    def predict(self, x):
        """
        Make predictions (the class/label) with the trained stacking model.
        """
        probabilities = self.predict_proba(x)[:,1]
        prediction = pd.Series([1] * len(probabilities))
        prediction[probabilities < 0.5] = 0

        return prediction.values
# train random forest classifier
my_stacking = MyStacking()
my_stacking.train(train_x, train_y)
# plot the decision boundary
plot_decision_boundary(my_stacking, train_x.values, train_y.values)
# make prediction and evaluate the model performance on test data
evaluate_model(my_stacking, test_x, test_y)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10542753439689373, 'fraction_numerical': 0.017388368363298726, 'mean_word_length': 3.324087591240876, 'pattern_counts': {'":': 6, '<': 26, '<?xml version=': 0, '>': 25, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '150125', 'license': 'MIT', 'n_tokens_mistral': 3577, 'n_tokens_neox': 3362, 'n_words': 1336, 'path': 'module05/resources/HW05.ipynb', 'repo_name': 'kegedy/MLEARN520', 'size': '11847'}