<jupyter_start><jupyter_text>Ensemble Methods AppliedAgenda:- Review code for Voting Classifier, Bagging Classifier, and Random Forest- Practice finding optimal hyperparameter for  Random Forest with gridsearch  Import and Prep Titanic dataset<jupyter_code>import pandas as pd
import numpy as np
from sklearn import metrics
from sklearn.ensemble import BaggingClassifier

# Read in data and split data to be used in the models
titanic = pd.read_csv('https://raw.githubusercontent.com/learn-co-students/nyc-ds-033020-lectures/master/Mod_3/decision_trees/cleaned_titanic.csv', index_col='PassengerId')


# Create matrix of features
X = titanic.drop('Survived', axis = 1) # grabs everything else but 'Survived'

# Create target variable
y = titanic['Survived'] # y is the column we're trying to predict

# Create a list of the features being used in the 
feature_cols = X.columns
# Use x and y variables to split the training data into train and test set then scale that data

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=1)

scaler = StandardScaler()  
scaler.fit(X_train)

X_train = scaler.transform(X_train)  
X_test = scaler.transform(X_test)<jupyter_output><empty_output><jupyter_text>Fit a KNN model<jupyter_code>from sklearn.neighbors import KNeighborsClassifier
knn = KNeighborsClassifier(n_neighbors=9)
knn.fit(X_train, y_train)

knn_preds = knn.predict(X_test)

knn_f1 = metrics.f1_score(y_test, knn_preds)


print(knn_f1)<jupyter_output>0.7975460122699386<jupyter_text>Fit a Logistic Regression model<jupyter_code>from sklearn.linear_model import LogisticRegression
lr = LogisticRegression(class_weight='balanced')
lr.fit(X_train, y_train)
lr_preds = lr.predict(X_test)

lr_f1 = metrics.f1_score(y_test, lr_preds)

print(lr_f1)<jupyter_output>0.8066298342541436<jupyter_text>Fit a Decision Tree Classifier<jupyter_code>from sklearn.tree import DecisionTreeClassifier
dtc = DecisionTreeClassifier(max_depth=5, class_weight='balanced')

dtc.fit(X_train, y_train)

dtc_preds  = dtc.predict(X_test)

dtc_f1 = metrics.f1_score(y_test, dtc_preds)

print(dtc_f1)<jupyter_output>0.8047337278106509<jupyter_text>Combine three models using Voting Classifier<jupyter_code>from sklearn.ensemble import VotingClassifier<jupyter_output><empty_output><jupyter_text>For the estimators, we must provide a list of tuples. The first value in the tuple is is the name given to the model/estimator in the second value. SKlearn requires this because there is additional functionality where you can access information about the specific models, so you need to name the models to access them later.<jupyter_code>voting_clf = VotingClassifier(
                estimators=[('logreg', lr), ('knneighbors', knn), ('decisiontree', dtc)], 
                voting='hard')
voting_clf.predict(X_test)


voting_clf.fit(X_train, y_train)

vc_preds = voting_clf.predict(X_test)

vc_f1 = metrics.f1_score(y_test, vc_preds)

print(vc_f1)
#This is harder to interpret. No way to say what features contribute. KNN already does that too<jupyter_output><empty_output><jupyter_text>Use a voting classifier with multiple Logistic regression models<jupyter_code>C_param_range = [0.001,0.01,0.1,1,10]
titles = ['lr_0_001', 'lr_0_01', 'lr_0_1', 'lr_1', 'lr_10']

params = dict(zip(titles, C_param_range)) 
models = {}
params
table = pd.DataFrame(columns = ['C_parameter','F1'])
table['C_parameter'] = C_param_range
j = 0

for k , v  in params.items():
    
    # Create model using different value for c  
    lr = LogisticRegression(penalty = 'l2', C = v, random_state = 1, class_weight='balanced')
    
    #save the model to a dictionary to use later in our voting classifiers
    models[k]= lr
    
    #the steps below this point are unnecessary in order to create a voting classifier, 
    #but it is easy to fit the model and see how performance changes for different levels of regularization
    lr.fit(X_train, y_train)
    
    # Predict using model
    y_preds = lr.predict(X_test)

    # Saving accuracy score in table
    table.iloc[j,1] = metrics.f1_score(y_test, y_preds)
    j += 1
#review performance for different levels of C
table

#invesitgate the models D=dictionary
list(models.items())<jupyter_output><empty_output><jupyter_text>Now that we have programmatically created multiple logistic regression models, let's use them in an ensemble model<jupyter_code>lr_voting = VotingClassifier(estimators=list(models.items()), 
                              voting='hard')

lr_voting.fit(X_train, y_train)

lrv_preds = lr_voting.predict(X_test)

lrv_f1 = metrics.f1_score(y_test, lrv_preds)

print(lrv_f1)<jupyter_output>0.8044692737430168<jupyter_text>Fit a Bagging Classifier for a Logistic Regression model.<jupyter_code># Prevent overfitting
X_train.shape
bc_lr = BaggingClassifier(
            base_estimator=LogisticRegression(class_weight='balanced'), 
            n_estimators= 100, #100 samples
            max_samples= 0.8, #grab random samples of this size
            max_features= 6, #getting a max of 6 features
            oob_score= True #out of bag score
                )
bc_knn = BaggingClassifier(
            base_estimator=KNeighborsClassifier(n_neighbors=9), 
            n_estimators= 100,
            max_samples= 0.8,
            max_features= 6,
            oob_score= True
                )
bc_dtc = BaggingClassifier(
            base_estimator=DecisionTreeClassifier(random_state = 1,max_depth=5, class_weight='balanced'), 
            n_estimators= 100,
            max_samples= 0.8,
            max_features= 6,
            oob_score= True
                )
voting_clf_bagged = VotingClassifier(
                estimators=[('bagged_logreg', bc_lr), ('bagged_knneighbors', bc_knn), ('bagged_decisiontree', bc_dtc)], 
                voting='hard')

voting_clf_bagged.fit(X_train, y_train)

vc_bagged_preds = voting_clf_bagged.predict(X_test)

vc_bagged_f1 = metrics.f1_score(y_test, vc_bagged_preds)

print(vc_bagged_f1)
bc_lr.fit(X_train, y_train)


# Use the oob_score to get some idea of how the model performs on a validation set

bc_lr.oob_score_
# See how the model performs on the test set

bc_lr_preds = bc_lr.predict(X_test)

bc_lr_f1 = metrics.f1_score(y_test, bc_lr_preds)

print(bc_lr_f1)<jupyter_output>0.7826086956521741<jupyter_text>***What is the difference in the `VotingClassifier` algorithm and the `BaggingClassifier` algorithm?*** Your answer: The VotingClassifier runs through multiple models before returning the best result. Bagging each model has different samples. Bagging samples/features can be changed **What is the difference between a BaggingClassifier that uses a decision tree as the base estimator and a Random Forest Classifier?** A random forest classifier will take a sample of features at each node, where as a bagging classifier will take a sample of features at to use for the whole model.   Fitting a Random Forest Classifier<jupyter_code># Instantiate the classifier using 100 trees
from sklearn.ensemble import RandomForestClassifier
rfc = RandomForestClassifier(random_state = 1, n_estimators=100, max_depth=2, max_features=4)
#let's look at all the different default features
rfc
#fit the model to the training data
rfc.fit(X_train, y_train)
#use the fitted model to predict on the test data
rfc_preds = rfc.predict(X_test)

rfc_f1 = metrics.f1_score(y_test, rfc_preds)

# checking accuracy on the test data
print('Test F1 score: ', rfc_f1)<jupyter_output>Test F1 score:  0.7074829931972789<jupyter_text>***Increase the number of trees and see how the model performs***  GridsearchCV with Random ForestLet's use grid search to identify the best tuning parameters to use for a random forest model.<jupyter_code>from sklearn.model_selection import GridSearchCV
RandomForestClassifier()
#create a dictionary of all the parameters you want to tune
param_grid = {
    'n_estimators': [300, 500, 700],
    'max_leaf_nodes': [10, 20, 30],
    'min_samples_leaf': list(range(1, 22, 5)),
    'rfr__max_features': ['range', None, 5, 6]
}
#create a grid search object and fit it to the data

grid_tree=GridSearchCV(RandomForestClassifier(), param_grid, cv=5, scoring='f1', verbose=1, n_jobs=-1)
grid_tree.fit(X_train, y_train)
# when you split the nodes what are you trying to do
    # create homogeanity. So if you just let it go, it'll divide until it's completely split
### Identify the best params 



# Single best score achieved across all params (min_samples_split)
print(grid_tree.best_score_)

# Dictionary containing the parameters (min_samples_split) used to generate that score
print(grid_tree.best_params_)

# Actual model object fit with those best parameters
# Shows default parameters that we did not specify
print(grid_tree.best_estimator_)
#Identify the best score during fitting with cross-validation

#Predict the response for test dataset
y_pred = grid_tree.best_estimator_.predict(X_test)

# Model F1, how often is the classifier correct?
print("F1:",metrics.f1_score(y_test, y_pred))<jupyter_output>F1: 0.7875000000000001
{'fraction_non_alphanumeric': 0.08008776741634668, 'fraction_numerical': 0.027427317608337904, 'mean_word_length': 4.877498388136686, 'pattern_counts': {'":': 0, '<': 38, '<?xml version=': 0, '>': 38, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '280945', 'license': 'MIT', 'n_tokens_mistral': 3220, 'n_tokens_neox': 3003, 'n_words': 879, 'path': 'Phase_3/Ensemble_Methods_Applied.ipynb', 'repo_name': 'ismizu/ds-east-042621-lectures', 'size': '9115'}