<jupyter_start><jupyter_text>Ensemble Methods Agenda1. Introduction to Ensemble Methods2. RandomForest3. AdaBoost4. GradientBoostingTree5. VotingClassifier  1. Introduction to Ensemble Method* Objective of ensemble methods is to combine the predictions of serveral base estimators ( Linear Regression, Decisison Tree, etc. ) to create a combined effect or more genralized model.* Two types of Ensemble Method  - Averaging Method : Build several estimators independently & average their predictions. Examples are RandomForest etc.  - Boosting Method : Base estimators are built sequentially using weighted version of data .i.e fitting models with data that were mis-classified. Examples are AdaBoost    2. RandomForest* Recap - Limitations of decison tree is that it overfits & shows high variance.* RandomForest is an averaging ensemble method whose prediction is function of prediction of 'n' decision trees.  Algorithm* Data consist of R rows & M features.* Sample of training data is taken.* Random set of features are selected.* As many as configured number of trees are created using above two steps.* Final prediction in case of classification is majority prediction.* Final prediction in case of regression is mean/median of individual tree prediction  Comparing Decision Tree & Random Forest for MNIST data<jupyter_code>from sklearn.datasets import load_digits
import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split
digits = load_digits()
X = digits.data
y = digits.target
trainX, testX, trainY, testY = train_test_split(X,y)
dt = DecisionTreeClassifier()
dt.fit(trainX,trainY)
dt.score(testX,testY)
rf = RandomForestClassifier()
rf.fit(trainX,trainY)
rf.score(testX,testY)<jupyter_output><empty_output><jupyter_text>Important Hyper-parameters* n_estimators : number of trees to be configured, larger is better but compute cost.* max_features : maximum number of features to be considered for splitting the node. For classification this equals to sqrt(n_features). And, for regression max_features = n_features.* n_jobs : Configure as -1 so that we can make use of all cores.  Advantages* Minimal data cleaning or dealing with missing values required.* Works well with high dimensional datasets* Minimizes variance even for low variance models* RandomForest can tell importance of features. We can find important features & use them in model training<jupyter_code>rf.feature_importances_
rf.n_outputs_<jupyter_output><empty_output><jupyter_text>3. AdaBoost* Boosting in general is about building a model from the training data, then creating a second model that attempts to correct the errors from the first model. Models are added until the training set is predicted perfectly or a maximum number of models are added.* AdaBoost was first boosting algorithm.* AdaBoost can be used for both classification & regression Algorithm* Core concept of adaboost is to fit weak learners ( like decision tree ) sequantially on repeatedly modifying data.* Initially, each data is assigned equal weights.* A base estimator is fitted with this data.* Weights of misclassified data are increased & weights of correctly classified data is decreased. * Repeat the above two steps till all data are correctly classified or max number of iterations configured.* Making Prediction : The predictions from all of them are then combined through a weighted majority vote (or sum) to produce the final prediction.<jupyter_code>from sklearn.ensemble import AdaBoostClassifier
ab = AdaBoostClassifier(base_estimator=DecisionTreeClassifier(max_depth=8),n_estimators=600)
ab.fit(trainX,trainY)
ab.score(testX,testY)
ab = AdaBoostClassifier(base_estimator=RandomForestClassifier(n_estimators=20),n_estimators=600)
ab.fit(trainX,trainY)
ab.score(testX,testY)<jupyter_output><empty_output><jupyter_text>4. GradientBoostingTree* A machine learning technique for regression and classification problems, which produces a prediction model in the form of an ensemble of weak prediction models, typically decision trees.* One of the very basic assumption of linear regression is that it's sum of residuals is 0.* These residuals as mistakes committed by our predictor model. * Although, tree based models are not based on any of such assumptions, but if sum of residuals is not 0, then most probably there is some pattern in the residuals of our model which can be leveraged to make our model better. * So, the intuition behind gradient boosting algorithm is to leverage the pattern in residuals and strenghten a weak prediction model, until our residuals don't show any pattern.* Algorithmically, we are minimizing our loss function, such that test loss reach it’s minima.  Problem : House Price Prediction using GradientBoostingTree<jupyter_code>from sklearn.datasets import load_boston
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
house_data = load_boston()
X = house_data.data
y = house_data.target
from sklearn.ensemble import GradientBoostingRegressor
gbt = GradientBoostingRegressor()
gbt
from sklearn.model_selection import train_test_split
trainX, testX, trainY, testY = train_test_split(X,y)
gbt.fit(trainX,trainY)
test_score = np.zeros(100, dtype=np.float64)
for i, y_pred in enumerate(gbt.staged_predict(testX)):
    test_score[i] = gbt.loss_(testY, y_pred)
plt.plot(test_score)
plt.xlabel('Iterations')
plt.ylabel('Least squares Loss')<jupyter_output><empty_output><jupyter_text>5. VotingClassifier* Core concept of VotingClassifier is to combine conceptually different machine learning classifiers and use a majority vote or weighted vote to predict the class labels. * Voting classifier is quite effective with good estimators & handles individual's limitations, ensemble methods can also participate. * Types of Voting Classifier  - Soft Voting Classifier, different weights configured to different estimator  - Hard Voting Classifier, all estimators have equal weighage  Problem : DIGIT identification using VotingClassifier<jupyter_code>from sklearn.ensemble import VotingClassifier,RandomForestClassifier,AdaBoostClassifier
from sklearn.svm import SVC
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
estimators = [ 
    ('rf',RandomForestClassifier(n_estimators=20)),
    ('svc',SVC(kernel='rbf', probability=True)),
    ('knc',KNeighborsClassifier()),
    ('abc',AdaBoostClassifier(base_estimator=DecisionTreeClassifier() ,n_estimators=20)),
    ('lr',LogisticRegression()) 
]
vc = VotingClassifier(estimators=estimators, voting='hard')
digits = load_digits()
X,y = digits.data, digits.target
trainX, testX, trainY, testY = train_test_split(X,y)
vc.fit(trainX,trainY)
vc.score(testX,testY)
for est,name in zip(vc.estimators_,vc.estimators):
    print (name[0], est.score(testX,testY))
    
vc = VotingClassifier(estimators=estimators, voting='soft', weights=[2,.1,3,2,2])
vc.fit(trainX,trainY)
vc.score(testX,testY)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0632752613240418, 'fraction_numerical': 0.005156794425087108, 'mean_word_length': 6.2265861027190335, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '257930', 'license': 'Apache-2.0', 'n_tokens_mistral': 2036, 'n_tokens_neox': 1931, 'n_words': 786, 'path': 'Code/16. Ensemble Methods.ipynb', 'repo_name': 'Abhishekjha111/AI-codes', 'size': '7175'}