<jupyter_start><jupyter_text>Improving Performance---<jupyter_code>from google.colab import drive
drive.mount('/content/drive')<jupyter_output>Mounted at /content/drive<jupyter_text>*   Bagging: Buildig Multiple models (typicaplly of same type) from different subsamples of the triaing dataset.*   Boosting: Building Multiple Models (typically of same type) each of which learn to fix the prediciton error of prior model in the sequence.*   Voting: Building multiple model (typically of different type) & simple statictics (like calculating the mean) are used to combine the prediction.  Bagging AlgorithmsBagging performs best with algorithms that have high variance. A popular example are decision trees, often constructed without pruning.Random Forests is an extension of bagged decision trees. Samples of the training dataset are taken with replacement, but the trees are constructed in a way that reduces the correlation between individual classifiers.Extra Trees are another modication of bagging where random trees are constructed from samples of the training dataset.<jupyter_code># Bagged Dicision Tree
from pandas import read_csv
from matplotlib import pyplot as plt
from sklearn.model_selection import KFold
from sklearn.model_selection import cross_val_score
from sklearn.ensemble import BaggingClassifier
from sklearn.tree import DecisionTreeClassifier

# Load Data
filename = "/content/drive/MyDrive/Colab Notebooks/ML Mastery python/Dataset/pima-indians-diabetes.csv"
names = ['preg', 'plas','pres','skin','test','mass','pedi','age','class']
dataframe = read_csv(filename, names=names)
array = dataframe.values
# separate array into input & output 
X = array[:,0:8]
Y = array[:,8]
seed = 7
kfold = KFold(n_splits=10)
cart = DecisionTreeClassifier()
trees = 100
model = BaggingClassifier(base_estimator = cart, n_estimators = trees, random_state=seed)
results = cross_val_score(model, X, Y, cv=kfold)
print(results.mean())
# Random Forest classification
from sklearn.ensemble import RandomForestClassifier
##############################################
seed = 7
kfold = KFold(n_splits=10)
trees = 100
model = RandomForestClassifier(n_estimators = trees, max_features = 3)
results = cross_val_score(model, X, Y, cv=kfold)
print(results.mean())
# Extra tree classification
from sklearn.ensemble import ExtraTreesClassifier
##############################################
seed = 7
kfold = KFold(n_splits=10)
trees = 100
model = ExtraTreesClassifier(n_estimators = trees, max_features = 3)
results = cross_val_score(model, X, Y, cv=kfold)
print(results.mean())<jupyter_output>0.7577238550922762<jupyter_text>Boosting AlgorithmsAdaBoost generally works by weighting instances in the dataset by how easy or dicult they are to classify, allowing the algorithm to pay less or more attention to them in the construction of subsequent models.Stochastic Gradient Boosting (also called Gradient Boosting Machines) are one of the most sophisticated ensemble techniques<jupyter_code># AdaBoost Classification
from pandas import read_csv
from sklearn.model_selection import KFold
from sklearn.model_selection import cross_val_score
from sklearn.ensemble import AdaBoostClassifier
filename = "/content/drive/MyDrive/Colab Notebooks/ML Mastery python/Dataset/pima-indians-diabetes.csv"
names = ['preg', 'plas', 'pres', 'skin', 'test', 'mass', 'pedi', 'age', 'class']
dataframe = read_csv(filename, names=names)
array = dataframe.values
X = array[:,0:8]
Y = array[:,8]
num_trees = 30
seed=7
kfold = KFold(n_splits=10)
model = AdaBoostClassifier(n_estimators=num_trees, random_state=seed)
results = cross_val_score(model, X, Y, cv=kfold)
print(results.mean())
# Stochastic Gradient Boosting Classification
from sklearn.ensemble import GradientBoostingClassifier
#######################################
seed = 7
num_trees = 100
kfold = KFold(n_splits=10)
model = GradientBoostingClassifier(n_estimators=num_trees, random_state=seed)
results = cross_val_score(model, X, Y, cv=kfold)
print(results.mean())<jupyter_output>0.7681989063568012<jupyter_text>Voting EnsembleVoting is one way of combining the predictions from multiple ML algorithms. It works by creating two or more standalone models from your training dataset.A Voting Classifier can then be used to wrap your models and average the predictions of sub-models when asked to make predictions for new data. The predictions of the sub-models can be weighted, but specifying the weights for classifiers manually is difficult. More advanced methods can learn how to best weight the predictions from sub-models, but this is called stacking (stacked aggregation)<jupyter_code># Voting Ensemble for Classification
from pandas import read_csv
from sklearn.model_selection import KFold
from sklearn.model_selection import cross_val_score
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.svm import SVC
from sklearn.ensemble import VotingClassifier
filename = "/content/drive/MyDrive/Colab Notebooks/ML Mastery python/Dataset/pima-indians-diabetes.csv"
names = ['preg', 'plas', 'pres', 'skin', 'test', 'mass', 'pedi', 'age', 'class']
dataframe = read_csv(filename, names=names)
array = dataframe.values
X = array[:,0:8]
Y = array[:,8]
kfold = KFold(n_splits=10)
# create the sub models
estimators = []
model1 = LogisticRegression(max_iter=10000)
estimators.append(('logistic', model1))
model2 = DecisionTreeClassifier()
estimators.append(('cart', model2))
model3 = SVC(max_iter=10000)
estimators.append(('svm', model3))
# create the ensemble model
ensemble = VotingClassifier(estimators)
results = cross_val_score(ensemble, X, Y, cv=kfold)
print(results.mean())<jupyter_output>0.7643028024606973
{'fraction_non_alphanumeric': 0.10575236759031918, 'fraction_numerical': 0.019116099614170465, 'mean_word_length': 6.8879668049792535, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '544724', 'license': 'MIT', 'n_tokens_mistral': 1792, 'n_tokens_neox': 1679, 'n_words': 559, 'path': 'Lessons/13_Ensemble_Methods.ipynb', 'repo_name': 'Bluelord/ML_Mastery_Python', 'size': '5702'}