<jupyter_start><jupyter_text>EnsemblesBefore looking at Random Forest, we explore ensembles in general.Ensemble works best if the individual models are different (by algorithm or training set). RFRandom Forest is an Ensemble: same algorithm, random training sets. Voting algorithmsHard voting: choose class with most votes.Soft voting: choose class with highest confidence (probability) predition.<jupyter_code># First example demonstrates that three heads are better than one.
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC
from sklearn.ensemble import VotingClassifier
from sklearn.metrics import accuracy_score

def compare_accuracy(classifiers):
    for classifier in classifiers:
        classifier.fit(X_train,y_train)
        y_pred = classifier.predict(X_test)
        print(classifier.__class__.__name__,accuracy_score(y_test,y_pred))

from sklearn.datasets import make_moons
X,y = make_moons(n_samples=200, noise=0.15)
# This is a wrapper for ShuffleSplit
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = \
    train_test_split(X, y, test_size=0.20, random_state=42)

# Hard voting.
lrc = LogisticRegression()  # inappropriate: linear model on nonlinear data
rfc = RandomForestClassifier()
svm = SVC()
vot = VotingClassifier(
    estimators=[('LRC',lrc),('RFC',rfc),('SVM',svm)],
    voting='hard')
vot.fit(X_train,y_train) # recursively trains the sub models
compare_accuracy([lrc,rfc,svm,vot])
# Book says ensemble always best. Not true for me.
# Soft voting.
lrc = LogisticRegression()
rfc = RandomForestClassifier()
svm = SVC(probability=True)
vot = VotingClassifier(
    estimators=[('LRC',lrc),('RFC',rfc),('SVM',svm)],
    voting='soft')
compare_accuracy([lrc,rfc,svm,vot])<jupyter_output>LogisticRegression 0.85
RandomForestClassifier 0.975
SVC 1.0
VotingClassifier 0.975
{'fraction_non_alphanumeric': 0.0930475692629378, 'fraction_numerical': 0.012545739675901725, 'mean_word_length': 6.168539325842697, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 4, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '202052', 'license': 'MIT', 'n_tokens_mistral': 637, 'n_tokens_neox': 616, 'n_words': 168, 'path': 'Geron/Geron_ch7_RandomForest/RF_01_EnsembleVoting.ipynb', 'repo_name': 'ShepherdCode/ShepherdML', 'size': '1913'}