<jupyter_start><jupyter_text>Gradient boost guided exampleNow that you've walked through gradient boost by hand, try it with scikit-learn. You'll still use the *European Social Survey* data, but now with a categorical outcome: whether or not someone lives with a partner.<jupyter_code>df = pd.read_csv((
    "https://raw.githubusercontent.com/Thinkful-Ed/data-201-resources/"
    "master/ESS_practice_data/ESSdata_Thinkful.csv")).dropna()

# Define outcome and predictors.
# Set the outcomes to `0` and `1`.
y = df['partner'] - 1
X = df.loc[:, ~df.columns.isin(['partner', 'cntry', 'idno'])]

# Make the categorical variable `country` into dummies.
X = pd.concat([X, pd.get_dummies(df['cntry'])], axis=1)

# Create training and test sets.
offset = int(X.shape[0] * 0.8)

# Put 80% of the data in the training set.
X_train, y_train = X[:offset], y[:offset]

# And put 20% in the test set.
X_test, y_test = X[offset:], y[offset:]<jupyter_output><empty_output><jupyter_text>Because you're now working with a binary outcome, you've switched to a classifier. Now your loss function can't be the residuals. Your options are `deviance` and `exponential`. The `deviance` loss function is used for logistic regression, and you'll try that here.<jupyter_code># You'll make 500 iterations, use 2-deep trees, and set your loss function.

# It's best to update this table for the drill
# at the end of the checkpoint.

# params = {"n_estimators": 500, "max_depth": 2, "loss": "deviance"}
# orginal params

# params = {"n_estimators": 750, "max_depth": 3, "loss": "exponential",'subsample':0.5}
# accuracy: down, precision: down, recall: up

# params = {"n_estimators": 750, "max_depth": 3, "loss": "exponential", "subsample": 0.75}
# accuracy: up, precision: up, recall: up

# params = {"n_estimators": 750, "max_depth": 3, "loss": "deviance","subsample": 0.25, "learning_rate": 0.8}
# accuracy: DOWN, precision: DOWN, recall: UP
# lower subsample and higher learning_rate results in HIGH recall
# and BIG drop in accurracy and precision


# params = {"n_estimators": 750, "max_depth": 3, "loss": "deviance", "subsample": 0.75, "learning_rate": 0.8,}
# accuracy: down, precision: down, recall: up

# params = {"n_estimators": 1000, "max_depth": 2, "loss": "deviance"}
# accuracy: down, precision: up, recall: down

# params = {"n_estimators": 1000, "max_depth": 2, "loss": "exponential"}
# accuracy: down, precision: down, recall: up

# params = {"n_estimators": 1000, "max_depth": 4, "loss": "deviance", "subsample": 0.75}
# accuracy: down, precision: down, recall: up

params = {
    "n_estimators": 1500,
    "max_depth": 2,
    "loss": "deviance",
    "subsample": 0.9,
    "learning_rate": 0.225,
}


# Initialize and fit the model.
clf = ensemble.GradientBoostingClassifier(**params)
clf.fit(X_train, y_train)

predict_train = clf.predict(X_train)
predict_test = clf.predict(X_test)<jupyter_output><empty_output><jupyter_text>Evaluation of model: Confusion matrix plotThis plot is based on an example from [scikit-learn's documentation](https://scikit-learn.org/0.18/auto_examples/model_selection/plot_confusion_matrix.html).<jupyter_code>def plot_confusion_matrix(
    cm, classes, normalize, title="Confusion matrix", cmap=plt.cm.Blues
):
    """
    This function prints and plots the confusion matrix.
    Normalization can be applied by setting `normalize=True`.
    """
    if normalize:
        cm = cm.astype("float") / cm.sum(axis=1)[:, np.newaxis]
        print("Normalized confusion matrix")
    else:
        print("Confusion matrix, without normalization")

    plt.imshow(cm, interpolation="nearest", cmap=cmap)
    plt.title(title)
    plt.colorbar()
    tick_marks = np.arange(len(classes))
    plt.xticks(tick_marks, classes)
    plt.yticks(tick_marks, classes)
    fmt = ".2f" if normalize else "d"
    thresh = cm.max() / 2.0
    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
        plt.text(
            j,
            i,
            format(cm[i, j], fmt),
            horizontalalignment="center",
            verticalalignment="center",
            color="white" if cm[i, j] > thresh else "black",
        )
    plt.ylabel("True label")
    plt.xlabel("Predicted label")
cm = confusion_matrix(y_test, predict_test)
plot_confusion_matrix(cm, [0, 1], False)
cm1 = confusion_matrix(y_test, predict_test)
plot_confusion_matrix(cm1, [0, 1], False)
cm2 = confusion_matrix(y_test, predict_test)
plot_confusion_matrix(cm2, [0, 1], False)
cm3 = confusion_matrix(y_test, predict_test)
plot_confusion_matrix(cm3, [0, 1], False)
cm = confusion_matrix(y_test, predict_test)
plot_confusion_matrix(cm, [0, 1], True)
cm2 = confusion_matrix(y_test, predict_test)
plot_confusion_matrix(cm2, [0, 1], True)
cm3 = confusion_matrix(y_test, predict_test)
plot_confusion_matrix(cm3, [0, 1], True)
cm4 = confusion_matrix(y_test, predict_test)
plot_confusion_matrix(cm4, [0, 1], True)<jupyter_output>Normalized confusion matrix<jupyter_text>Accuracy<jupyter_code># 0.747239263803681
# 0.7239263803680982
# 0.7337423312883435
# 0.38282208588957056
# 0.6680981595092025
# 0.7423312883435583
# 0.7404907975460123
# 0.7141104294478527
# 0.7411042944785277

clf.score(X_test, y_test)<jupyter_output><empty_output><jupyter_text>Precision<jupyter_code># 0.7035647279549718
# 0.6524701873935264
# 0.6695652173913044
# 0.38137193203272496
# 0.553921568627451
# 0.7181628392484343
# 0.6845878136200717
# 0.6347107438016529
# 0.6885245901639344

y_pred = clf.predict(X_test)
precision_score(y_test, y_pred)<jupyter_output><empty_output><jupyter_text>Recall<jupyter_code># 0.5961844197138315
# 0.6089030206677265
# 0.6120826709062003
# 0.9634340222575517
# 0.7186009538950715
# 0.5468998410174881
# 0.6073131955484896
# 0.6104928457869634
# 0.6009538950715422

recall_score(y_test, y_pred)<jupyter_output><empty_output><jupyter_text>Unlike decision trees, gradient boost solutions aren't terribly easy to interpret on the surface. But they aren't quite black boxes either. You can get a measure of how important various features are by counting how many times a feature is used over the course of many decision trees.<jupyter_code>feature_importance = clf.feature_importances_

# Make importances relative to max importance.
feature_importance = 100.0 * (feature_importance / feature_importance.max())
sorted_idx = np.argsort(feature_importance)
pos = np.arange(sorted_idx.shape[0]) + 0.5
plt.subplot(1, 2, 2)
plt.barh(pos, feature_importance[sorted_idx], align="center")
plt.yticks(pos, X.columns[sorted_idx])
plt.xlabel("Relative Importance")
plt.title("Variable Importance")
plt.show()<jupyter_output><empty_output><jupyter_text>It appears that age and happiness are the most important features in predicting whether or not someone lives with a partner.  Drill: Improve this gradient boost modelThis model is already doing all right, but there is definitely room for improvement. Your task is to see how low you can get the error rates to go in the test set, based on your model in the training set. Here are some strategies that you might use:* Creating new features* Applying more overfitting-prevention strategies like subsampling* Making more iterations* Trying a different loss function* Changing the structure of the weak learner by allowing more leaves in the tree or using other modificationsYou can learn more about the parameters for the gradient boost model [in scikit-learn's documentation](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingClassifier.html). You can update the *params* table featured earlier, before you initialized and fit the model.Here are a few of the parameters that you could tune: - `learning_rate` - `max_depth` - `n_estimators` - `loss` - `subsample` - `min_samples_split` You can then use the same metrics to evaluate your results.          Have fun! Increased iterations can improve accuracy and precision a little. Lower supsampling with higher learning_rates results in a drastic change from lower recall/higher accuracy and precision to higher recall and lower accuracy and precision.<jupyter_code># fmt: off
params = {
    "model__subsample": [0.1, 0.25, 1.0],
    "model__min_samples_split": [2, 3, 4],
    "model__max_depth": [2,4,6],
}
n_trees = 500
learning_rate = 2 / n_trees
from sklearn.pipeline import Pipeline
from sklearn.model_selection import GridSearchCV
pipeline = Pipeline(
    [
        (
            "model",
            ensemble.GradientBoostingClassifier(
                n_estimators=n_trees, learning_rate=learning_rate
            ),
        ),
    ]
)
pipeline_cv = GridSearchCV(pipeline, params, verbose=1, n_jobs=-1, cv=2)
pipeline_cv.fit(X_train, y_train)

pipeline_cv.score(X_test, y_test)


y_pred = pipeline_cv.predict(X_test)
precision_score(y_test, y_pred)
recall_score(y_test, y_pred)
pipeline_cv.best_params_<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11462450592885376, 'fraction_numerical': 0.0686617730095991, 'mean_word_length': 5.020394289598912, 'pattern_counts': {'":': 38, '<': 32, '<?xml version=': 0, '>': 33, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '94944', 'license': 'MIT', 'n_tokens_mistral': 3400, 'n_tokens_neox': 2953, 'n_words': 834, 'path': 'gradient_boosting_example_and_challenge.ipynb', 'repo_name': 'Ninjaneer1/theworks', 'size': '8855'}