<jupyter_start><jupyter_text>📃 Solution for Exercise M7.01This notebook aims at building baseline classifiers, which we'll use tocompare our predictive model. Besides, we will check the differences withthe baselines that we saw in regression.We will use the adult census dataset, using only the numerical features.<jupyter_code>import pandas as pd

adult_census = pd.read_csv("../datasets/adult-census-numeric-all.csv")
data, target = adult_census.drop(columns="class"), adult_census["class"]<jupyter_output><empty_output><jupyter_text>First, define a `ShuffleSplit` cross-validation strategy taking half of thesample as a testing at each round.<jupyter_code># solution
from sklearn.model_selection import ShuffleSplit

cv = ShuffleSplit(n_splits=10, test_size=0.5, random_state=0)<jupyter_output><empty_output><jupyter_text>Next, create a machine learning pipeline composed of a transformer tostandardize the data followed by a logistic regression.<jupyter_code># solution
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

classifier = make_pipeline(StandardScaler(), LogisticRegression())<jupyter_output><empty_output><jupyter_text>Get the test score by using the model, the data, and the cross-validationstrategy that you defined above.<jupyter_code># solution
from sklearn.model_selection import cross_validate

result_classifier = cross_validate(classifier, data, target, cv=cv, n_jobs=2)

test_score_classifier = pd.Series(
    result_classifier["test_score"], name="Classifier score")<jupyter_output><empty_output><jupyter_text>Using the `sklearn.model_selection.permutation_test_score` function,check the chance level of the previous model.<jupyter_code># solution
from sklearn.model_selection import permutation_test_score

score, permutation_score, pvalue = permutation_test_score(
    classifier, data, target, cv=cv, n_jobs=2, n_permutations=10)
test_score_permutation = pd.Series(permutation_score, name="Permuted score")<jupyter_output><empty_output><jupyter_text>Finally, compute the test score of a dummy classifier which would predictthe most frequent class from the training set. You can look at the`sklearn.dummy.DummyClassifier` class.<jupyter_code># solution
from sklearn.dummy import DummyClassifier

dummy = DummyClassifier(strategy="most_frequent")
result_dummy = cross_validate(dummy, data, target, cv=cv, n_jobs=2)
test_score_dummy = pd.Series(result_dummy["test_score"], name="Dummy score")<jupyter_output><empty_output><jupyter_text>Now that we collected the results from the baselines and the model, plotthe distributions of the different test scores. We concatenate the different test score in the same pandas dataframe.<jupyter_code># solution
final_test_scores = pd.concat(
    [test_score_classifier, test_score_permutation, test_score_dummy],
    axis=1,
)<jupyter_output><empty_output><jupyter_text>Next, plot the distributions of the test scores.<jupyter_code># solution
import matplotlib.pyplot as plt

final_test_scores.plot.hist(bins=50, density=True, edgecolor="black")
plt.legend(bbox_to_anchor=(1.05, 0.8), loc="upper left")
plt.xlabel("Accuracy (%)")
_ = plt.title("Distribution of the test scores")<jupyter_output><empty_output><jupyter_text>We observe that the dummy classifier with the strategy `most_frequent` isequivalent to the permutation score. We can also conclude that our modelis better than the other baseline. Change the strategy of the dummy classifier to `stratified`, compute theresults and plot the distribution together with the other results. Explainwhy the results get worse.<jupyter_code># solution
dummy = DummyClassifier(strategy="stratified")
result_dummy_stratify = cross_validate(dummy, data, target, cv=cv, n_jobs=2)
test_score_dummy_stratify = pd.Series(
    result_dummy_stratify["test_score"], name="Dummy 'stratify' score")
final_test_scores = pd.concat(
    [
        test_score_classifier, test_score_permutation,
        test_score_dummy, test_score_dummy_stratify,
    ],
    axis=1,
)
final_test_scores.plot.hist(bins=50, density=True, edgecolor="black")
plt.legend(bbox_to_anchor=(1.05, 0.8), loc="upper left")
plt.xlabel("Accuracy (%)")
_ = plt.title("Distribution of the test scores")<jupyter_output><empty_output><jupyter_text>We see that using `strategy="stratified"`, the results are much worse thanwith the `most_frequent` strategy. Since the classes are imbalanced,predicting the most frequent involves that we will be right for theproportion of this class (~75% of the samples). However, the `"stratified"`strategy will randomly generate predictions by respecting the trainingset's class distribution, resulting in some wrong predictions even forthe most frequent class, hence we obtain a lower accuracy.Please refer to the scikit-learn documentation on [DummyClassifier](https://scikit-learn.org/stable/modules/generated/sklearn.dummy.DummyClassifier.html)for more details on the meaning of the `strategy="stratified"` parameter.<jupyter_code>from sklearn.model_selection import cross_val_score

dummy_models = {
    "Dummy 'most_frequent'": DummyClassifier(strategy="most_frequent"),
    "Dummy 'stratified'": DummyClassifier(strategy="stratified"),
}
n_runs = 3

for run_idx in range(n_runs):
    final_scores = pd.DataFrame({
        f"{name} score": cross_val_score(model, data, target, cv=cv, n_jobs=2)
        for name, model in dummy_models.items()
    })

    final_scores.plot.hist(bins=50, density=True, edgecolor="black")
    plt.legend(bbox_to_anchor=(1.05, 0.8))
    plt.xlabel("Accuracy (%)")
    _ = plt.title(f"Distribution of scores in run #{run_idx}")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09220482139714939, 'fraction_numerical': 0.007214499384128101, 'mean_word_length': 7.005633802816901, 'pattern_counts': {'":': 3, '<': 41, '<?xml version=': 0, '>': 41, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '553576', 'license': 'CC-BY-4.0', 'n_tokens_mistral': 1800, 'n_tokens_neox': 1719, 'n_words': 481, 'path': 'notebooks/cross_validation_sol_02.ipynb', 'repo_name': 'Sjonnie404/scikit-learn-mooc', 'size': '5683'}