<jupyter_start><jupyter_text>*Credits: this notebook origin (shared under MIT license) belongs to [ML course at ICL](https://github.com/yandexdataschool/MLatImperial2020) held by Yandex School of Data Analysis. Special thanks to the course team for making it available online.*  day07: Cross-validation riddle Here's a small example of cross-validation done wrongly. Can you spot the problem?<jupyter_code># Some imports...
import numpy as np
import matplotlib.pyplot as plt

from sklearn.svm import LinearSVC
from sklearn.model_selection import KFold, cross_val_score
from sklearn.metrics import accuracy_score<jupyter_output><empty_output><jupyter_text>**Plan:**- Let's create a binary classification dataset where targets are completely independent from the features  - *(i.e. no model could ever predict them well)*- We'll do some simple feature selection- And cross-validate a model on this data**Q:** what accuracy do we expect (classes are even)? We'll start from writing a class to select the best features:<jupyter_code>class FeatureSelector:
    def __init__(self, num_features):
        self.n = num_features # number of best features to select

    def fit(self, X, y):
        # Select features that describe the targets best, i.e. have
        # highest correlation with them:
        covariance = ((X - X.mean(axis=0)) * (y[:,np.newaxis] - y.mean())).mean(axis=0)
        self.best_feature_ids = np.argsort(np.abs(covariance))[-self.n:]

    def transform(self, X):
        return X[:,self.best_feature_ids]

    def fit_transform(self, X, y):
        self.fit(X, y)
        return self.transform(X)
num_features_total = 1000
num_features_best = 100

N = 100

# Dataset generation
X = np.random.normal(size=(N, num_features_total))
y = np.random.randint(2, size=N)

# Feature selection:
X_best = FeatureSelector(num_features_best).fit_transform(X, y)

# Simple classification model
model = LinearSVC()

# Estimatin accuracy using cross-validation:
cv_score = cross_val_score(model, X_best, y, scoring='accuracy', cv=10, n_jobs=-1).mean()
print(f"CV score is {cv_score}")<jupyter_output><empty_output><jupyter_text>What's going on?! Why accuracy is so high?Maybe it just happened by chance? Let's repeat this experiment many times and histogram the results:<jupyter_code>num_features_total = 1000
num_features_best = 100

N = 100
def experiment():
    # Dataset generation
    X = np.random.normal(size=(N, num_features_total))
    y = np.random.randint(2, size=N)

    # Feature selection:
    X_best = FeatureSelector(num_features_best).fit_transform(X, y)

    # Simple classification model
    model = LinearSVC()

    # Estimatin accuracy using cross-validation:
    return cross_val_score(model, X_best, y, scoring='accuracy', cv=10, n_jobs=-1).mean()

results = [experiment() for _ in range(100)]
plt.hist(results, bins=10);<jupyter_output><empty_output><jupyter_text>Can you explain and fix this?<jupyter_code># It's dangerous to go alone. Take this!
from sklearn.pipeline import Pipeline
# YOUR BEAUTIFUL FIX HERE<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10241356816699282, 'fraction_numerical': 0.013372472276581865, 'mean_word_length': 4.830798479087452, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '179384', 'license': 'MIT', 'n_tokens_mistral': 998, 'n_tokens_neox': 943, 'n_words': 298, 'path': 'day07_cross_validation_bias_variance/day07_cross_validation_riddle.ipynb', 'repo_name': 'girafe-ai/intro_to_ml_harbour', 'size': '3066'}