<jupyter_start><jupyter_text>Using Cross-Validation to Solve Anscombe's Quartet[Anscombe's Quartet](https://en.wikipedia.org/wiki/Anscombe%27s_quartet) is a collection of four data sets with nearly exactly the same summary statistics. It was developed by statistician <NAME> in 1973 to illustrate the importance of plotting data and the impact of outliers on statistical analysis. We can "solve" Anscombe's Quartet with [cross-validation](https://en.wikipedia.org/wiki/Cross-validation_(statistics)) in the sense that we can use statistics to determine which of the four data sets are *actualy* well represented by a linear model. Before we dive in, let's take a look at the data sets. [Seaborn](https://seaborn.pydata.org/) provides an easy way to load Anscombe's Quartet, as shown below.<jupyter_code>import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split, KFold
df = sns.load_dataset("anscombe")
sns.lmplot(x="x", y="y", col="dataset", hue="dataset", data=df,
           col_wrap=2, ci=None, palette="muted", height=4,
           scatter_kws={"s": 50, "alpha": 1})
plt.gcf().suptitle("Anscombe's Quartet", x=0.5, y=1.02)
plt.tight_layout()
plt.show()<jupyter_output><empty_output><jupyter_text>Visually, the four data sets look quite different. Group `I` is the only group that actually seems to be observations of a linear relationship with random noise. However, they all appears to have the same linear regression line. Next, let's calculate all the summary statistics, to show that they are identical.<jupyter_code>grouped = df.groupby('dataset')
summary_results = pd.DataFrame(columns=['mean_x', 'mean_y', 'std_x', 'std_y', 'correlation', 'slope', 'offset', 'R2'])
for key in grouped.groups.keys():
    data = grouped.get_group(key)
    fit = LinearRegression().fit(data['x'].values.reshape(-1, 1), data['y'].values)
    slope = fit.coef_[0]
    offset = fit.intercept_
    r2 = fit.score(data['x'].values.reshape(-1, 1), data['y'].values)
    summary_results.loc[key] = (
        grouped.mean().loc[key]['x'],
        np.round(grouped.mean().loc[key]['y'], 2),
        np.round(grouped.std().loc[key]['x'], 5),
        np.round(grouped.std().loc[key]['y'], 2),
        np.round(grouped.corr().loc[(key, 'x')]['y'], 3),
        np.round(slope, 3),
        np.round(offset, 2),
        np.round(r2, 2)
    )
summary_results<jupyter_output><empty_output><jupyter_text>As expected, all summary statistics are (nearly) identical. But what if we wanted to actually figure out which data set is best described by the linear model? We can do that with cross-validation.The idea of cross-validation is simple, you randomly hold out some amount of your data, and fit the model with the reduced set. Then, you predict on the hold out set and look at the residuals. This process is repeated $k$ times ("$k$-fold cross-validation"), so that every piece of data is in the test set exactly once. Finally, you calculate the standard deviation ([RMSE](https://en.wikipedia.org/wiki/Root-mean-square_deviation)) and mean ([MBE](https://en.wikipedia.org/wiki/Bias_of_an_estimator)) of all the residuals. A "good" model will have low RMSE and nearly zero MBE.We can repeat this process for each of the four groups in Anscombe's Quartet. Typically, we use cross-validation to pick the best model for a data set. In this case, we are finding which data set best fits a simple linear regresssion model.<jupyter_code>hold_out_validation = pd.DataFrame(columns=['rmse', 'mbe'])
for key in grouped.groups.keys():
    data = grouped.get_group(key)
    X = data['x'].values.reshape(-1, 1)
    y = data['y'].values
    splits = KFold(n_splits=3, shuffle=True, random_state=42).split(X)
    residuals = []
    for train_ix, test_ix in splits:
        fit = LinearRegression().fit(X[train_ix], y[train_ix])
        y_pred = fit.predict(X[test_ix])
        residuals.append(y[test_ix] - y_pred)
    residuals = np.concatenate(residuals)
    hold_out_validation.loc[key] = (np.std(residuals), np.mean(residuals))
hold_out_validation
plt.figure(figsize=(8, 5))
plt.stem(hold_out_validation['rmse'], label='RMSE')
plt.stem(hold_out_validation['mbe'], markerfmt='C1o', label='MBE')
plt.xticks([0, 1, 2, 3], list(hold_out_validation.index))
plt.legend()
plt.title("Cross-Validation Error Metrics on Anscombe's Quartet");<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.127492717902756, 'fraction_numerical': 0.009634774815146763, 'mean_word_length': 4.782383419689119, 'pattern_counts': {'":': 2, '<': 14, '<?xml version=': 0, '>': 14, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '371809', 'license': 'MIT', 'n_tokens_mistral': 1490, 'n_tokens_neox': 1399, 'n_words': 450, 'path': 'notebooks/Solving_Anscombes_Quartet.ipynb', 'repo_name': 'bmeyers/bmeyers.github.io', 'size': '4473'}