<jupyter_start><jupyter_text>Understanding Sampling VarianceEach sample of data we observe is assumed to be a realisation from some underlying density $p(x)$, with some (possibly deterministic) distribution of labels $p(y|x)$. Since we only observe an empirical sample, we are unable to calculate the optimal $\hat{p}(y|x)$ for the problem, and instead calculate an estimate, *that depends on the particular sample of data we observe*: $\hat{p}_{est}(y|x)$. Suppose we observe multiple datasets $\mathcal{D}_1$ and $\mathcal{D}_2$. We will obtain two estimates, $\hat{p}_{\mathcal{D}_1}(y|x)$, $\hat{p}_{\mathcal{D}_2}(y|x)$, where in general $\hat{p}_{\mathcal{D}_1}(y|x) \ne\hat{p}_{\mathcal{D}_2}(y|x)$. We say the difference in observed data between $\mathcal{D}_1$ and $\mathcal{D}_2$ is due to **sampling variance**, and the difference in our classifier or regressors is due to the same.Suppose our $\hat{p}_{est}(y|x)$ are parameterised by some $\theta$. Every time we perform an experiment we expect to get a different $\theta$. The variance between these $\theta$s for *iid* data is due to sampling variance, and we may characterise a given algorithm for estimating $\hat{p}_{est}(y|x)$ by the **variance** it exhibits between training sets, and the **bias** it demonstrates over all datasets.Below is a visualisation of this process. Suppose we observe a set of data for a simple 1D regression problem. The least squares solution in the case where the true $p(y|x)$ is Gaussian is well characterised and understood. In the below we look at an example of how the regression parameters vary across different datasets. In the final cells we plot a distribution over the slopes our regression model has learned. This is an example of an algorithm with low bias and low variance. Note that for a more interesting dataset, the algorithm would usually be characterised as *high* bias, low variance.<jupyter_code>import numpy as np
import matplotlib.pyplot as plt
w_true = [2,0.7]
def generate_sample(n, sigma):
    x = np.random.randn(n)
    y = x*w_true[1] + w_true[0]
    y = y + sigma*np.random.randn(n)
    return x, y

def estimate_w(x, y):
    x = np.stack((np.ones_like(x), x), 1)
    q, r = np.linalg.qr(x)
    return np.linalg.solve(r, q.T.dot(y))

save_est = []
%matplotlib notebook
def abline(slope, intercept):
    """Plot a line from slope and intercept - thanks to https://stackoverflow.com/a/43811762"""
    axes = plt.gca()
    x_vals = np.array(axes.get_xlim())
    y_vals = intercept + slope * x_vals
    plt.plot(x_vals, y_vals, '--', color='red')

x, y = generate_sample(500, 0.5)
plt.scatter(x, y, alpha=0.2)
w = estimate_w(x,y)
abline(*w[::-1])
plt.title('Estimate of slope: ' + str(np.round(w[1],4)))
save_est.append(w)<jupyter_output><empty_output><jupyter_text>Now let's take a look at all the estimates we have obtained so far(Make sure you have run the above cell at least 5-10 times.)<jupyter_code>save_est
import seaborn as sns
%matplotlib notebook
sns.distplot(np.array(save_est)[:,1])
import seaborn as sns
%matplotlib notebook
sns.kdeplot(np.array(save_est))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11911623439000961, 'fraction_numerical': 0.011847582452769772, 'mean_word_length': 4.87218045112782, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '302786', 'license': 'MIT', 'n_tokens_mistral': 1013, 'n_tokens_neox': 973, 'n_words': 386, 'path': 'sample_variance.ipynb', 'repo_name': 'prateekpg2455/teachingnotes', 'size': '3123'}