A second option lies in classical bootstrapping [1]}, [2]}: we create new datasets by sampling with replacement from the original dataset, and use these samples to estimate the bias and variance. Repeating this procedure by resampling from the bootstrap samples in turn estimates the quantity required to correct [2]} the bootstrapped estimate (alg:bootstrap,fig:bootstrap). This approach does not require reducing the size of the training set, but bootstrapped samples will contain duplicate points.
