With our simulation results we could show that confidence intervals using the naive variance estimation (without correction) results in way too narrow intervals.
While the simple correction term by [1]} does not always provide the desired coverage probability, it is a vast improvement over the naive approach.
We therefore recommend using the correction when computing confidence intervals for learner-PD and learner-PFI – it is currently the best approach available.
We also recommend refitting the model around 15 times.
For more cautious confidence intervals we recommend using confidence intervals based on resampling with replacement (bootstrap) over sampling wihtout replacement (subsampling).
However, beside wider confidence intervals, the bootstrap requires additional attention when model tuning with internal resampling is used, as data points may otherwise end up in both training and validation data.
