As recommended in [1]}, we used 15 refits.
We additionally analyzed how the coverage and interval width changed by increasing refits from 2 to 30 and noticed that the coverage worsened with more refits, while the width of the confidence intervals decreased.
Increasing the number of refits comes with an inherent trade-off between interval width and coverage: The more refits are considered, the more accurate the learner-PFI and learner-PD become and also the more certain the variance estimates become, scaling with \(1/m\) .
But there is a limit to the information in the data, so that additional refits falsely reduce the variance estimate and the confidence intervals become too narrow.
To refit the model 10 - 20 times seemed to be an acceptable trade-off between coverage and interval width, see for example Figure REF .
Below \(\sim 10\)  refits, the confidence intervals were large, and also the mean PD/PFI estimates have a high variance.
Above \(\sim 20\)  refits, the widths did not decrease by much anymore.
The figures for the other scenarios can be found in Appendix .
<FIGURE>