Examples of naturally shifted datasets are CIFAR-10.1 [1]}, CIFAR-10.2 [2]} and ImageNet-v2 [3]}. These datasets are collected following the identical process to that of the original reference dataset. Such datasets have been utilised to measure the lack of generalisation and robustness of many classification models [4]}, [5]}. This is the first attempt at evaluating calibration-refinement under natural data-shift to the best of our knowledge. Assessment of calibration under synthetic shift has been reported by [5]}. However, we believe natural data-shift is a scenario which a deployed DNN is more likely to face and hence requires equal if not more attention. By evaluating calibration-refinement trade-off we will be able to highlight the severity and extent of the problem induced by many calibration approaches.
