Looking for Out-of-Distribution Environments in Critical Care: A case study with the eICU Database
Generalizing to new populations and domains in machine learning is still an
open problem which has seen increased interest recently. In particular,
clinical models show a significant performance drop when tested in settings not
seen during training, e.g., new hospitals or population demographics. Recently
proposed models for domain generalisation promise to alleviate this problem by
learning invariant characteristics across environments, however, there is still
scepticism about whether they improve over traditional training. In this work,
we take a principled approach to identifying Out of Distribution (OoD)
environments, motivated by the problem of cross-hospital generalization in
critical care. We propose model-based and heuristic approaches to identify OoD
environments and systematically compare models with different levels of
held-out information. In particular, based on the assumption that models with
access to OoD data should outperform other models, we train models across a
range of experimental setups that include leave-one-hospital-out training and
cross-sectional feature splits. We find that access to OoD data does not
translate to increased performance, pointing to inherent limitations in
defining potential OoD environments in the eICU Database potentially due to
data harmonisation and sampling. Echoing similar results with other popular
clinical benchmarks in the literature, new approaches are required to evaluate
robust models in critical care.