The second outstanding issue is a lack of understanding of the reliability and failure modes of these models. Deep learning models, in general, are known to produce over-confident predictions, even when the predictions are completely wrong [1]}. In other words, there is little correlation between the confidence of a deep learning model in its predictions and how accurate the predictions actually are. Deep learning models also produce confident predictions on out-of-distribution (OOD) data, i.e., when the test data is from an entirely different distribution than the training data [2]}, [3]}. Needless to say, there is no performance guarantee on OOD data. In fact, in general the model predictions on OOD data are not expected to be any more useful than random assignment.
