Predictive Multiplicity in Probabilistic Classification
For a prediction task, there may exist multiple models that perform almost
equally well. This multiplicity complicates how we typically develop and deploy
machine learning models. We study how multiplicity affects predictions -- i.e.,
predictive multiplicity -- in probabilistic classification. We introduce new
measures for this setting and present optimization-based methods to compute
these measures for convex empirical risk minimization problems like logistic
regression. We apply our methodology to gain insight into why predictive
multiplicity arises. We study the incidence and prevalence of predictive
multiplicity in real-world risk assessment tasks. Our results emphasize the
need to report multiplicity more widely.