StaDRe and StaDRo: Reliability and Robustness Estimation of ML-based Forecasting using Statistical Distance Measures
Reliability estimation of Machine Learning (ML) models is becoming a crucial
subject. This is particularly the case when such \mbox{models} are deployed in
safety-critical applications, as the decisions based on model predictions can
result in hazardous situations. In this regard, recent research has proposed
methods to achieve safe, \mbox{dependable}, and reliable ML systems. One such
method consists of detecting and analyzing distributional shift, and then
measuring how such systems respond to these shifts. This was proposed in
earlier work in SafeML. This work focuses on the use of SafeML for time series
data, and on reliability and robustness estimation of ML-forecasting methods
using statistical distance measures. To this end, distance measures based on
the Empirical Cumulative Distribution Function (ECDF) proposed in SafeML are
explored to measure Statistical-Distance Dissimilarity (SDD) across time
series. We then propose SDD-based Reliability Estimate (StaDRe) and SDD-based
Robustness (StaDRo) measures. With the help of a clustering technique, the
similarity between the statistical properties of data seen during training and
the forecasts is identified. The proposed method is capable of providing a link
between dataset SDD and Key Performance Indicators (KPIs) of the ML models.