Metzger et al. [1]} propose to use estimates of the reliability of predictions (calculated as per the method in [2]}), in addition to a prediction score and other features, as input to an online reinforcement learning (RL) method. The RL method learns a policy that optimizes the percentage of cases that finish with a desirable outcome.
However, the reliability estimates used by Metzger et al. do not come with confidence guarantees. Also, their approach learns a black-box policy since the reinforcement learner uses neural networks. Thus, the policy rules cannot be directly explained to a business user. Also, Metzger et al. use online RL, which means that the RL agent learns from trial-and-error in an actual operating environment until it converges into an optimal policy. In this paper, we seek to discover policies offline (i.e., based on past data).
