Retrieval-based systems are of interest because they admit a natural evaluation metric, namely the recall and precision measures. First introduced for evaluating user simulations by Schatzmann et al. (2005) , such a framework has gained recent prominence for the evaluation of end-to-end dialogue systems<cite> (Lowe et al., 2015a</cite>; Kadlec et al., 2015; Dodge et al., 2015) .