Deep neural networks have for some years been the state of the art for a number of predictive tasks, such as image classification [1]}, [2]}, [3]}, language modeling [4]}, [5]} and reinforcement learning [6]}, [7]}, [8]}. This has led to their widespread adoption in many areas of machine learning. Such models are however notorious for their black box nature: due to the large numbers of parameters and complex neural architectures, their predictions become very difficult or even impossible to understand. Interpretability of predictive models is a very useful property for many different reasons: it allows us to extract understandable knowledge from large datasets, potentially leading to new knowledge about the data itself, debug models when they fail, and explain predictions to end users to build trust in the system [9]}. In some cases, the ability to explain predictions is crucial for model deployment.
