Explaining Neural Networks without Access to Training Data
We consider generating explanations for neural networks in cases where the
network's training data is not accessible, for instance due to privacy or
safety issues. Recently, $\mathcal{I}$-Nets have been proposed as a sample-free
approach to post-hoc, global model interpretability that does not require
access to training data. They formulate interpretation as a machine learning
task that maps network representations (parameters) to a representation of an
interpretable function. In this paper, we extend the $\mathcal{I}$-Net
framework to the cases of standard and soft decision trees as surrogate models.
We propose a suitable decision tree representation and design of the
corresponding $\mathcal{I}$-Net output layers. Furthermore, we make
$\mathcal{I}$-Nets applicable to real-world tasks by considering more realistic
distributions when generating the $\mathcal{I}$-Net's training data. We
empirically evaluate our approach against traditional global, post-hoc
interpretability approaches and show that it achieves superior results when the
training data is not accessible.