Two important topics in deep learning both involve incorporating humans into the modeling process: Model priors transfer information from humans to a model by regularizing the model's parameters; Model attributions transfer information from a model to humans by explaining the model's behavior. Previous work has taken important steps to connect these topics through various forms of gradient regularization. We find, however, that existing methods that use attributions to align a model's behavior with human intuition are ineffective. We develop an efficient and theoretically grounded feature attribution method, expected gradients, and a novel framework, attribution priors, to enforce prior expectations about a model's behavior during training. We demonstrate that attribution priors are broadly applicable by instantiating them on three different types of data: image data, gene expression data, and health care data. Our experiments show that models trained with attribution priors are more intuitive and achieve better generalization performance than both equivalent baselines and existing methods to regularize model behavior. Recent work on interpreting machine learning models has focused on feature attribution methods. Given an input feature, a model, and a prediction on a particular sample, such methods assign a number to the input feature that represents how important the input feature was for making the prediction. Previous literature about such methods has focused on the axioms they should satisfy (Lundberg and Lee, 2017; Sundararajan et al., 2017; Štrumbelj and Kononenko, 2014; Datta et al., 2016) , and how attribution methods can give us insight into model behavior (Lundberg et al., 2018a; Sayres et al., 2019; Zech et al., 2018) . These methods can be an effective way of revealing problems in a model or a dataset. For example, a model may place too much importance on undesirable features, rely on many features when sparsity is desired, or be sensitive to high frequency noise. In such cases, we often have a prior belief about how a model should treat input features, but for neural networks it can be difficult to mathematically encode this prior in terms of the original model parameters. Ross et al. (2017b) introduce the idea of regularizing explanations to train models that better agree with domain knowledge. Given a binary variable indicating whether each feature should or should not be important for predicting on each sample in the dataset, their method penalizes the gradients of unimportant features. However, two drawbacks limit the method's applicability to real-world problems. First, gradients don't satisfy the theoretical guarantees that modern feature attribution methods do (Sundararajan et al., 2017) . Second, it is often difficult to specify which features should be important in a binary manner. More recent work has stressed that incorporating intuitive, human priors will be necessary for developing robust and interpretable models (Ilyas et al., 2019) . Still, it remains challenging to encode meaningful, human priors like "have smoother attribution maps" or "treat this group of features similarly" by penalizing the gradients or parameters of a model. In this work, we propose an expanded framework for encoding abstract priors, called attribution priors, in which we directly regularize differentiable functions of a model's axiomatic feature attributions during training. This framework, which can be seen as a generalization of gradient-based regularization (LeCun et al., 2010; Ross et al., 2017b; Yu et al., 2018; Jakubovitz and Giryes, 2018; Roth et al., 2018) , can be used to encode meaningful domain knowledge more effectively than existing methods. Furthermore, we introduce a novel feature attribution method -expected gradientswhich extends integrated gradients (Sundararajan et al., 2017) , is naturally suited to being regularized under an attribution prior, and avoids hyperparameter choices required by previous methods. Using attribution priors, we build improved deep models for three different prediction tasks. On images, we use our framework to train a deep model that is more interpretable and generalizes better to noisy data by encouraging the model to have piecewise smooth attribution maps over pixels. On gene expression data, we show how to both reduce prediction error and better capture biological signal by encouraging similarity among gene expression features using a graph prior. Finally, on a patient mortality prediction task, we develop a sparser model and improve performance when learning from limited training data by encouraging a skewed distribution of the feature attributions. The immense popularity of deep learning has driven its application in many domains with diverse, complicated prior knowledge. While it is in principle possible to hand-design network architectures to encode this knowledge, we propose a simpler approach. Using attribution priors, any knowledge that can be encoded as a differentiable function of feature attributions can be used to encourage a model to act in a particular way in a particular domain. We also introduce expected gradients, a feature attribution method that is theoretically justified and removes the choice of a single reference value that many existing feature attribution methods require. We further demonstrate that expected gradients naturally integrates with attribution priors via sampling during SGD. The combination allows us to improve model performance by encoding prior knowledge across several different domains. It leads to smoother and more interpretable image models, biological predictive models that incorporate graph-based prior knowledge, and sparser health care models that can perform better in data-scarce scenarios. Attribution priors provide a broadly applicable framework for encoding domain knowledge, and we believe they will be valuable across a wide array of domains in the future. Normally, training with a penalty on any function of the gradients would require solving a differential equation. To avoid this, we adopt a double back-propagation scheme in which the gradients are first calculated with respect to the training loss, and alternately calculated with the loss with respect to the attributions (Yu et al., 2018; Drucker and Le Cun, 1992) . Our attribution method, expected gradients, requires background reference samples to be drawn from the training data. More specifically, for each input in a batch of inputs, we need k additional inputs to calculate expected gradients for that input batch. As long as k is smaller than the batch size, we can avoid any additional data reading by re-using the same batch of input data as a reference batch, as in Zhang et al. (2017) . We accomplish this by shifting the batch of input k times, such that each input in the batch uses k other inputs from the batch as its reference values.
A method for encouraging axiomatic feature attributions of a deep model to match human intuition.
{"src": "scitldr-AIC_train", "paper_id": "rygPm64tDH"}