With the rise in employment of deep learning methods in safety-critical scenarios, interpretability is more essential than ever before. Although many different directions regarding interpretability have been explored for visual modalities, time-series data has been neglected with only a handful of methods tested due to their poor intelligibility. We approach the problem of interpretability in a novel way by proposing TSInsight where we attach an auto-encoder with a sparsity-inducing norm on its output to the classifier and fine-tune it based on the gradients from the classifier and a reconstruction penalty. The auto-encoder learns to preserve features that are important for the prediction by the classifier and suppresses the ones that are irrelevant i.e. serves as a feature attribution method to boost interpretability. In other words, we ask the network to only reconstruct parts which are useful for the classifier i.e. are correlated or causal for the prediction. In contrast to most other attribution frameworks, TSInsight is capable of generating both instance-based and model-based explanations. We evaluated TSInsight along with other commonly used attribution methods on a range of different time-series datasets to validate its efficacy. Furthermore, we analyzed the set of properties that TSInsight achieves out of the box including adversarial robustness and output space contraction. The obtained results advocate that TSInsight can be an effective tool for the interpretability of deep time-series models. Deep learning models have been at the forefront of technology in a range of different domains including image classification (Krizhevsky et al., 2012) , object detection (Girshick, 2015) , speech recognition (Dahl et al., 2010) , text recognition (Breuel, 2008) , image captioning and pose estimation (Cao et al., 2018) . These models are particularly effective in automatically discovering useful features. However, this automated feature extraction comes at the cost of lack of transparency of the system. Therefore, despite these advances, their employment in safety-critical domains like finance (Knight, 2017) , self-driving cars and medicine (Zintgraf et al., 2017) is limited due to the lack of interpretability of the decision made by the network. Numerous efforts have been made for the interpretation of these black-box models. These efforts can be mainly classified into two separate directions. The first set of strategies focuses on making the network itself interpretable by trading off some performance. These strategies include SelfExplainable Neural Network (SENN) (Alvarez-Melis & Jaakkola, 2018) and Bayesian non-parametric regression models (Guo et al., 2018) . The second set of strategies focuses on explaining a pretrained model i.e. they try to infer the reason for a particular prediction. These attribution techniques include saliency map (Yosinski et al., 2015) and layer-wise relevance propagation (Bach et al., 2015) . However, all of these methods have been particularly developed and tested for visual modalities which are directly intelligible for humans. Transferring methodologies developed for visual modalities to time-series data is difficult due to the non-intuitive nature of time-series. Therefore, only a handful of methods have been focused on explaining time-series models in the past (Kumar et al., 2017; Siddiqui et al., 2019) . We approach the attribution problem in a novel way by attaching an auto-encoder on top of the classifier. The auto-encoder is fine-tuned based on the gradients from the classifier. Rather than asking the auto-encoder to reconstruct the whole input, we ask the network to only reconstruct parts which are useful for the classifier i.e. are correlated or causal for the prediction. In order to achieve this, we introduce a sparsity inducing norm onto the output of the auto-encoder. In particular, the contributions of this paper are twofold: • A novel attribution method for time-series data which makes it much easier to interpret the decision of any deep learning model. The method also leverages dataset-level insights when explaining individual decisions in contrast to other attribution methods. • Detailed analysis of the information captured by different attribution techniques using a simple suppression test on a range of different time-series datasets. This also includes analysis of the different out of the box properties achieved by TSInsight including generic applicability, contraction in the output space and resistance against trivial adversarial noise.
We present an attribution technique leveraging sparsity inducing norms to achieve interpretability.
{"src": "scitldr-AIC_train", "paper_id": "B1gzLaNYvr"}