Don't Get Me Wrong: How to apply Deep Visual Interpretations to Time Series

The correct interpretation and understanding of deep learning models is essential in many applications. Explanatory visual interpretation approaches for image and natural language processing allow domain experts to validate and understand almost any deep learning model. However, they fall short when generalizing to arbitrary time series data that is less intuitive and more diverse. Whether a visualization explains the true reasoning or captures the real features is difficult to judge. Hence, instead of blind trust we need an objective evaluation to obtain reliable quality metrics. We propose a framework of six orthogonal metrics for gradient- or perturbation-based post-hoc visual interpretation methods designed for time series classification and segmentation tasks. An experimental study includes popular neural network architectures for time series and nine visual interpretation methods. We evaluate the visual interpretation methods with diverse datasets from the UCR repository and a complex real-world dataset, and study the influence of common regularization techniques during training. We show that none of the methods consistently outperforms any of the others on all metrics while some are ahead at times. Our insights and recommendations allow experts to make informed choices of suitable visualization techniques for the model and task at hand.


Introduction
Due to its high performance on complex multi-modal data, deep learning (DL) becomes increasingly popular in many real-world applications that process time series data (Fawaz et al., 2019b). While we fundamentally rely on the networks' correct operation in many applications that consider safety (Berkenkamp et al., 2017) they remain difficult to interpret. Typical applications are the monitoring of industrial processes (Löffler et al., 2021), the support of healthcare and sports (Dorschky et al., 2020), or safety in autonomous driving (Schmidt et al., 2021). The need for improved model understanding (Carvalho et al., 2019), along with regulatory guidelines (Goodman and Flaxman, 2017), led to a myriad of new approaches to the visual interpretation problem (Zhang and Zhu, 2018).
Post-hoc visual interpretation allows a domain expert to validate and understand how (almost) arbitrary deep learning models operate. Their central idea is highlighting features on the input that are "relevant" for the prediction of a learned model (Adebayo et al., 2018). Many of these techniques do not require a modification of the original model (Simonyan et al., 2014;Ribeiro et al., 2016) which makes them compatible with different architectures. Thus, they are useful as a general-purpose validation tool for neural networks across different tasks (Arrieta et al., 2020).
However, while visual interpretation yields intuitive and correct explanations for images (Samek et al., 2021), the application of these methods on time series data is still an unsolved problem (Rojat et al., 2021). Time series are inherently more diverse (Rojat et al., 2021) because they may originate from a variety of different sensors and processes, and often do not allow an obvious patch-or texture-based localization of critical features for human observers. This makes the application and the evaluation of visual interpretability methods difficult. A domain expert can not easily judge if explanations are correct in (i) delivering the reasoning of the decision process in the DL model and in (ii) capturing the real features in the dataset that lead to a correct classification.
Hence, it is important not to blindly apply different visualization methods. This requires quality metrics that evaluate visual interpretations and that enable an expert to select a suitable visualization for a given model and task at hand. However, both stateof-the-art visualization techniques and metrics that evaluate visual interpretations, e.g., Pixel Flipping (Samek et al., 2017), Sanity Check (Adebayo et al., 2018), and sensitivity checks (Rebuffi et al., 2020), so far have been examined on images (Rojat et al., 2021) or on NLP tasks (Arras et al., 2017), but only rarely on more diverse time series (Schlegel et al., 2019). This lack of objective and subjective evaluation seriously limits their application and utility for time series.
This paper provides a study of six orthogonal metrics that characterize the quality of visual interpretations for time series data. We investigate "sanity" (Adebayo et al., 2018), "faithfulness" (Alvarez Melis and Schlegel et al., 2019), "sensitivity" (Rebuffi et al., 2020), "robustness" (Yeh et al., 2019), "stability" (Fel and Vigouroux, 2020;Li et al., 2021), and a novel metric based on annotations: "localization". These metrics rate and validate distinct qualities of saliency. We further investigate how training choices and conditions, like the dataset, model architecture, and regularization methods, modify the quality and characteristics of visual interpretations. Based on this investigation, we give clear recommendations that allow experts to choose and validate appropriate visualization methods for time series data.
In our extensive evaluation, we train four different architectures on two different types of tasks: U-Time (Perslev et al., 2019) and bidirectional Long Short-Term Memory (bi-LSTM) (Schuster and Paliwal, 1997) on segmentation tasks, and Fully Convolutional Network (FCN) (Long et al., 2015) and Temporal Convolutional Network (TCN) (Bai et al., 2018) on classification tasks. We use diverse datasets from the UCR repository (Dau et al., 2018) (GunPointAgeSpan, FordA, FordB, ElectricDevices, MelbournePedestrian, NATOPS) for classification and the more complex real-world tool tracking dataset (Löffler et al., 2021) for segmentation. Additionally, we evaluate the effects of dropout, 1 , and 2 regularization. The experiments show the necessity of all categories to create an objective rating for methods, models and tasks, and highlight the importance of the regularization technique during model training. Our recommendations help domain experts to understand, rate, and validate saliency for time series in safety-critical applications.
The remainder of this article is structured as follows. Section 2 discusses background and related work. Section 3 introduces extended and novel metrics for both the classification and segmentation task. Section 4 discusses our experimental results. Based on these findings, Section 5 proposes recommendations. Section 6 concludes.

Background and Related Work
Interpretation methods for DL models can be divided into ante-hoc methods, that are inherently part of the model, and post-hoc methods, that provide the interpretation after training (Rojat et al., 2021). We focus on post-hoc methods and divide them into gradientbased and perturbation-based methods (Li et al., 2021;Warnecke et al., 2020;Ismail et al., 2020).
Gradient-based methods compute the relevance for all input features by passing gradients backwards through the neural network (Ancona et al., 2018). Gradient (Simonyan et al., 2014) computes the saliency map M c of a class c using the derivative of the class score P c of the model with respect to the input sample x, as M c (x) = ∂P c ∂x . The gradient indicates the importance of points in the input sequence for predictions. The advantage of gradient-based methods lies in their computational efficiency, as they use only a small number of backward passes to compute M c (x).
Perturbation-based methods perturb known input samples and measure the effects of specific perturbations on the predicted class via a forward pass through the network. For instance, Local Interpretable Model-Agnostic Explanations (LIME) (Ribeiro et al., 2016) fits a local surrogate model (e.g., a linear regression) as an explanation, and calculates relevance based on this surrogate. Perturbation-based methods are computationally expensive as they require multiple forward passes per sample. However, they do not need gradient information and hence also work with black-box models.

Metrics for Saliency on Time Series
Most interpretation methods were originally designed for image or text data. Understanding and comparing visual interpretation is intuitive on image data, compared to more abstract time series. Furthermore, the diversity of interpretation methods complicates an objective choice for the model and task (Rojat et al., 2021). For example, when Wang et al. (2017) and Fawaz et al. (2019b) apply Class Activation Maps (CAM) (Zhou et al., 2016) on well-known UCR datasets (Dau et al., 2018), they notice a qualitative difference of CAM interpretations between network architectures. This suggests that the difference in quality should be quantified. Similarly, other work relies on domain experts that perform a (costly) qualitative evaluation that is costly compared to an algorithmic one (Strodthoff and Strodthoff, 2019;Fawaz et al., 2019a;Oviedo et al., 2019). Hence, there is an increasing need for objective evaluation metrics to make the interpretations' quality measurable and comparable.

Categories of Metrics
Metrics can be divided into different categories, depending on the question they answer. Doshi-Velez and Kim (2017) propose a distinction between human-grounded and functional metrics. The former involve human perception and intuition with the goal of generating qualitative, intuitive visualizations, e.g., bounding boxes for image object detection for testing the localization of saliency maps (Jianming et al., 2016), or questionnaires to indicate participants' opinions on the quality of explanations (Li et al., 2021). Functional metrics provide statistical measures that aim to capture true model behavior instead, e.g., to aggregate performance metrics automatically (Rojat et al., 2021), and make use of proxy tasks to generate a quantitative evaluation. Another categorization (Li et al., 2021) identifies multiple broad categories for image data, that we may transfer to time series, i.e., the faithfulness of salient features, the class sensitivity of an explanation, and the stability of explanations given noisy inputs. This article proposes a framework of metrics with orthogonal categories, specifically to time series. We adapt and extend metrics to (multivariate) time series and propose an intra-class stability metric and a concept of relevance localization: we build on the pointing game (Jianming et al., 2016) and combine it with the precision and recall for time series framework (Tatbul et al., 2018). Furthermore, we provide an in-depth evaluation and discussion of saliency methods and the influence of regularization on time series classification.

Scoring Categories
We propose a set of six distinct categories (sanity, faithfulness, sensitivity, robustness, stability, and localization) to assess visual interpretation methods and to determine their performance and trustworthiness in classification or segmentation tasks on time series. For each of them we propose a metric that enables a comparative evaluation of diverse types of visual interpretation methods. Why do we need six scoring categories? It seems tempting to rely on a single metric or on a single aggregated score across multiple metrics to assess the quality of a visual interpretation. However, we show that the six presented categories are orthogonal (see Fig. 2) and capture distinct qualities. Interpretations depend on model parameters (sanity check (Adebayo et al., 2018)), predictive features (faithfulness (Alvarez Melis and Jaakkola, 2018)), coherence of class predictions (intra-class stability (Fel and Vigouroux, 2020)), robustness against noise (max sensitivity with adversarial noise (Yeh et al., 2019)), specific sequences (or even points) in a time series like by Tatbul et al. (2018) (novel localization metric), and the relevance map's specificity (inter-class sensitivity (Rebuffi et al., 2020)). It is important to assess if a given interpretation accurately captures these dependencies.
We give a brief overview of our framework's metrics in with respect to a suitable distance metric (Fel and Vigouroux, 2020). (f) saliency should be localized on the segments of the predicted classes (t 0 to t 1 ).
Notation. We first introduce a unified notation that we adapt from Fawaz et al. (2019b): A multivariate time series is defined by X = [X 1 , ..., X H ], where H is the number of input channels, X i = (x i 1 , ..., x i T ) ∈ R T is an ordered set of real values, and T denotes the number of time steps. For H = 1, we consider a univariate time series, otherwise we consider a multivariate time series. Time series often include complex temporal dependencies, i.e., distinct points are not independent. Time series classification defines a mapping X → y that minimizes the error on a dataset D = {(X 1 , y 1 ), ..., (X N , y N )}, where N is the number of data samples, X ∈ D is a time series, y i ∈ R C denotes the one-hot vector of a class label the input belongs to, and C is the number of classes. In time series segmentation, we search X → Y that maps an input sample to a dense classification Y = [y 1 , ..., y T ] ∈ R C×T (Perslev et al., 2019), i.e., a class label is predicted for each time step. Post-hoc visual interpretation methods compute a relevance map M c m ∈ R H×T , M c m (X) = [R 1 , ..., R H ], where R i = (r i 1 , ..., r i T ), representing the importance of each input feature with respect to the class c and a model m, for each time step. We use M as a function to produce the saliency map. For clarity, we will omit the dependency on m, i.e., M c m ≡ M c , if it is not explicitly required. An evaluation metric for visual interpretation methods defines a score S metric (·) that rates the quality of the relevance map M at a sample X given a model m and optional parameters. We provide a unified view, so that for all scores, a higher score corresponds to a better visualization according to the perspective.

Sanity
The sanity metric measures the idea that a visual interpretation depends on the model that is interpreted. Intuitively, if the weights and biases of a trained network model were re-initialized with random values, the network's predictions and generated saliency maps should also be different from the original maps. However, this is not always the case. Despite a drop in model accuracy, saliency may remain stable. Hence, we test sanity using a variant of sanity check (Adebayo et al., 2018), that performs a layer-wise cascading randomization of the network's weights and biases, starting from the output to the input. In contrast to independent randomization, the cascading approach results in a mostly continuous performance degradation of predictions, see Fig. 1a for an illustration. Network accuracy should increasingly resemble a random guessing. Following Adebayo et al. (2018), we compare saliency using the structural similarity index measure (SSIM) (Wang et al., 2004), that compares the distribution of sub-sequences of M c . We define the sanity score as where |D| = N , i enumerates the L layers of m, whose parameters are randomized and M c m i (x) is the saliency map after randomizing layer i of m. We average the SSIM over L layers and compute the average over all samples in D.

Faithfulness
A relevance measure is faithful, if input features with a high relevance (w.r.t. M c ) have a high influence on the model prediction, see Fig. 1b for an illustration. Alvarez Melis and Jaakkola (2018) propose a perturbation-based metric that evaluates the faithfulness of predictions. The metric measures the correlation between input (features) with high saliency on the one hand and predictive accuracy on the other hand. For time series data, there are two variants of faithfulness that are based on different assumptions of the data distribution. Both rank inputs of time step of X T ∈ R T according to the relevance of its saliency map. However, they differ in perturbations: Alvarez Melis and Jaakkola (2018) (further called Temporal Importance (TI)) strictly adhere to the ranked importance of saliency to choose what to perturb, while Schlegel et al. (2019) assume temporal correlation of inputs and perturb a connected sub-sequence instead (Temporal Sub-sequences (TS)). While the latter is a more intuitive model for perturbations on time series data, it may misrepresent importance of features, e.g., if they are not clustered or do not follow the assumption that they should be sequential. Furthermore, there may be several crucial sub-sequences in a single sample.
We use either variant of faithfulness on multivariate data X H T as follows. Based on the relevance r, we select time points t ∈ {1, ..., T } and features i ∈ {1, ..., H}. For TI, we select and perturb the inputs of the sample X with highest relevance r i t and replace their values with a mean value of the test dataset at time t. For TS, we select t according to the maximum relevance r i t and perturb a sub-sequence of the inputs of the sample X by replacing its values with a mean value of the test dataset of length L at time t, as ), to break the temporal correlation.
The perturbed sample is denoted as X . We compute the mean faithfulness scores over the whole dataset D for TI as and for TS as where m c : R c → R is the softmax prediction of the target class c. For TS the gap score between the softmax prediction of the original and the perturbed sample is m c (X)−m c (X ). For TI we use the Area Under Curve (AUC) instead, where L is the total perturbation length and X l is the perturbed sample at step l.

Inter-Class Sensitivity
In multi-class prediction tasks the classifier needs to identify relevant features for each of the classes to make a correct prediction. Hence, the relevance map M c should identify different salient features for those different classes (Li et al., 2021), see Fig. 1c. If a method is not sensitive to the class, the score may indicate that the saliency is misleading, or that the classifier failed to learn correct features for these classes, and the sensitivity score should raise a red flag. Concretely, inter-class sensitivity (Rebuffi et al., 2020) measures class specific sensitivity of the generated relevance map with respect to the most likely (c max ) and least likely (c min ) class according to the model. We compute the mean inter-class sensitivity score as We compute similarity of two saliency maps as sim(M cmax (x), M c min (x)) where sim(·, ·) is a similarity function (e.g., a cosine similarity) that is easy to interpret via its geometric interpretation. It is defined as the angle between two non-zero vectors that measures the similarity between their inner product space (Han et al., 2012). Similarity of M in binary classification would result in a negative cosine similarity, meaning nearly inverted saliency maps for max-and min-classes. The sensitivity of the saliency map is not always strictly related to how well the saliency method captures model behavior. Saliency methods of low faithfulness could introduce their bias to the sensitivity score and hide true model behavior. However, inter-class sensitivity is still a useful indicator for the quality of a model and visualization, with the important caveat that a score is only meaningful if the visualization is both faithful and sane. See Section 5 for further details and recommendations on this.

Robustness
Recently, Dombrowski et al. (2019) showed the vulnerability of saliency methods to adversarial attacks. For reliable interpretations, saliency methods should be as robust to small changes in the input as the model itself. We evaluate a method's robustness with respect to adversarial noise (Yeh et al., 2019) via its sensitivity of the most likely class (c max ), see Fig. 1d for an illustration. Intuitively, even with noisy inputs the saliency map of a model should not change significantly (Alvarez-Melis and Jaakkola, 2018) and saliency maps with high sensitivity are less reliable. Yeh et al. (2019) define the sensitivity of the saliency map derived from the gradient as for any j ∈ {1, ..., |H · T |}, where e j ∈R H×T is the j-th coordinate vector and the j-th entry is one while the others are zero. We use Monte Carlo sampling of e j , where | e j |<a (a is a user-specified radius), to generate differentX = X + e j . We compareX with the original X to compute In summary, evaluating the robustness of saliency methods should not hide the robustness, or lack thereof, of underlying models. Especially if these models are not robust themselves, saliency methods should accurately reflect their vulnerable behavior. Hence, robustness of saliency methods and models should be jointly evaluated.

Intra-Class Stability
Fel and Vigouroux (2020) propose algorithmic stability measures to test whether saliency methods produce similar interpretations for samples from the same class. Two explanations of the same feature should be similar, independent of their exact location within the input sample. This is helpful for generalization, as features may be translation-invariant.
To be able to compute this metric, we need to make several assumptions. First, we assume that the order of temporal features is correlated between different samples of the same class. Overall, this is a reasonable assumption, since time series data often contains distinct temporal features that follow a dataset-specific order. However, note that scores for intra-class stability are highly dataset-specific, and should not be compared across different datasets. See Fig. 4 for this evaluation. Second, we assume that the visualization method is faithful, and that both the visualization and the model are robust. This assumption has to be validated for every combination of model and visualization technique.
Based on these assumptions, we can test the intra-class stability of saliency maps M c for a given dataset D using distance statistics, see Fig. 1e for an illustration. Specifically, we compute pairwise distances between saliency maps M c for different samples X i , X j ∈ D and aggregate these across classes: We use Dynamic Time Warping (DTW, Vintsyuk (1968)) as the distance function d DTW . The score compares each classes' sample's M c with all other samples' saliency maps in the dataset, using d DTW . DTW provides a crucial benefit by allowing us to compare time series that have similar features in the same order, but not on the exact-same point in time.

Localization
The temporal location of class-specific features with high relevance in a time series segmentation task should intuitively be situated within (or close-by) its labeled segment (temporal sub-sequence). This labelled sequence should also determine the location of saliency for models, see Fig. 1f for an illustration. Localization is a novel metric that measures this notion and allows experts to uncover potential problems in model-operation like shortcut learning.
Similar to intra-class stability, localization requires a faithful visualization. Additionally (but somewhat obviously), accurate labels are needed to compute a meaningful score. Given a faithful visualization, samples with lower scores (lower overlaps) warrant closer inspection. A low localization indicates spurious correlations due to unexpected features outside of the labeled sequence and can also help pinpoint faulty labels.
Our metric is an extension of the Pointing Game (Jianming et al., 2016) metric, that was previously used for object detection, to time series using range-based metrics (Tatbul et al., 2018). Localization systematically analyzes the saliency maps' ranges in segmentation tasks. We argue that Pointing Game's original hit-and-miss accuracy is inadequate for time series methods, as segmentation is point-wise exact. Thus, we replace it with range-based testing, yielding localization. This extension with front, middle and back biases can help uncover problematic issues such as temporal biases within the labeled sub-sequence or imprecise margins of the labels themselves. Furthermore, the metric may hint at learned shortcuts outside of the labeled sequence.
We compute localization as follows. First, we filter out non-relevant predictions (according to the relevance map). A prediction at time t, denoted withŶ (t), is relevant if there exists an i ∈ {1, ..., M } such that r i t > max(|r|) · θ where θ is a value between [0, 1]. We select the model's predicted class at time t if the prediction is relevant, otherwise we set the class to none. The resulting relevancy-filtered prediction is denoted asŶ . Finally, we can compareŶ with the ground truth Y and evaluate how well high saliency features lie within the annotated sub-sequences (Tatbul et al., 2018).
To compute the mean localization score for the whole dataset D, we compare any of the N sub existing labeled sub-sequence Y sub i ∈ R T from D each with its temporally co-located For each pair (Y sub i ,Ŷ i ) we calculate a range-based recall score based on the point-wise comparison proposed by Tatbul et al. (2018) as where α weighs the two reward terms for "existence" and "overlap". Existence is 1 if any time point was correctly predicted within the labeled region, and 0 otherwise. The parameterized "overlap" function determines the finer properties of cardinality, size, and position. The cardinality parameter discounts the score if the prediction is an interrupted and fragmented range instead of being continuous. The overall size of the overlap of predicted and label ranges depends on a positional bias. It may favor "front", "middle", or "back" overlap. Practically, for some applications an early detection is preferable over a late detection.
Salient features outside of the labeled range Y sub i can be scored analogously, simply by inverting the range. For further details see Appendix A or Tatbul et al. (2018).

Experiments
Our evaluation compares visual interpretation methods on a set of network architectures and diverse classification datasets from the UCR repository, and on a more complex segmentation dataset. In total, we aggregate the results from 729 experiments. This section is divided into three parts. First, we discuss the metrics for the classification task (faithfulness, sensitivity, stability, robustness, sanity). Next, we investigate the influence of regularized model training on the faithfulness variants TS and TI and explain their differences. Finally, we discuss the new localization metric for the segmentation task. The code is available at https://github.com/crispchris/saliency.

Evaluation on Classification Task
In this section, we discuss the six visual interpretation quality metrics for the classification task. Fig. 3 summarizes the scores for each metric over model architectures and datasets. Fig. 4 summarizes the scores for each metric over model architectures and methods.
Faithfulness. We evaluated both variants of faithfulness metrics and found significant differences. In general, methods are able to achieve higher scores in the Temporal Importance variant, compared to Temporal Sequence. This indicates that the visualiza- tion methods are able to pinpoint individual features very faithfully, but fail to accurately capture dependencies on longer sequences. We also observe a relatively low correlation between methods -the visualizations that performed best in TI, i.e., GradCAM and Guided GradCAM, achieve only comparatively low scores in TS.
TS assumes that features are clustered and sequential. We hypothesize that this assumption is not valid for many time series domains. In particular, TS shows very low scores of low-variance for the FordA and NATOPS datasets in Fig. 4, which indicates that this assumption is potentially violated. Especially the multivariate NATOPS seems to exhibit diversely distributed features. For these reasons, we focus the following discussion on TI.
The TI scores of GradCAM and Guided GradCAM are above average, and Integrated Gradients below average. The more coarse GradCAM even slightly overtakes its Guided variant, likely due to GradCAM capturing the higher level features, and Guided-GradCAM's sample-level interpolation introducing a bias. Integrated Gradients may fail due to the bias introduced by its user-defined baseline. The other methods perform similarly on average. The dataset centered view in Fig. 4 shows that the complex NATOPS dataset is the hardest to explain. This may be due to the potentially multi-dimensional features that are not perturbed.
Sanity. Across all datasets, we find that Gradient, Integrated Gradients, LIME, and Kernel-SHAP consistently achieve high sanity scores. In contrast, LRP, Guided GradCAM and Guided Backpropagation often assign similar saliency even after randomizing network parameters. This replicates sanity results by Adebayo et al. (2018). Low sanity methods highlight features in the input that may also be randomly extracted but do not necessarily predict classes.
Intra-class Stability. This metric is highly dependent on the dataset, e.g., centered samples as in GunPointAgeSpan. However, when accounting for variance introduced by the dataset (see Fig. 10b in Appendix C.2), SmoothGrads, LIME, and Kernel-SHAP show unstable saliency maps for the same classes. Not only the location in the sample, but also the warped distance between samples using DTW is different. Guided GradCAM is able to produce highly stable saliency maps across all datasets, likely due to its lower resolution.
It is important to notice that some datasets, e.g., FordA, do not necessarily have stable or similar features. However, many real datasets exhibit clearly stable features across classes, as evidenced through considerably higher scores for the ElectricDevices, GunPointAgeSpan, MelbournePedestrian, and NATOPS datasets. We found that a visual analysis of samples can be a useful indication for the inherent intra-class stability of a datasets -datasets with high scores exhibit relatively clear temporal features in specific orders, while those with low scores generally showed less localized features.
Methods that introduce additional biases into the visualization calculations, like Integrated Gradients and Guided GradCAM, can produce highly stable saliency maps. LIME and Kernel-SHAP look at individual sample points independently, and thus produce unstable saliency maps. We hypothesize that SmoothGrad's addition of noise to sharpen the saliency maps can lead to an unintended side effect of over sharpening and overly focusing the saliency map onto a single, sample-local feature, that does not generalize to the rest of the dataset, which in turn leads to the low intra-class stability.
Robustness. Leaving aside the underlying model's robustness, our evaluation of Gradient shows it is less robust, a finding that confirms Smilkov et al. (2017). Furthermore, LIME and, interestingly, SmoothGrads also show a low average robustness. The additive noise of SmoothGrads may compound and lead to diverging saliency maps. On the other hand, LRP, (Guided) GradCAM, and KernelSHAP show relatively robust saliency maps. Compared to purely gradient-based metrics, LRP and GradCAM use class activations, which might have a regularizing effect. Similarly, Kernel-SHAP can be thought of as a more constrained version of LIME. Overall, the saliency methods should capture the underlying model's robustness and not produce saliency for divergent behavior, due to introduced biases. The robustness metric's weakness is its computational cost: on {TCN,ElectricDevices} it reaches its time-out of 12 days.
Sensitivity. GradCAM shows the highest sensitivity for classes -the saliency maps for the most and least likely classes differ largely. On the other hand, Integrated Gradient and Guided Backpropagation produce similar saliency maps, independent of a sample's class. This holds across all datasets. A possible explanation for this is that methods that introduce additional biases into the visualization calculations might lead to less sensitive saliency maps. Integrated Gradients integrates from a user-defined baseline. Guided Backpropagation intentionally suppresses the negative part of the gradient signals to find more visually focused and intuitive interpretations. While these biases can improve the visual appeal of the saliency maps, our results suggest that they have a not negligible impact on saliency maps and should be avoided when sensitivity is a concern.

Influence of Datasets and Models
First, datasets heavily impact scores, see the outliers in Fig. 3. The plots of the categories over each dataset for all {model architectures, methods} in Fig. 4 emphasizes this finding. The choice of visual interpretation method depends, for the most part, on the task. Still, after normalizing for dataset bias, the relative scores do not diverge significantly but confirm the initial scores. We report the relative scores in Fig. 10b in Appendix C.2. Surprisingly, the model architecture (FCN or TCN) only contributes little to the scores, see Fig. 10c in Appendix C.2. The scores for FCN and TCN diverge slightly more for faithfulness TS and TI. This may be due to the higher capacity of the TCN, which learns more diverse features, so that its TS score is lower and its TI score is higher. The FCN may learn fewer and less predictive features that are clustered more but are overall less predictive, resulting in higher TS score and lower TI score.

Influence of Regularization
Neural networks (among other parametric models) benefit from regularization techniques such as weight decay or dropout (Srivastava et al. (2014)). The positive effects can include increased accuracy and better generalization, essentially as a result of a decrease of overfitting to training data. However, we show that there are also unintended side effects of dropout, 1 and 2 regularization on the faithfulness of saliency methods, for which we give recommendations. We focus this section on an investigation of faithfulness, because other metrics only have an indirect relation to accuracy and thus regularization.
Hypotheses. We hypothesize differences in faithfulness due to regularization effects. Dropout regularization can be understood as computing the average of a number of thinned networks (Srivastava et al., 2014). Regularized models rely less on individual features. Hence, models should score higher in Temporal Importance, because of its arbitrary choice of input features of the highest saliency, and lower for Temporal Sequence that has a bias towards clusters of important features. Weight decay (Plaut et al., 1986), such as 1 or 2 regularization should result in the opposite learning effect. An 1 loss term causes sparsity of the parameters and supports finding the smallest required feature set. Models may learn fewer features, that in turn are of higher importance (Krogh and Hertz, 1991). This should result in higher scores for TS. More specifically, 2 regularization optimizes models to learn more features that are of similar importance, and thus equalize the importance of learned features (Plaut et al., 1986). Hence, their saliency's faithfulness should perform similarly with the TI and TS variants.
Accuracy. Fig. 5 shows how regularization impacts the accuracy of an FCN on FordA, FordB and NATOPS in the top row of plots. We show the relative change from the nonregularized baseline, indicated by with the hyper-parameter λ = 0.0, and average the results over two random seeds. Changes in accuracy vary between dropout, 1 and 2 and also between datasets. The accuracy on the FordA dataset is relatively stable with regularization, except for an outlier at the large 1 reg. of λ = 0.05. In contrast, the accuracy declines on FordB, with a downwards trend for increasing values of λ. Interestingly, the FCN benefits from regularization on the more complex NATOPS dataset. Dropout reliably improves accuracy, while the positive effect of weight decay is not as stable for larger λ of 0.05 ( 1 ) and 0.03 ( 2 ). The effect of a too large 1 weight decay is most detrimental, the model even fails to learn the two-class task FordA.
Absolute Faithfulness. We show the saliency methods' aggregated Faithfulness TI and TS scores. Despite mostly similar accuracy for the regularization schemes, faithfulness scores differ strongly between them. For TI, a small dropout regularization produces better saliency than 1 or 2 regularization across all datasets, even though the accuracy for models is close, e.g., models for FordA perform within 1 %. Interestingly, the opposite holds for TS, where 2 regularization has a positive effect on the FordA and FordB datasets. 1 regularization displays a similar effect on all datasets. This confirms our hypotheses, that enforcing feature sparsity can have different effects on faithfulness TI and TS. We observe outliers for dropout regularization with λ = 0.2, that markedly improved faithfulness scores across datasets. A meaningful faithfulness score could not be computed for 1 regularization with λ = 0.05 on FordA, because the model did not exceed the performance of random guessing.
Accuracy and Faithfulness. Interestingly, we do not see that an increase in accuracy, e.g., for NATOPS or for 2 with λ = 0.01, consistently correlates with an increase in a TI score. This may be due to the limited fraction of input values (we use 20%) that we perturb while calculating TI. However, a decrease in accuracy tends to come with worse TI. Additionally to less accurate models, there may be a second effect at play. The thinning of features due to weight decay leads to fewer learned features. Perturbing less predictive features naturally results in lower TI scores. The FCN's concentration on fewer features with 1 regularization is especially apparent in the Ford datasets' trends in TS. This variant of faithfulness has a bias towards clustered features and benefits from feature sparsity.
We conclude from these experiments that regularization is important for both accuracy and faithful saliency. Any type of faithfulness directly depends on the relationship between accuracy of the models and saliency maps, even if it is not directly apparent when using a fixed fraction of perturbation when calculating TI, because regularization may increase or reduce feature sparsity in the model. Practitioners should choose a suitable regularization depending on the faithfulness variant. Similar test accuracy does not automatically imply similar faithfulness of saliency. This may be due to the numerical effects of weight decay and dropout on the networks' parameters and feature learning.

Evaluation on Segmentation Task
This section presents the key findings from our evaluation when applying the localization metric on saliency maps generated from U-time (results for bi-LSTM are in Appendix C.1), trained for a segmentation task on the tool tracking dataset (Löffler et al., 2021). Fig. 6a shows the localization classical positional bias for each visualization method. Fig. 6b shows an exemplary saliency map for each method. Fig. 7 summarizes the statistics for the biases "classic", "front", "middle" and "back".
At first glance, GradCAM produces saliency maps with top scores in the localization metric. Additionally, its saliency maps appear much smoother and continuous, with high relevance located on the annotated segments. The high fidelity saliency map of Guided GradCAM instead produces lower scores due to its more nuanced relevancy assignments. This difference results from GradCAM's coarser relevance, as its gradients do not flow to the input but stop at the last convolution layer. In contrast, LRP and Integrated Gradient produce lower scores. As Fig. 6b shows, the saliency maps highlight fewer relevant input features for the segmentation task, but instead features within the labeled sub-sequence that may be important for a classification.  Analysing the scores in Fig. 7 shows that saliency maps are located more in the "middle" of the annotated segments, compared to "front" and "back". This is also visible in the example in Fig. 6b where the relevance values start decreasing below the threshold of 0.5 already before the end of the labeled segment at 100 ms. This points towards a biased labeling process with longer sub-sequences assigned to the classes than necessary, and can hence help identifying faulty labels. Furthermore, the high variance of most results (except for GradCAM) shows that saliency maps for time series segmentation are noisy. This is especially the case for Kernel SHAP, where relevance is assigned seemingly at random.
GradCAM assigns saliency nearly perfectly to the labeled segments, compared to the other methods in this study that only marginally align with the annotated segments in the input features. Their saliency maps are comparatively noisy. However, each methods' faithfulness has to be considered before choosing a saliency method to trust for producing interpretations, or even locating faulty labels or even spurious correlations. Interestingly, no method worked reliably on the bi-LSTM. We identify a lack of useful visualization methods for LSTMs, as CAM variants (GradCAM, Guided GradCAM, and Guided Backprop) are not applicable, see Fig. 9 in Appendix C.1 for detailed results.
In summary, localization scores of saliency for faithful models provide useful analysis of temporal biases and label inconsistencies, and can support the detection of shortcut learning. It is a red flag if an otherwise faithful saliency method has a low overlap of the saliency map and the potentially faulty label, or spurious correlations may be present. If saliency maps lie well within the labeled sub-sequence, but the saliency itself is not faithful, the score may hint at a bias present in the saliency method.

Recommendations
The quality of visualizations differs greatly between methods. No method passes the tests for all categories on all six datasets. This emphasizes the need to evaluate all metrics for every visual interpretation. We recommend to use a summary as in Fig. 8 to judge visualizations on every category, and propose the following guidelines for relative ranking. The absolute scores may be understood in comparison with a random baseline, similar to shuffled AUC (Borji et al., 2013).
Setup model training and regularization hyperparameters. As an initial step, we recommend to select a suitable set of regularization methods and their hyper parameters to optimize the models, depending on the task at hand. Dropout, 1 or 1 regularization may have a large effect on model's inductive bias, and thus also on faithfulness. This also depends on the datasets' feature distribution. Note that the right regularization can meaningfully improve the model interpretability, and we suggest changing parameters if objective metrics indicate a low trust in visualizations.
First: Ensure Faithfulness and Sanity. The general purpose of interpretability methods is to provide insights into model behavior. We propose to use the faithfulness and sanity scores to ascertain that a saliency map represents the model behavior. Faithfulness ensures that the saliency matches the model's predictive features. Note that, depending on the prevalence of temporal correlations and sequential features in a dataset, an expert should choose between TS (when sequential features are important) and TI (when this is not the case). When in doubt, we propose to use TI.
Sanity checks confirm that saliency maps are sensitive to model parameters. This is important to avoid finding highly salient features, such as edges in images (Adebayo et al., 2018), while being insensitive to model parameters. We avoid saliency maps with low scores in either metric, as they do not reflect the model behavior. See Fig. 8 for an example: Gradient, Integrated Gradients, and LIME achieve the three highest-ranked scores. However, random saliency maps (second to the last column) are dissimilar (hence they also show high sanity scores) and their relatively high TI score of 0.57 hints that the faithfulness of the other methods is not completely reliable and additional tests are required. Guided Backprop, Guided GradCAM and LRP fail the sanity check, while Kernel-SHAP performs poorly in faithfulness. Second: Check Sensitivity and Robustness. Once faithfulness and sanity are established, we propose to look at sensitivity and robustness of the generated saliency maps. A low inter-class sensitivity can indicate that the saliency maps only focus on the predicted class and underestimate the importance of features that do not belong to this class. A low robustness score suggests that the visualization method is susceptible to adversarial examples and small perturbations in the input. Given that the model predictions are robust (Zhang et al., 2019), and the saliency is faithful, a low robustness implies that the saliency method cannot be trusted, and may even be manipulated (Dombrowski et al., 2019). For non-robust models we recommend to rely less on visual interpretations, especially if they have low faithfulness. According to Fig. 8, we keep Gradient and LIME, but disregard Integrated Gradients due to its lower sensitivity.
Detailed Analysis: Analyze Intra-class Stability and Localization. Intra-class stability measures how much saliency maps for one class agree between different samples. The localization metric, like other Pointing Game-like metrics, measures semantic precision based on annotations. They allow experts to choose and analyze saliency methods that are more intuitive and understandable. Note that it is crucial to ensure the faithfulness of a method before relying on this metric. If the method is not faithful, then stable and localized visualizations are visually pleasing, but do not reflect model behavior. This can hide issues like spurious correlations (Arjovsky et al., 2019) or the shortcut learning problem (Geirhos et al., 2020) behind a higher score.

Conclusion
We propose an evaluation scheme for visual interpretations on time series data and provide extensive empirical results for different visualizations, models, datasets, and regularization methods across time series classification and segmentation. We recommend six orthogonal metrics that provide a disentangled evaluation of important characteristics of visual interpretations. All these metrics should be evaluated for every visual interpretation to prevent relying on interesting but spurious results. We also show how each perspective evaluates different strengths and weaknesses of visualization methods, and give recommendations on their use. The main findings are: • No individual visualization method achieves high scores on all evaluation methods. For practitioners, we thus recommend to focus on faithfulness (TI) and sanity first. • The strong impact of datasets onto the saliency methods was consistent across all evaluation perspectives. This indicates that the interpretability of the underlying model is similarly dependent on the learned distribution as its predictive accuracy. • Similarly, visual interpretation methods may introduce an additional bias into the visualization calculations. We show that this can lead to highly stable saliency maps.
• Regularization during training had a large, but inconsistent influence on faithfulness.
High regularization tends to decrease TI, but increase the TS variant. • Model architecture of convolutional networks alone had a relatively low impact on evaluation scores for visualization methods, compared to MLP or LSTM, that were noisy or suffered from vanishing saliency. • Our novel localization metric for segmentation analyzes if saliency is assigned correctly to local features and help to evaluate label ranges. GradCAM displays an outstanding localization of features, surpassing other methods.
Overall, our work demonstrates how the explanations of state-of-the-art visualization methods differ wildly. Using faithfulness and sanity scores, experts can distinguish between plausible but wrong saliency maps. Since visualization methods are used to explain neural networks to human experts, more quality measures are necessary. Explanations need to be sensitive for classes, robust (if the model is robust) and stable (if model operation is consistent), while staying true to the model behavior. We combine these requirements into a simple and clear framework that can be used to assess visual interpretations on time series data for arbitrary neural networks.
Future work investigates the relatively high scores of the random baseline in the two main metrics sanity and faithfulness. Furthermore, our study of regularization shows that model training should be optimized for more reliable interpretability, so that they behave more intuitively, without sacrificing predictive performance. Finally, the localization metric may be adapted to detect spurious correlations in time series problems.

Reproducibility
This paper includes supplemental materials to improve reproducibility. Most importantly, we publish the code for model training and dataset loaders, for generating visual interpretations and of all evaluation metrics. Furthermore, all datasets are publicly available.
Appendix A.

A. Localization Metric Positional Biases
This section explains the "existence" and "overlap" terms of our localization metric. Note again the recall score as The "existence" and "overlap" terms are defined by Tatbul et al. (2018) as follows. First, existence is defined as a correct prediction of one sample point at index j of the correct class within the labeled region: The overlap determines the finer properties cardinality γ(), size ω() and position δ().
The term α (0 ≤ α ≤ 1) weights existence and the following qualitative measures of overlap. The three user-defined functions return values of 0 ≤ γ() ≤ 1, 0 ≤ ω() ≤ 1 and δ() ≥ 1. The cardinality γ() function weighs the prediction of a continuous range Y covering the whole labeled range Y sub versus interrupted ranges in a fragment manner. The overall size ω() of the agreement of predicted and label ranges depends on the positional bias δ() and can favor "front", "middle", or "back". Practically, for some applications an early detection is preferable to a late detection. For the four functions, we selected α is equal to 0, γ is equal to 1. ω depends on the position bias δ and for δ we have three variants, "front", "middle" or "back", proposed by Tatbul et al. (2018).

B. Experimental Setup
We describe each visual interpretation method shortly in Sec. B.1, then the hyper parameters for each visual interpretation method in Sec. B.2 and for the optimizer and networks in Sec. B.3.

B.1 Categorization of Methods
This Section provides a short introduction to the methods that we use in our experiments.
Gradient-based methods. Gradient (Simonyan et al., 2014) computes class c's saliency map M c using the derivative of the class score P c of model with respect to the input sample x, as M c (x) = ∂P c ∂x . However, Gradient suffers from the saturation problem (a feature may have global importance, but its local derivative is small (Smilkov et al., 2017)), local sensitivity, and noisy saliency maps (due to (sharp) local variations in the gradients (Smilkov et al., 2017)). Follow-up work smooths the gradients to reduce noise (Smilkov et al., 2017), applies special propagation rules instead of propagating a gradient (Arras et al., 2017), or propagates only up to a specific intermediate layer (Selvaraju et al., 2017).
Perturbation-based methods. Ribeiro et al. (2016) proposed Local Interpretable Model-Agnostic Explanations (LIME), that fits a local surrogate model (e.g., a linear regression) as an explanation and uses this surrogate to calculate relevance. Kernel SHAP (Lundberg and Lee, 2017) builds on LIME, but calculates Shapley values that measure the contribution of individual features to the input more accurately.

B.2 Visualization Methods Hyper Parameters
We set the hyper parameters for each visual interpretation method according to their recommendations from literature. We provide the reasoning behind the selected parameters in the following paragraphs.
Gradient computes the derivative of the target class score with respect to the input sample and returns the saliency map of the input sample at the end. Integrated Gradient uses the linear path method to compute gradient along the path from a baseline x . We use a zero vector for the baseline. As suggested by Smilkov et al. (2017), the number of steps for the path should be selected between 20 and 300. Hence, we use steps N = 60, meaning that it takes 60 steps from baseline x to the original input sample x, according to x = x + (x−x ) N · n, n is the current step.
SmoothGrad also computes the derivative of the target class score with respect to the input sample. However, it adds Gaussian noise N (0, σ 2 ) to the input sample multiple times and computes the gradients from the perturbed samples (x + N (0, σ 2 )). The number of iteration of adding noise is chosen N = 60 and the standard deviation of Gaussian noise is chosen σ = 0.2.
For LRP, we select the -propagation rule for every DL model with = 1e − 9. Due to the residual block in TCN model the propagated relevances should therefore be added together (R = R 1 + R 2 ).
GradCAM and Guided-GradCAM are designed for models with convolutional layers, i.e. FCN, TCN and U-time. GradCAM focuses on the last convolutional layer, which produces feature maps, whose shapes usually are smaller than the shape of input samples. Therefore, we use interpolation to up-sample the saliency maps from the last convolution layer to match their shape to the shape of the input samples, which allows us to visualize them in the input space. Because of ReLU functions, GradCAM returns only positive relevances.
For LIME, we use the cosine distance function as the kernel function with width w = 5.0 to weight the perturbed samples, and perform 1000 iterations. For the perturbation, we consider that neighbors along time dimension should have similar relevance to reduce the computational time, so we set the number of features along time dimension 50 for GunPointAgeSpan. This means that saliency maps for samples with length 150 in Gun-PointAgeSpan have same relevance for every 3 neighbors and saliency maps for dense labeling samples in tool tracking have same relevance for every 4 neighbors.
For Kernel SHAP, we use 1000 iterations and set the number of features along time dimension to 50. Furthermore, the sampling of feature perturbation in Kernel SHAP is based on the distribution p(f ) = (F −1) (f ·(F −f )) , where f is the number of selected features and F is the total number of features in interpretation space.

B.3 Network Architectures
We use the Adam optimizer with a learning rate of 0.002. We train every dataset for 600 epochs (with early stopping after 80 epochs). We use a Cross Entropy loss for time series classification and a Generalized Dice loss with Cross Entropy function for time series segmentation.
For the time series segmentation task on the tool tracking dataset, we report an accuracy of 83% for U-time and 85% for the bi-LSTM.

B.3.1 Fully Convolution Network
We use a slightly modified FCN, similar to Wang et al. (2017). Ours contains four convolution blocks with a convolutional layer, a batch normalization layer (Ioffe and Szegedy, 2015) and a ReLU layer in each block. The kernel shapes and numbers of filter for convolution layers are {7, 5, 3, 3} and {16, 32, 32, 16}. Therefore, there are four convolutional layers. Each convolutional layer has unit stride and no padding, which means the time sequence will be reduced continuously by the blocks. The final convolution block, which is behind the four convolution blocks, does not have a ReLU layer and contains a 1x1 convolutional layer. The 1x1 convolutional layer serves as a projection layer. It can not only reduce the channel size of feature maps but also keep their salient features. Finally, we apply Global Max Pooling on the features maps, before the softmax operation.

B.3.2 Temporal Convolution Network
For the TCN, that was first proposed by (Bai et al., 2018), we use a global pooling layer for the prediction. In our architecture, TCN has the convolution filters {16, 32, 32, 32} and kernel shapes for convolution layers {7, 5, 5, 5} in four residual blocks. Therefore, the total number of layers is 8.

B.3.3 bidirectional Long Short-Term Memory
We use a standard, single-layer bi-LSTM to predict dense labels for the segmentation task, as implemented in PyTorch, with 512 hidden units. There is a dense layer behind the LSTM model to fit the hidden units 512 to the number of classes. Also, the dropout rate is set to 0.2 to prevent overfitting.

B.3.4 U-time
We use the U-time (Perslev et al., 2019) architecture with the following configuration. In each convolution block, there are two dilated convolution layers with dilation 3, followed by a ReLU layer, a batch normalization layer, and a Max-Pooling layer at the end. The number of filters for convolution layers in four convolution blocks are {16, 32, 64, 128} and the pooling window sizes are all 2. Two additional convolutions with filter numbers {256, 256} follow after four convolution blocks. In each transposed convolution block, a nearest-neighbor upsampling (Odena et al., 2016) of its input is implemented, followed by a dilated convolution layer with dilation 3, a ReLU layer and a batch normalization layer. The number of filters for convolution layers in four transposed convolution blocks are in the reverse order of the encoder {128, 64, 32, 16}. The kernel size of convolution layers in both encoder and decoder is 7.

C. Additional results
We report additional results to supplement our discussion for a bi-LSTM architecture on the segmentation task in Sec. C.1, and for the classification task in Sec. C.2.

C.1 Evaluation on Segmentation Task: bi-LSTM
The results for bi-LSTM on the segmentation task for tool tracking in Fig. 9 show that none of the visual interpretation methods, that are compatible with the model's architecture, produce satisfactory saliency maps.

C.2 Classification Task
We show the variability of each visualization method with respect to each dataset and metric category for the TCN in Fig. 10a. It is noteworthy that some datasets are more problematic for the methods, especially for faithfulness and intra-class stability, than others. We also show that the model has only a small influence. For this result, we aggregate all datasets and visualization methods separately for FCN and TCN in Fig. 10c.
In order to compare methods independent of datasets, we control for their bias by normalizing the scores of all methods for each {category, dataset} so that their mean is 0 and variance is 1. With this, we can correctly assess the "relative" performance of the visual interpretation methods across different datasets. The plot in Fig. 10b shows these relative (or marginal) scores that a method can achieve, compared to other methods on the same datasets.
We prove the orthogonality of metrics in Figure 11. No combination of our metric scores shows a high correlation. This shows that each metric measures a quality independent of all other metrics.
We show examples for saliency of the TCN architecture on the FordB dataset for the visual interpretation methods for class 0 in Figure 12 and class 1 in Figure 13. We argue that the choice of a suitable method to generate saliency maps should be guided by our framework.   shows results for different model architectures, aggregated across datasets and visual interpretability methods. Model architecture has a comparatively small influence on the performance, which speaks to their generalization capabilities.  Figure 11: Pairplots of the correlations of metric scores with each other. No combination of metrics has a meaningful correlation with each other, proving that they provide independently useful signals. This analysis is based on scores that were normalized for dataset bias, but the results also hold when this normalization is not done.