timeXplain -- A Framework for Explaining the Predictions of Time Series Classifiers

Modern time series classifiers display impressive predictive capabilities, yet their decision-making processes mostly remain black boxes to the user. At the same time, model-agnostic explainers, such as the recently proposed SHAP, promise to make the predictions of machine learning models interpretable, provided there are well-designed domain mappings. We bring both worlds together in our timeXplain framework, extending the reach of explainable artificial intelligence to time series classification and value prediction. We present novel domain mappings for the time and the frequency domain as well as series statistics and analyze their explicative power as well as their limits. We employ timeXplain in a large-scale experimental comparison of several state-of-the-art time series classifiers and discover similarities between seemingly distinct classification concepts such as residual neural networks and elastic ensembles.


Introduction
Artificial intelligence technology has become ubiquitous in industry and our everyday life in recent years. From dynamic pricing and recommender systems to autonomous vehicles and predictive policing, machine learning (ML) models continuously expand their influence on society. As they are entrusted with more and more responsibility, we should wonder whether we can actually trust them. ML systems must be safe, exhibit predictable and expected behavior, and must not discriminate [13]. Whether a system satisfies such fuzzy, yet indispensable criteria cannot be determined by computing a single "trustworthiness" metric. Instead, researchers are pushing towards technologies that explain the decisionmaking process of ML models. Through understanding the reasoning behind the predictions of complicated estimators, engineers are enabled to both spot flaws and build trust in their systems. The movement towards explainable artificial intelligence has been further fueled by a "right to explanation" recently proposed by the European Union [8]. The Communications of the ACM dedicated their  Fig. 1: Explanations of the predictions of two classifiers on time series #46 from the UCR GunPoint test set generated with timeXplain. The explanations show which temporal slices contribute towards (red) or against (blue) the correct classification as a "gun draw". Confirming previous literature, the shapelet transform (ST) classifier mostly focuses on the absence of a dip where the actor lowered their arm [12], while the dictionary-based WEASEL mainly focuses on the draw [17].
January 2020 issue to interpretable machine learning. In the cover article, Du, Liu, and Hu categorized the existing approaches by two major criteria: are the techniques intrinsic or post-hoc and are they global or local [5]. Intrinsic methods are specialized ML models that have interpretability built into them, see [2]. Post-hoc approaches treat the model as a black box. Also, the derived explanations can either be global, aiming at understanding the whole model, or they can be local, focusing on the prediction with respect to a specific input. All the categories have in common that one's ability to comprehend the decision-making process highly depends on the model's complexity. Understanding a linear regression might be feasible [15], but grasping the whole inner workings of a neural network with millions of weights is considered to be impossible today [13]. Time series are sequences of values whose fluctuations are associated with the passing of time. All sensor data is given this way and the pervasive use of monitoring devices in digital health [19], predictive maintenance [10], or cybersecurity [24] has led to an enormous increase of available time series data. Automated time series estimators (TSE) are applied to process this data. This subsumes all ML models that assign any kind of number to a time series and includes classification [3,7,21] as well as other value prediction tasks. With the rise of machine learning applications on time series, explanations of their behavior are much needed. We introduce timeXplain, a post-hoc local explanation framework for time series estimators.
To generate a local explanation, model-agnostic approaches often use perturbation in that they query the ML model in the vicinity of the input whose analysis they want to explain. We call this input the specimen. By slightly altering its features, one can gain insight into which of those features are relevant to the model, yielding an explanation as the result. To flesh out such an approach, one needs to define mapping functions detailing how to perturb the specimen. Ribeiro et al. pioneered work in this direction with LIME, introducing mappings for images, text, and tabular data [20]. Multiple extensions have subsequently been proposed, recently unified by Lundberg and Lee with their SHAP tool, in particular its model-agnostic explanation generator Kernel SHAP [16]. In the time series domain, this explanation approach has not realized its full potential yet. Guillemé et al. [9] gave some first mappings for time series and conducted a study suggesting that the derived explanations are understandable to human users. However, a collection of mappings covering all information available to time series estimators and a thorough discussion of the strengths and weaknesses of the applied methods are still mostly lacking. We introduce the timeXplain framework by defining mappings that build on the time domain, the frequency domain, and the statistics of a time series. Exemplary explanations generated with timeXplain are shown in Figure 1. They confirm previous literature on which fragments are relevant to shapelet transform (ST) [12] and WEASEL [17] classifiers. We further discuss the implicit assumptions made by our mappings and how they affect the generated explanations. Finally, we provide a reference implementation of our framework and employ it in a large-scale automated comparison of state-of-the-art time series classifiers. This comparison reveals surprising similarities between seemingly distinct classification concepts.
After reviewing the basics of SHAP in Section 2, the domain mappings of timeXplain are defined in Section 3. Section 4 shows how to make them robust in practice. Section 5 conducts an experimental comparison between several time series classifiers using timeXplain. We conclude the work in Section 6.

Post-hoc Local Explanations with SHAP
For SHAP [16], the black-box ML model to be interpreted is a function f : I → R from the input space I. For a classification task, we employ one such function f c for each class c, outputting the probability that the input belongs to that class. Intuitively, SHAP "disables" portions of the specimen x ∈ I, called fragments, yielding a perturbed specimen z ∈ I and then computes f (z). Doing this multiple times with different fragments explores the behavior of the prediction function f in the vicinity of the specimen. Using the gained insight, an interpretable linear model is built that approximates the original model near x. This is illustrated in Figure 2. The interpretable linear model then allows us to estimate the impact that each fragment of x has on the prediction f (x). One thus needs a way to divide x into some d fragments that can be disabled individually. This is formalized as the space of simplified inputs I = {0, 1} d . Each 1 or 0 in z ∈ I respectively stands for an activated or deactivated fragment of the specimen x. The actual deactivation of fragments is done by a mapping function h x : I → I; z → z. We require that evaluating h x on the all-ones vector 1 ∈ I recovers the unperturbed specimen h x (1) = x. Definition 1. An explanation model with respect to a mapping function h x is a linear regression model g : The set of all possible explanation models is denoted by G.
The impact vector φ acts as the post-hoc local explanation with respect to x. For any index k ∈ {1, . . . , d }, the so-called SHAP value φ k quantifies the impact of the k-th fragment on the prediction of the model f . To generate meaningful explanations, g should be faithful to the original model [20]. That is, it approximates f in the vicinity of x, observing g(z ) ≈ f (h x (z )) if z ∈ I disables only a few fragments. This is made formal by defining an inverse distance π 1 : I → R + 0 . Intuitively, the further away the simplified specimen z is from the all-ones vector 1, the smaller π 1 (z ) is. For an appropriate choice of π 1 , we refer the reader to [16]. The optimal explanation model is then Kernel SHAP learns the impact vector by training the linear regression model on randomly sampled simplified perturbed specimens z to approximate g * .

The timeXplain Mappings
While Kernel SHAP can generate model-agnostic explanations, it crucially depends on well-designed mappings for the specific application domain; a task that is left to the user. The fragments of the specimen x defined by h x need to be intuitively understandable for the impact values to be meaningful. Conversely, unsuitable mappings not adapted to the model or the data could produce misleading explanations, counteracting the whole idea of interpretability. This makes the construction of good mappings an important challenge.
We now describe the timeXplain framework and propose expressive mappings for time series data. It is well-known that time series can be seen not only as a sequence of values over time but also as a superposition of different frequencies, with the Fourier transform connecting these dual views [18]. Our mappings take both views as well as the statistics of the specimen into account.
Throughout this paper, we define the input space I = R d to be the set of all time series of length d. The value of a series z ∈ I at time t ∈ {1, . . . , d} is denoted by z t .

Time Slice Mapping
We first introduce the time slice mapping. The main idea is to partition the time series x into d connected subsequences, called slices, whose lengths differ by at To disable a slice of x, that is, a fragment of the specimen, one cannot just remove it or fill it with null values as most models can cope neither with series of varying lengths nor with null values. Instead, any slice is replaced with the corresponding slice of a so-called replacement time series r ∈ I. Six such replacement series will be presented below, each one giving rise to another variant of the time slice mapping. An exemplary perturbation (using the global mean replacement) is shown in Figure 3.

Definition 2.
Let r ∈ I be a replacement time series. The time slice mapping function h x : I → I using r for replacement yields the perturbed time series that is defined, for all t ∈ {1, . . . , d} and z ∈ I = {0, 1} d , as The time slice mapping makes the following two assumptions implicitly.
1. Feature space assumption: The ML model to be explained bases its decisions solely on the occurrence of patterns in the time domain. 2. Temporal coherence assumption: Neighboring points, that is, points in the same slice, have a similar impact on the predictions of the model.
Many models throughout the field of time series estimation seem to fulfill these assumptions. However, to the best of our knowledge, the effects of their violation on impact computation have not been discussed in the previous literature. We examine those effects in Section 4.2.
Synthetic Void Information. Finding a good replacement series r is nontrivial. We first propose to use synthetic void information, where replacing a slice of x with that of r removes all local structure so that the model cannot utilize any information of the "disabled" slice. Replacing a slice this way should also not insert any new structure so as not to distort the explanation. This requires r to carry as little information as possible. We propose five replacement options with void information. In view of their application in learning, they are defined with respect to a reference set S of time series, for example test data. A possible replacement is the zero series with r (1) t = 0 for all t. This, however, may lead to large jumps at the edges of the disabled slices if the neighboring values are very different from 0, which can be mitigated by setting r t to some average instead. We define the local mean by averaging over the t-th value in all series in S. Averaging over all values in all series gives the global mean.
Local mean: r For the next two replacement series, we employ Gaussian noise whose parameters are adapted to the series in S, again leading to a local and global variant.
The replacements are illustrated in Figure 4.
Authentic Opposing Information. Naturally, none of these five replacements fully succeed in removing all information in the disabled slices. Also, if the ML model has not seen synthetic void information in training, this approach might lead to distorted explanations. To tackle these issues, one can also use authentic opposing information as a sixth option for the replacement series r. Instead of avoiding any structure, we intentionally insert information different from that of the specimen. We accomplish this via sample replacement meaning that we draw the replacement series r (6) uniformly from the reference set S. This mapping has previously been suggested in [9]. The impact vector now shows how the model discriminates between x and the particular choice for r (6) , but this does not necessarily include all slices responsible for the model's prediction of x. We present a robust modification towards the latter in Section 4.1. Observe that authentic opposing information implicitly assumes reference set homogeneity, alleging that all series in S are somewhat similar, in order to ensure that multiple runs yield similar impacts. For a detailed discussion of this assumption see Section 4.2.

Frequency Band Mapping
Some time series estimators may not only work on the temporal structure of a time series but also examine its frequency spectrum. This might even be done by an explicit conversion of the series from time domain to the frequency domain via the Fourier transform. While the time slice mapping allows us to compute interpretable impacts of temporal slices of a time series, it fails to capture any frequency bands that may be considered impactful by some ML model. To explain the decisions of those models as well, we need to introduce a mapping that considers the frequency domain. For this frequency band mapping, we split the spectrum of the specimen into d frequency bands with quadratically increasing bandwidths. We use a quadratic scaling to provide greater resolution at the information-rich lower frequencies, while not loosing too much resolution at higher frequencies (as opposed to, say, a logarithmic scaling). Each band k ∈ {1, . . . , d } now corresponds to one fragment of the specimen. Its activation is governed by the entry z k of the simplified input. This mapping makes two assumptions, analogous to those for the time slice mapping. Firstly, the model solely considers frequency information in its decisionmaking (feature space assumption), and secondly, neighboring frequencies inside the same band have a similar impact on the predictions of the model (band coherence assumption). Both assumptions are discussed further in Section 4.2.
Synthetic Void Information. Proceeding like we did with the time slice mapping, we start with the synthetic void information approach. We propose the filter mapping that disables fragments by cutting the corresponding frequency bands from the spectrum of x using a so-called bandstop filter [18]. Any type of bandstop filter gives another variant of the filter mapping. For a fixed filter type, when the mapping function is queried with some simplified input z ∈ I , it cuts all the respective frequency bands k of the specimen x for which z k = 0. An illustration of the resulting perturbation is given in Figure 5.
The purpose of the filter is to disable a specified frequency band without introducing artifacts into the resulting spectrum (frequency domain) or the time series (time domain). We compare two types of filters from digital signal processing: elliptic and FIRLS filters. Their frequency responses are depicted in Figure 6. The elliptic filter [18] is an infinite impulse response (IIR) filter that is fast in execution and can be designed to feature a clear frequency response with sharp cutoffs. However, IIR filters are notorious for their instability [18]. The straightforward choice of an elliptic filter with the goal to disable the fragments might thus not be ideal as a general purpose solution for the design of a suitable frequency band mapping function. It may have its advantages, however, when the goal is to disable very narrow frequency bands to investigate their influence on the ML model. In general though, we recommend to employ a finite impulse response (FIR), in particular the FIRLS filters designed with least-squares optimization [18]. While a FIRLS filter may not demonstrate the same cutoff sharpness as the elliptic filter, its reliability makes it a robust choice for the automated generation of explanations for unknown specimens.
Authentic Opposing Information. As the ML model might be unable to handle zeroed-out frequency bands, we apply the approach of authentic opposing information in the frequency domain as well. We propose the patch mapping. It replaces a frequency band with that of a randomly picked patch time series r ∈ S. Once again, this approach assumes reference set homogeneity.
Let rDFT : I → C 1+ d /2 be the discrete Fourier transform with real input, see [23]. In the resulting series, the fluctuations of values are associated with the change of frequency. Coupled with the inverse transform (rDFT) -1 : C 1+ d /2 → I, we can manipulate time series in the frequency domain. Let the function κ assign to each frequency bin ω > 0 the corresponding band number κ(ω) = k.

Statistics Mapping
A time series estimator may base its decision on the mean and standard deviation of the input series. We employ the statistics mapping to detect this. It has always d = 2 fragments. If z 1 = 0, then the output series h x (z ) is a re-normalization of the specimen x such that its mean is now equal to some replacement mean; same for the standard deviation if z 2 = 0. The replacement values can be computed with respect to a reference set as shown in Section 3.1 (synthetic void information), or by using the mean and standard deviation of a randomly drawn time series (authentic opposing information). The transformation defined this way applies globally to the whole series. It thus only requires the feature space assumption, with the authentic opposing information approach additionally requiring reference set homogeneity in terms of mean and variance.

Robustness of timeXplain
Applying the above mappings without taking the application context into account may result in misleading interpretations. We highlight the most common pitfalls and how to avoid them when working with timeXplain, in particular regarding the assumptions that the various mappings require.

Multi-Run Averaging and Environment Classes
Most of the mappings above depend on a reference set. When randomly sampling from S for authentic opposing information, the resulting series may be uncharacteristic or, even worse, similar to x itself. The derived impacts are then not meaningful. We mitigate this issue by taking n distinct picks into account instead of relying on a single sample. Asking the model to discern x from several different sources of opposing information leads to the detection of more fragments that are relevant to the model's prediction in general. Still, sampling from or averaging over S introduces bias from patterns that occur frequently in S. This often happens in classification if there is a single class dominating S. At least for classification, we combat this problem by treating the classes separately. A classifier provides an individual probability function f c for each class. To derive the corresponding impact vector φ c , we average over intermediate impacts φ c,c with respect to a so-called environment class c .
To make these ideas precise, we need some additional notation. We model the process of computing the impact vectors of the explanation model g as an operator Φ(f, h x ) that takes the original ML model f and a mapping function h x as input. In turn, this mapping function is constructed from some reference set S or a (randomly drawn) time series r ∈ S, denoted by H x (S) or H x (r), respectively. For a classification task, let C denote the collection of all classes, and let S c contain the series in S with true class c ∈ C.
We benefit from the fact that most classifiers compute their class-specific models f c simultaneously. Thus, for some fixed environment class c , we can also compute the vectors φ c,c simultaneously for all c, speeding up the runtime.

Avoiding Artifacts from False Assumptions
The mappings all make some assumptions on the specimen or model. We explore what happens if they are violated in order to help the user accurately assess the merits of the explanations generated by timeXplain.
Feature Space Assumption. The feature space assumption states that the ML model bases its decisions on the properties of the specimen in the respective feature space. For us, that is the time domain, the frequency domain, or the statistics of a time series. What happens if one assumes the model to work in the time domain, and therefore applies the time slice mapping, while in actuality the model works in the frequency domain? When disabling a fragment in the time domain, for example removing some spike, the removal naturally affects the frequency domain as well, presumably by lowering the magnitude of certain frequencies. If these frequencies are impactful to the model, this affects the confidence of the model's prediction. However, timeXplain only witnesses the indirect effect of removing the spike and thus marks it to be impactful.
The end user who is inspecting the impact distribution needs to be aware that the marked fragments might not actually be the important features, but instead only artifacts. Conversely, the timeXplain user may need additional knowledge of the domain of the model to choose an appropriate mapping.
Coherence Assumption. Assuming coherence alleges that neighboring values have a similar impact on the model's prediction, both in the time and frequency domain. Intuitively, this is reasonable as a model that bases its decision on isolated points, ignoring their neighborhood, would be substantially overfitting.

Time Time
Specimen Slice edges Impactful slices Fig. 7: The effect of using too few time slices. The left plot assumes only shortreaching coherence, the two rising slopes have a high impact. The right plot assumes far-reaching coherence, resulting in a misleading impact distribution.
Contrarily, coherence spanning a longer and longer range of points or frequencies becomes increasingly implausible. Assuming such a far-reaching coherence may result in misleading explanations, if for example several meaningful peaks of a time series are assumed to be one single fragment as illustrated in Figure 7. Moreover, as the resolution of the mapping function implicitly defines how far-reaching the coherence is, the derived explanation cannot serve as aposteriori justification of the initial assumption. Those who view the resulting impact distribution have to take into account that the truly important portions of the specimen might only be small parts lying inside an impactful fragment. To mitigate these issues, we recommend to use a moderate number of time slices or frequency bands incorporating a small but non-empty neighborhood.
Simplified Input Dimension Trade-off. The choice of the dimension d of the simplified input space is also a source of artifacts in another regard. Above, we have argued that too small of a value results in low resolution and potentially misleading impacts. Intuitively, the higher we choose d , the more fine-grained the derived explanation becomes. Furthermore, the fidelity to the original model should increase. This intuition, however, is not entirely true. A high-dimensional simplified input space also makes the explanation susceptible to noise in practice.
Even for a moderately sized d , it is prohibitive to brute-force all 2 d perturbations of a specimen for a single explanation, let alone with multiple specimens and mapping functions. We thus resort to sampling. However, if one were to choose the dimension d too high and then sample only a few perturbations, the resulting explanation model g would be noisy and overfitted. Additionally, Alvarez-Melis and Jaakola observed fluctuations of SHAP caused by slight changes to the specimen [1]. They concluded that perturbation-based approaches are prone to unstable behavior. These results incentivize to choose d small enough such that a suitable number of samples for the linear regression can be collected.
In summary, the simplified input dimension being either too low or too high may both reduce the quality of the explanation. In this work, we choose d based on preliminary experiments. Finding general bounds on d is left as future work.
Reference Set Homogeneity. Experiments by Guillemé et al. [9] suggest that in the time slice mapping, sample replacements produce explanations at least as, but often more, faithful to the original ML model than constant replacements. This behavior likely extends to our proposed authentic opposing information approach in the frequency domain as well as to the statistics mapping. So, is there a justification to use synthetic void information instead?
The reference set S is likely heterogeneous to some degree, with the series exhibiting differences ranging from a time shift to completely distinct behavior. This may have an adversarial effect on the robustness of the impact computation over multiple tries, even when multi-run averaging is applied, see Section 4.1. When dealing with a very heterogeneous reference set or when otherwise observing unstable impacts, synthetic void information is a good fall-back approach. It applies the same replacement strategy over multiple tries and thus increases both robustness and reproducibility of the derived explanation.

Experimental Classifier Comparison
We employ timeXplain to compare a collection of state-of-the-art time series classifiers. We thereby show how local explanations can be applied to discover similarities in the decision-making processes of distinct classifiers.

Classifiers
We include one classifier from each paradigm outlined by Bagnall et al. [3].
1. Whole series techniques compare time series pairwise with a distance measure that operates on time series as a whole in the time domain. For our experiments, we choose the elastic ensemble EE of 1-nearest-neighbor classifiers using various distance measures, including dynamic time warping [3]. 2. Interval techniques analyze only training-selected, phase-independent intervals of a series, ignoring noisy regions and focusing on discriminatory ones. As a representative, we choose the time series forest classifier TSF [3]. 3. Shapelet techniques classify series based on the occurrence of class-specific, phase-independent subseries found during training, named shapelets. We choose shapelet transform ST due to its superior performance over other shapelet methods [3] and couple it with random forest for classification. 4. Dictionary-based techniques not only take into account the presence or absence of subseries, but also their repetition. Histograms are built from these frequency counts and then fed into a classifier. We choose the word extraction for time series classification WEASEL and combine it with a logistic regressor, as proposed by Schäfer and Leser [22].
Additionally, we include random interval spectral ensemble RISE, a classifier recently proposed by Lines et al. [11] that extracts features from the frequency domain and utilizes them to learn a time series forest. We also investigate whether the behavior of general purpose classifiers shares any resemblance with that of classifiers specific to time series. The rotation forest RotF is known to lead the field of general purpose classifiers applied to time series data [3]. Finally, we add a support vector machine with a linear kernel SVM/Lin and the residual network ResNet as the current best representative of deep learning on time series [7].

Experimental Setup
The classifiers were implemented in Python using the sktime [14], pyts [6], and tslearn [25] packages. The experiments were run on a server with four Intel Xeon Gold 5118 CPUs at 2.30 GHz, 48 physical cores in total, and 62 GB RAM. The data was taken from the public UCR time series classification archive [4]. We excluded datasets with missing values or varying time series lengths, as some models cannot cope with such series. Datasets with more than four classes were also excluded to keep the multi-run averaging costs low. From the remaining 67 datasets, 33 were picked at random. One instance of each classifier was trained on each dataset. Due to its high runtime, we limited the EE classifier to 10 randomly picked datasets. From each dataset's test set, five specimens were selected for the comparison. They were drawn at random in a way that specimen 1 is of class 1, specimen 2 of class 2, and so on, wrapping around once all classes were covered. We applied timeXplain to explain the behavior of the class-specific model for the true class of the specimen. For each specimen and each classifier, we computed two impact vectors, one with the time slice mapping with sample replacement and one with the frequency band patch mapping. The test sets served as reference sets. We employed multi-run averaging over n = 10 runs with environment classes. The simplified input dimension d was set to min{ d /5 , 30}, based on preliminary experiments. In each pass, the classifier was probed with perturbations based on 1000 random simplified inputs per specimen. Note that we chose authentic opposing information mappings since the impacts computed by such mappings are more faithful to the model given that the reference sets are homogeneous (see Section 4.2), which seems to be mostly true for the UCR test sets.  We used the following method to quantify the similarities between explanations from two different classifiers on the same specimen. Given impact vectors φ and ψ, we derived the set of correlation points defined as {(φ k , ψ k )} k∈{1,...,d } . The linear correlation of the points was evaluated via the Pearson correlation coefficient. This coefficient then served as the similarity score of the explanations. The process is illustrated in Figure 8. To compare the classifiers, we computed the pairwise correlations separately in the time domain and the frequency domain. For all 33 datasets and five specimens per dataset, the derived explanations were compared with the above method. The respective median correlations in the two domains are shown in Figure 9.

Results
We first focus on the median correlations obtained from the time slice mapping in the left part of Figure 9. Note that all medians are positive since negative correlations were outnumbered by positive ones. While most median correlations stay below 0.40, we can observe multiple distinctly pronounced median correlations between classifiers.
With a median correlation of 0.56, RISE is notably similar to TSF. This is quite surprising as TSF is a time-based interval technique while RISE operates exclusively in the frequency domain. TSF also shares a median correlation of 0.56 with EE. Even though EE works on whole series while TSF only looks at subseries, one can see why the two concepts may be related. In a similar way, the median correlation of 0.49 between EE and ST seems reasonable. More interesting is the median correlation of 0.5 between EE and ResNet, considering that the latter may learn a highly complex function, while the former only applies simple distance measures. The general purpose classifiers RotF and SVM/Lin do not share any notable median correlation with any other classifier, which seems reasonable as they interpret time series only as unstructured collections of points. WEASEL not sharing notable median correlations hints towards dictionary-based techniques being a truly unique classification concept.
The median correlations computed in the frequency domain are shown on the right side of Figure 9. These similarities are substantially higher overall compared to the time domain. We have conducted extensive manual spot checks to expound this observation. From examining several time series spectra and impacts, we conjecture that for the majority of UCR datasets, most of the information of a time series that is useful to classifiers is contained in its lower frequencies.
Note that all classifier pairs with pronounced median correlations in the time domain have above average median correlations in the frequency domain. This may be due to artifacts induced by violations of the feature space assumption, see Section 4.2. Furthermore, we cannot identify a noise floor in the frequency domain and, as such, cannot determine a group of pronounced median correlations. Still, the high median correlation of 0.82 respectively 0.73 between EE and RISE respectively RotF are surprising considering they do not appear in the time domain and neither EE nor RotF have explicit insight into the frequency domain.

Conclusion
The increasing adoption of machine learning systems in our everyday life raises the need to interpret them. We devised timeXplain, a model-agnostic local explanation framework for time series estimators. We introduced several domain mappings building on the SHAP framework. These mappings were embedded in a system of categories, depending on whether they use time slices, frequency bands, or statistics, and whether they employ synthetic void information or authentic opposing one. This categorization makes it easy to extend and adapt the framework. We also discussed common assumptions underlying the mappings and how they affect the robustness of timeXplain.
Finally, we presented a new use case for explainable artificial intelligence by employing timeXplain in an experimental comparison of the behavior of several time series classifiers. Our analysis found unexpected similarities between models which are very different in design and focus.
Future work should examine these similarities further and may also discover hidden parallels between machine learning systems in other areas of time series estimation. Environment classes may be refined, extending the concept to non-classifier models and analyzing the semantics of intermediate impacts, especially negative ones. Most importantly, to advance towards zero prior knowledge explanations, there is need for a technique that detects and eliminates artifact impacts induced by violations of the feature space assumption.