High-dimensional time series are common in many domains. Since human cognition is not optimized to work well in high-dimensional spaces, these areas could benefit from interpretable low-dimensional representations. However, most representation learning algorithms for time series data are difficult to interpret. This is due to non-intuitive mappings from data features to salient properties of the representation and non-smoothness over time.
 To address this problem, we propose a new representation learning framework building on ideas from interpretable discrete dimensionality reduction and deep generative modeling. This framework allows us to learn discrete representations of time series, which give rise to smooth and interpretable embeddings with superior clustering performance. We introduce a new way to overcome the non-differentiability in discrete representation learning and present a gradient-based version of the traditional self-organizing map algorithm that is more performant than the original. Furthermore, to allow for a probabilistic interpretation of our method, we integrate a Markov model in the representation space.
 This model uncovers the temporal transition structure, improves clustering performance even further and provides additional explanatory insights as well as a natural representation of uncertainty.
 We evaluate our model in terms of clustering performance and interpretability on static (Fashion-)MNIST data, a time series of linearly interpolated (Fashion-)MNIST images, a chaotic Lorenz attractor system with two macro states, as well as on a challenging real world medical time series application on the eICU data set. Our learned representations compare favorably with competitor methods and facilitate downstream tasks on the real world data. Interpretable representation learning on time series is a seminal problem for uncovering the latent structure in complex systems, such as chaotic dynamical systems or medical time series. In areas where humans have to make decisions based on large amounts of data, interpretability is fundamental to ease the human task. Especially when decisions have to be made in a timely manner and rely on observing some chaotic external process over time, such as in finance or medicine, the need for intuitive interpretations is even stronger. However, many unsupervised methods, such as clustering, make misleading i.i.d. assumptions about the data, neglecting their rich temporal structure and smooth behaviour over time. This poses the need for a method of clustering, where the clusters assume a topological structure in a lower dimensional space, such that the representations of the time series retain their smoothness in that space. In this work, we present a method with these properties.We choose to employ deep neural networks, because they have a very successful tradition in representation learning BID5 . In recent years, they have increasingly been combined with generative modeling through the advent of generative adversarial networks (GANs) BID13 and variational autoencoders (VAEs) BID18 . However, the representations learned by these models are often considered cryptic and do not offer the necessary interpretability . A lot of work has been done to improve them in this regard, in GANs as well as VAEs BID16 BID9 . Alas, these works have focused entirely on continuous representations, while discrete ones are still underexplored.In order to define temporal smoothness in a discrete representation space, the space has to be equipped with a topological neighborhood relationship. One type of representation space with such a structure is induced by the self-organizing map (SOM) BID21 . The SOM allows to map states from an uninterpretable continuous space to a lower-dimensional space with a predefined topologically interpretable structure, such as an easily visualizable two-dimensional grid. However, while yielding promising results in visualizing static state spaces, such as static patient states BID27 , the classical SOM formulation does not offer a notion of time. The time component can be incorporated using a probabilistic transition model, e.g. a Markov model, such that the representations of a single time point are enriched with information from the adjacent time points in the series. It is therefore potentially fruitful to apply the approaches of probabilistic modeling alongside representation learning and discrete dimensionality reduction in an end-to-end model.In this work, we propose a novel deep architecture that learns topologically interpretable discrete representations in a probabilistic fashion. Moreover, we introduce a new method to overcome the non-differentiability in discrete representation learning architectures and develop a gradient-based version of the classical selforganizing map algorithm with improved performance. We present extensive empirical evidence for the model's performance on synthetic and real world time series from benchmark data sets, a synthetic dynamical system with chaotic behavior and real world medical data. A schematic overview of our proposed model is depicted in FIG0 . An input x ∈ R d is mapped to a latent encoding z e ∈ R m (usually m < d) by computing z e = f θ (x), where f θ (·) is parameterized by the encoder neural network. The encoding is then assigned to an embedding z q ∈ R m in the dictionary of embeddings E = {e 1 , . . . , e k | e i ∈ R m } by sampling z q ∼ p(z q |z e ). The form of this distribution is flexible and can be a design choice. In order for the model to behave similarly to the original SOM algorithm (see below), in our experiments we choose the distribution to be categorical with probability mass 1 on the closest embedding to z e , i.e. p(z q |z e ) = 1[z q = arg min e∈E z e − e 2 ], where 1[·] is the indicator function. A reconstructionx of the input can then be computed asx = g φ (z), where g φ (·) is parameterized by the decoder neural network. Since the encodings and embeddings live in the same space, one can compute two different reconstructions, namelyx e = g φ (z e ) andx q = g φ (z q ).To achieve a topologically interpretable neighborhood structure, the embeddings are connected to form a self-organizing map. A self-organizing map consists of k nodes V = {v 1 , . . . , v k }, where every node corresponds to an embedding in the data space e v ∈ R d and a representation in a lower-dimensional discrete space m v ∈ M , where usually M ⊂ N 2 . During training on a data set D = {x 1 , . . . , x n }, a winner nodẽ v is chosen for every point x i according toṽ = arg min v∈V e v − x i 2 . The embedding vector for every [red] . In order to achieve a discrete representation, every latent data point (z e ) is mapped to its closest node in the SOM (z q ). A Markov transition model [blue] is learned to predict the next discrete representation (z t+1 q ) given the current one (z t q ). The discrete representations can then be decoded by another neural network back into the original data space. node u ∈ V is then updated according to e u ← e u + N (m u , mṽ)η(x i − e u ), where η is the learning rate and N (m u , mṽ) is a neighborhood function between the nodes defined on the representation space M . There can be different design choices for N (m u , mṽ). A more thorough review of the self-organizing map algorithm is deferred to the appendix (Sec. A).We choose to use a two-dimensional SOM because it facilitates visualization similar to BID27 . Since we want the architecture to be trainable end-to-end, we cannot use the standard SOM training algorithm described above. Instead, we devise a loss function term whose gradient corresponds to a weighted version of the original SOM update rule (see below). We implement it in such a way that any time an embedding e i,j at position (i, j) in the map gets updated, it also updates all the embeddings in its immediate neighborhood N (e i,j ). The neighborhood is defined as N (e i,j ) = {e i−1,j , e i+1,j , e i,j−1 , e i,j+1 } for a two-dimensional map.The loss function for a single input x looks like DISPLAYFORM0 where x, z e , z q ,x e andx q are defined as above and α and β are weighting hyperparameters.Every term in this function is specifically designed to optimize a different model component. The first term is the reconstruction loss L reconstruction (x,x q ,x e ) = x−x q 2 + x−x e 2 . The first subterm of this is the discrete reconstruction loss , which encourages the assigned SOM node z q (x) to be an informative representation of the input. The second subterm encourages the encoding z e (x) to also be an informative representation. This ensures that all parts of the model have a fully differentiable credit assignment path to the loss function, which facilitates training. Note that the reconstruction loss corresponds to the evidence lower bound (ELBO) of the VAE part of our model BID18 . Since we assume a uniform prior over z q , the KL-term in the ELBO is constant w.r.t. the parameters and can be ignored during optimization.The term L commitment encourages the encodings and assigned SOM nodes to be close to each other and is defined as DISPLAYFORM1 2 . Closeness of encodings and embeddings should be expected to already follow from the L reconstruction term in a fully differentiable architecture. However, due to the nondifferentiability of the embedding assignment in our model, the L commitment term has to be explicitly added to the objective in order for the encoder to get gradient information about z q . DISPLAYFORM2 2 , where N (·) is the set of neighbors in the discrete space as defined above and sg [·] is the gradient stopping operator that does not change the outputs during the forward pass, but sets the gradients to 0 during the backward pass. It encourages the neighbors of the assigned SOM node z q to also be close to z e , thus enabling the embeddings to exhibit a self-organizing map property, while stopping the gradients on z e such that the encoding is not pulled in the direction of the neighbors. This term enforces a neighborhood relation between the discrete codes and encourages all SOM nodes to ultimately receive gradient information from the data. The gradient stopping in this term is motivated by the observation that the data points themselves do not get moved in the direction of their assigned SOM node's neighbors in the original SOM algorithm either (see above). We want to optimize the embeddings based on their neighbors, but not the respective encodings, since any single encoding should be as close as possible to its assigned embedding and not receive gradient information from any other embeddings that it is not assigned to. Note that the gradient update of a specific SOM node in this formulation depends on its distance to the encoding, while the step size in the original SOM algorithm is constant. It will be seen that this offers some benefits in terms of optimization and convergence (see Sec. 4.1). The SOM-VAE can recover topologically interpretable state representations on time series and static data. It provides an improvement to standard methods in terms of clustering performance and offers a way to learn discrete two-dimensional representations of the data manifold in concurrence with the reconstruction task. It introduces a new way of overcoming the non-differentiability of the discrete representation assignment and contains a gradient-based variant of the traditional self-organizing map that is more performant than the original one. On a challenging real world medical data set, our model learns more informative representations with respect to medically relevant prediction targets than competitor methods. The learned representations can be visualized in an interpretable way and could be helpful for clinicians to understand patients' health states and trajectories more intuitively.It will be interesting to see in future work whether the probabilistic component can be extended to not just improve the clustering and interpretability of the whole model, but also enable us to make predictions. Promising avenues in that direction could be to increase the complexity by applying a higher order Markov Model, a Hidden Markov Model or a Gaussian Process. Another fruitful avenue of research could be to find more theoretically principled ways to overcome the non-differentiability and compare them with the empirically motivated ones. Lastly, one could explore deviating from the original SOM idea of fixing a latent space structure, such as a 2D grid, and learn the neighborhood structure as a graph directly from data.
We present a method to learn interpretable representations on time series using ideas from variational autoencoders, self-organizing maps and probabilistic models.
{"src": "scitldr-AIC_train", "paper_id": "rygjcsR9Y7"}