Dynamical system models (including RNNs) often lack the ability to adapt the sequence generation or prediction to a given context, limiting their real-world application. In this paper we show that hierarchical multi-task dynamical systems (MTDSs) provide direct user control over sequence generation, via use of a latent  code z that specifies the customization to the
 individual data sequence. This enables style transfer, interpolation and morphing within generated sequences. We show the MTDS can improve predictions via latent code interpolation, and avoid the long-term performance degradation of standard RNN approaches. Time series data often arise as a related 'family' of sequences, where certain characteristic differences exist between the sequences in a dataset. Examples include the style of handwritten text (Graves, 2013) , the response of a patient to an anaesthetic (Bird et al., 2019) , or the style of locomotion in motion capture (mocap) data (Ghosh et al., 2017) . In this paper, we will consider how such variation may be modelled, and effectively controlled by an end user. Such related data is often pooled to train a single dynamical system, despite the internal variation. For a simple model, such as a linear dynamical system (LDS), this will result in learning only an average effect. In contrast, a recurrent neural network (RNN) may model this variation, but in an implicit and opaque manner. Such a 'black-box' approach prohibits end-user control, and may suffer from mode drift, such as in Ghosh et al. (2017) , where a generated mocap sequence performs an unprompted transition from walking to drinking. Some of these problems may be alleviated by appending 'context labels' to the inputs (see e.g. Goodfellow et al., 2016, §10.2.4) which describe the required customization. However, such labels are often unavailable, and the approach may fail to model the variation adequately even when they are. To move beyond these approaches, we consider latent variable models, where a latent variable z characterizes each sequence. This may be seen as a form of multi-task learning (MTL, see Zhang & Yang, 2017) , from which we derive the name multi-task dynamical system (MTDS), with each sequence treated as a task. A straightforward approach is to append the latent z to the inputs of the model, similarly to the 'context label' approach, thereby providing customization of the various bias (or offset) parameters of the model. A number of examples of this have been proposed recently, e.g. in Yingzhen & Mandt (2018) and Miladinović et al. (2019) . Nevertheless, this 'bias customization' has limited expressiveness and is often unsuitable for customizing simple models. In this paper we investigate a more powerful form of customization which modulates all the system and emission parameters. In this approach, the parameters of each task are constrained to lie on a learned low dimensional manifold, indexed by the latent z. Our experiments show that this approach results in improved performance and/or greater data efficiency than existing approaches, as well as greater robustness to unfamiliar test inputs. Further, varying z can generate a continuum of models, allowing interpolation between sequence predictions (see Figure 1b for an example), and potentially morphing of sequence characteristics over time. Contributions In this paper we propose the MTDS, which goes beyond existing work by allowing full adaptation of all parameters of general dynamical systems via use of a learned nonlinear manifold. We show how the approach may be applied to various popular models, and provide general purpose . . . . . . learning and inference algorithms. Our experimental studies use synthetic data (sum of two damped harmonic oscillators) and real-world human locomotion mocap data. We illuminate various properties of the MTDS formulation in our experiments, such as data efficiency, user control, and robustness to dataset shift, and show how these go beyond existing approaches to time series modelling. We finally utilize the increased user control in the context of mocap data to demonstrate style morphing. To this end, we introduce the model in Section 2, giving examples and discussing the particular challenges in learning and inference. We discuss the relation to existing work in Section 3. Experimental setup and results are given in Section 4 with a conclusion in Section 5. In this work we have shown how to extend dynamical systems with a general-purpose hierarchical structure for multi-task learning. Our MTDS framework performs customization at the level of all parameters, not just the biases, and adapts all parameters for general classes of dynamical systems. We have seen that the latent code can learn a fine-grained embedding of sequence variation and can be used to modulate predictions. Clearly good predictive performance for sequences requires task inference, whether implicit or explicit. There are three advantages of making this inference explicit. Firstly, it enhances control over predictions. This might be used by animators to control the style of predictions for mocap models, or to express domain knowledge, such as ensuring certain sequences evolve similarly. Secondly, it can improve generalization from small datasets since task interpolation is available out-of-the-box. Thirdly, it can be more robust against changes in distribution at test time than a pooled model: (2014) is a unit Gaussian p(z) = N (0, I). This choice allows simple sampling schemes, and straight-forward posterior approximations. It is also a useful choice for interpolation, since it allows continuous deformation of its outputs. An alternative choice might be a uniform distribution over a compact set, however posterior approximation is more challenging, see Svénsen (1998) for one approach. Sensible default choices for h φ include affine operators and multilayer perceptrons (MLPs). However, when the parameter space R d is large, it may be infeasible to predict d outputs from an MLP. Consider an RNN with 100k parameters. If an MLP has m L−1 = 300 units in the final hidden layer, the expansion to the RNN parameters in the final layer will require 30×10 6 parameters alone. A practical approach is to use a low rank matrix for this transformation, equivalent to adding an extra linear layer of size m L where we must have m L m L−1 to reduce the parameterization sufficiently. Since we will typically need m L to be O(10), we are restricting the parameter manifold of θ to lie in a low dimensional subspace. Since MLP approaches with a large base model will then usually have a restricted final layer, are there any advantages over a simple linear-Gaussian model for the prior p(z) and h φ ? There may indeed be many situations where this simpler model is reasonable. However, we note some advantages of the MLP approach: 1. The MLP parameterization can shift the density in parameter space to more appropriate regions via nonlinear transformation. 2. A linear space of recurrent model parameters can yield highly non-linear changes even to simple dynamical systems (see e.g. the bifurcations in §8 of Strogatz, 2018). We speculate it might be advantageous to curve the manifold to avoid such phenomena. 3. More expressive choices may help utilization of the latent space (e.g. Chen et al., 2017) . This may in fact motivate moving beyond a simple MLP for the h φ .
Tailoring predictions from sequence models (such as LDSs and RNNs) via an explicit latent code.
{"src": "scitldr-AIC_train", "paper_id": "Bkln2a4tPB"}