Tractable Dendritic RNNs for Reconstructing Nonlinear Dynamical Systems
In many scientific disciplines, we are interested in inferring the nonlinear
dynamical system underlying a set of observed time series, a challenging task
in the face of chaotic behavior and noise. Previous deep learning approaches
toward this goal often suffered from a lack of interpretability and
tractability. In particular, the high-dimensional latent spaces often required
for a faithful embedding, even when the underlying dynamics lives on a
lower-dimensional manifold, can hamper theoretical analysis. Motivated by the
emerging principles of dendritic computation, we augment a dynamically
interpretable and mathematically tractable piecewise-linear (PL) recurrent
neural network (RNN) by a linear spline basis expansion. We show that this
approach retains all the theoretically appealing properties of the simple
PLRNN, yet boosts its capacity for approximating arbitrary nonlinear dynamical
systems in comparatively low dimensions. We employ two frameworks for training
the system, one combining back-propagation-through-time (BPTT) with teacher
forcing, and another based on fast and scalable variational inference. We show
that the dendritically expanded PLRNN achieves better reconstructions with
fewer parameters and dimensions on various dynamical systems benchmarks and
compares favorably to other methods, while retaining a tractable and
interpretable structure.