Many time series are generated by a set of entities that interact with one another over time. This paper introduces a broad, flexible framework to learn from multiple inter-dependent time series generated by such entities. Our framework explicitly models the entities and their interactions through time. It achieves this by building on the capabilities of Recurrent Neural Networks, while also offering several ways to incorporate domain knowledge constraints into the model architecture. The capabilities of our approach are showcased through an application to weather prediction, which shows gains over strong baselines.
A similar model has been proposed for learning on spatio-temporal graphs ( @cite_11 ). Unlike our models however, the RNNs in their model cannot affect each other. In their setting, each node and each edge has an input feature vector at every time step. Since the RNNs on the edges do not take input from RNNs at nodes, the forward pass through their model is equivalent to running independent siloed RNNs, with one set of RNNs for each node. In contrast, we do not have any data input on edges. Additionally, in our case, summaries flow along each edge which are computed using hidden states of RNNs nearby and thus, all RNNs evolve together. Another assumption in their approach is that the same RNN on the edge provides input to RNNs on both sides. In our case, the edges can be directed and hence the interactions can be modeled differently. Finally we note that this model is a special case of our model with each edge replaced by a node and creating a disconnected graph with one connected component per node in the original graph.
@cite_11
2176353499
Deep Recurrent Neural Network architectures, though remarkably capable at modeling sequences, lack an intuitive high-level spatio-temporal structure. That is while many problems in computer vision inherently have an underlying high-level structure and can benefit from it. Spatio-temporal graphs are a popular tool for imposing such high-level intuitions in the formulation of real world problems. In this paper, we propose an approach for combining the power of high-level spatio-temporal graphs and sequence learning success of Recurrent Neural Networks (RNNs). We develop a scalable method for casting an arbitrary spatio-temporal graph as a rich RNN mixture that is feedforward, fully differentiable, and jointly trainable. The proposed method is generic and principled as it can be used for transforming any spatio-temporal graph through employing a certain set of well defined steps. The evaluations of the proposed approach on a diverse set of problems, ranging from modeling human motion to object interactions, shows improvement over the state-of-the-art with a large margin. We expect this method to empower new approaches to problem formulation through high-level spatio-temporal graphs and Recurrent Neural Networks.