For this purpose, we have tried out several network architectures that can leverage both spatial as well as temporal components in the heatmap sequences. A well-known Deep Learning model which can predict spatio-temporal sequences is the Convolutional LSTM [1]}. However, in our experimental trials it could not produce accurate heatmaps. After extensive trials with a variety of Deep Neural Network based models that use both recurrent and convolutional architectures, we find that a simplified version of the CNN with Dilations model presented in [2]}, modified to work with multi-channel heatmaps performs the best. We use Causal Padding to ensure that the Network can only use past data to predict future values, and there is no leakage of data. Figure REF  shows the effect of using Dilations in a Convolutional layer.
<FIGURE>