In this paper we developed a hierarchical network model, called Hierarchical Prediction Network (HPNet) to understand how spatiotemporal memories might be learned and encoded in a representational hierarchy for predicting future video frames. The model is inspired by the feedforward, feedback and lateral recurrent circuits in the mammalian hierarchical visual system. It assumes that spatiotemporal memories are encoded in the recurrent connections within each level and between different levels of the hierarchy. The model contains a feed-forward path that  computes and encodes spatiotemporal features of successive complexity and a feedback path that projects interpretation from a higher level to the level below. Within each level, the feed-forward path and the feedback path intersect in a recurrent gated circuit that integrates their signals as well as the circuit's internal memory states to generate  a prediction of the incoming signals. The network learns by comparing the incoming signals with its prediction, updating its internal model of the world by minimizing the prediction errors at each level of the hierarchy in the style of {\em predictive self-supervised learning}. The network processes data in blocks of video  frames rather than  a frame-to-frame basis.   This allows it to learn relationships among movement patterns, yielding state-of-the-art performance in long range video sequence predictions in benchmark datasets. We observed that hierarchical interaction in the network introduces sensitivity to memories of global movement patterns even in the population representation of the units in the earliest level. Finally, we provided neurophysiological evidence, showing that neurons in the early visual cortex of awake monkeys exhibit very similar sensitivity and behaviors. These findings suggest that  predictive self-supervised learning might be an important principle for representational learning in the visual cortex.   While the hippocampus is known to play a critical role in encoding episodic memories, the storage of these memories might ultimately rest in the sensory areas of the neocortex BID27 . Indeed, a number of neurophysiological studies suggest that neurons throughout the hierarchical visual cortex, including those in the early visual areas such as V1 and V2, might be encoding memories of object images and of visual sequences in cell assemblies BID54 BID14 BID52 BID2 BID21 . As specific priors, these memories, together with the generic statistical priors encoded in receptive fields and connectivity of neurons, serve as internal models of the world for predicting incoming visual experiences. In fact, learning to predict incoming visual signals has also been proposed as an objective that drives representation learning in a recurrent neural network in a self-supervised learning paradigm, where the discrepancy between the model's prediction and the incoming signals can be used to train the network using backpropagation, without the need of labeled data BID9 BID46 BID42 BID34 BID21 .In computer vision, a number of hierarchical recurrent neural network models, notably PredNet BID24 and PredRNN++ , have been developed for video prediction with state-of-the-art performance. PredNet , in particular, was inspired by the neuroscience principle of predictive coding BID31 BID39 BID21 BID7 BID11 . It learns a LSTM (long short-term memory) model at each level to predict the prediction errors made in an earlier level of the hierarchical visual system. Because the error representations are sparse, the computation of PredNet is very efficient. However, the model builds a hierarchical representation to model and predict its own errors, rather than learning a hierarchy of features of successive complexities and scales to model the world. The lack of a compositional feature hierarchy hampers its ability in long range video predictions.Here, we proposed an alternative hierarchical network architecture. The proposed model, HPNet (Hierarchical Prediction Network), contains a fast feedforward path, instantiated currently by a fast deep convolutional neural network (DCNN) that learns a representational hierarchy of features of successive complexity, and a feedback path that brings a higher order interpretation to influence the computation a level below. The two paths intersect at each level through a gated recurrent circuit to generate a hypothetical interpretation of the current state of the world and make a prediction to explain the bottom-up input. The gated recurrent circuit, currently implemented in the form of LSTM, performs this prediction by integrating top-down, bottom-up, and horizontal information. The discrepancy between this prediction and the bottom-up input at each level is called prediction error, which is fed back to influence the interpretation of the gated recurrent circuits at the same level as well as the level above.To facilitate the learning of relationships between movement patterns, HPNet processes data in the unit of a spatiotemporal block that is composed of a sequence of video frames, rather than frame by frame, as in PredNet and PredRNN++. We used a 3D convolutional LSTM at each level of the hierarchy to process these spatiotemporal blocks of signals BID1 , which is a key factor underlying HPNet's better performance in long range video prediction.In the paper, we will first demonstrate HPNet's effectiveness in predictive learning and its competency in long range video prediction. Then we will provide neurophysiological evidence showing that neurons in the early visual cortex of the primate visual system exhibit the same sensitivity to memories of global movement patterns as units in the lowest modules of HPNet. Our results suggest that predictive self-supervised learning might indeed be an important strategy for representation learning in the visual cortex, and that HPNet is a viable computational model for understanding the computation in the visual cortical circuits. In this paper, we developed a hierarchical prediction network model (HPNet), with a fast DCNN feedforward path, a feedback path and local recurrent LSTM circuits for modeling the counterstream / analysis-by-synthesis architecture of the mammalian hierarchical visual systems. HPNet utilizes predictive self-supervised learning as in PredNet and PredRNN++, but integrates additional neural constraints or theoretical neuroscience ideas on spatiotemporal processing, counter-stream architecture, feature hierarchy, prediction evaluation and sparse convolution into a new model that delivers the state-of-the-art performance in long range video prediction. Most importantly, we found that the hierarchical interaction in HPNet introduces sensitivity to global movement patterns in the representational units of the earliest module in the network and that real cortical neurons in the early visual cortex of awake monkeys exhibit very similar sensitivity to memories of global movement patterns, despite their very local receptive fields. These findings support predictive self-supervised learning as an important principle for representation learning in the visual cortex and suggest that HPNet might be a viable computational model for understanding the cortical circuits in the hierarchical visual system at the functional level. Further evaluations are needed to determine definitively whether PredNet or HPNet is a better fit to the biological reality. APPENDIX A 3D CONVOLUTIONAL LSTM Because our data are in the unit of spatitemporal block, we have to use a 3D form of the 2D convolutional LSTM. 3D convolutional LSTM has been used by BID1 in the stereo setting. The dimensions of the input video or the various representations (I, E and H) in any module are c×d×h×w, where c is the number of channels, d is the number of adjacent frames, h and w specify the spatial dimensions of the frame. The 3D spatiotemporal convolution kernel is m × k × k in size, where m is kernel temporal depth and k is kernel spatial size. The spatial stride of the convolution is 1. The size of the output with n kernels is n × d × h × w. We define the inputs as X 1 , ..., X t , the cell states as C 1 , ..., C t , the outputs as H 1 , ..., H t , and the gates as i t , f t , o t . Our 3D convolutional LSTM is specified by the equations below, where the function of 3D convolution is indicated by and the Hadamard product is indicated by •. FIG7 of the main text of the paper. The figures demonstrate that as more higher order modules are stacked up in the hierarchy, the semantic clustering into the six movement classes become more pronounced even in the early modules, suggesting that the hierarchical interaction has steered the feature representation into semantic clusters even in the early modules. Module 4-1 means representation of module 1 in a 4-module network. DISPLAYFORM0 We use linear decoding (multi-class SVM) to assess the distinctiveness of the semantiuc clusters in the representation of the different modules in the different networks. The decoding results in TAB2 shows that the decoding accuracy based on the reprsentation of module 1 has improved from chance (16%) to 26%, an improvement of 60% between a 1-module HPNet and a 4-module HPNet, and that the representation of module 4 of a 4-module HPNet can achieve a 63% accuracy in classifying the six movement classes, suggesting that the network only needs to learn to predict unlabelled video sequences, and it automatically learns reasonable semantic representations for recognition. For comparison, we also performed decoding on the output representations of each LSTM layer in the PredRNN++ and PredNet to study their representations of the six movement patterns. The results shown below indicate that the semantic clustering of the six movements is not very strong in the PredRNN++ hierarchy. We realized that this might be because the PredRNN++ behaves essentially like an autoencoder. The four-layer network effectively only has two layers of feature abstraction, with layer 2 being the most semantic in the hierarchy and layers 3 and 4 representing the unfolding of the feedback path. Decoding results indicate that the hierarchical representation based on the output of the LSTM at every layer in PredNet, which serve to predict errors of prediction errors of the previous layer, does not contain semantic information about the global movement patterns. Figure 8 : Results of video sequence learning experiments showing prediction suppression can be observed in E, P , and R units in every module along the hierarchical network. The abscissa is time after stimulus onset -where we set each video frame to be 25 ms for comparison with neural data. The ordinate is the normalized averaged temporal response of all the units within the center 8×8 hypercolumns, averaged across all neurons and across the 20 movies in the Predicted set (blue) and the Unpredicted set (red) respectively. Prediction suppression can be observed in all types of units, though more pronounced in the E and P units.
A new hierarchical cortical model for encoding spatiotemporal memory and video prediction
{"src": "scitldr-AIC_train", "paper_id": "BJl_VnR9Km"}