The temporal component of videos provides an important clue for activity recognition, as a number of activities can be reliably recognized based on the motion information. In view of that, this work proposes a novel temporal stream for two-stream convolutional networks based on images computed from the optical flow magnitude and orientation, named Magnitude-Orientation Stream (MOS), to learn the motion in a better and richer manner. Our method applies simple nonlinear transformations on the vertical and horizontal components of the optical flow to generate input images for the temporal stream. Experimental results, carried on two well-known datasets (HMDB51 and UCF101), demonstrate that using our proposed temporal stream as input to existing neural network architectures can improve their performance for activity recognition. Results demonstrate that our temporal stream provides complementary information able to improve the classical two-stream methods, indicating the suitability of our approach to be used as a temporal video representation.
As it can be inferred from the reviewed methods, most of them use either convolution operations over raw pixels or optical flow to model temporal information. The former do not decouple spatial and temporal information, letting appearance information prevail @cite_15 , while the latter approaches rely on horizontal and vertical components of the optical flow. Despite the optical flow-based methods produce promising results, they focus only on displacement information. In view of that, aiming at capturing more information from the optical flow, our method captures not only the displacement, by using orientation, but also captures the magnitude providing information regarding the velocity of the movement.
@cite_15
2165406874
We introduce a new technique for estimating the optical flow field, starting from image sequences. As suggested by Fleet and Jepson (1990), we track contours of constant phase over time, since these are more robust to variations in lighting conditions and deviations from pure translation than contours of constant amplitude. Our phase-based approach proceeds in three stages. First, the image sequence is spatially filtered using a bank of quadrature pairs of Gabor filters, and the temporal phase gradient is computed, yielding estimates of the velocity component in directions orthogonal to the filter pairs' orientations. Second, a component velocity is rejected if the corresponding filter pair's phase information is not linear over a given time span. Third, the remaining component velocities at a single spatial location are combined and a recurrent neural network is used to derive the full velocity. We test our approach on several image sequences, both synthetic and realistic.