Fusing appearance and flow.
A popular extension of 2D CNNs to handle video is the Two-Stream architecture by Simonyan and Zisserman [1]}. Their method consists of two separated CNNs (streams) that are trained to extract features from a sampled RGB video frame paired with the surrounding stack of optical flow images, followed by a late fusion of the prediction scores of both streams. The image stream encodes the appearance information while the optical flow stream encodes the motion information, that are often found to complement each other for action recognition.
Several works followed this approach to find a suitable fusion of the streams at various depths [2]} and to explore the use of residual connections [3]} or multiplicative gating functions [4]} between them. These approaches rely on optical flow images for motion information, and a single RGB frame for appearance information, which is limiting when reasoning about the temporal context is required for video understanding.
