Learning multimodal representation from unlabeled videos has received considerable attention [1]}. Audio-visual learning is of particular interest due to the abundance of videos with natural audio-visual co-occurrence [2]}, [3]}, [4]}, [5]}, [6]}, [7]}. However, existing approaches learn localized representations from short videos (hundreds of milliseconds to just under a few seconds), capturing only short-term dependencies in data. While this is useful for certain applications, e.g., source separation [5]} and atomic action recognition [9]}, learning representation that captures long-term dependencies is equally important, e.g., for activity recognition [10]}, [11]}, [12]}. Unfortunately, processing long videos requires large memory resource and capturing long-term dependencies is a long-standing problem [13]}, [14]}, [15]}.
