With a typical unsupervised event retrieval task, the goal is the retrieval of all related videos semantically relevant to the query video sample. A major challenge is the construction of both compact and discriminative video representations. Conventional methods [1]}, [2]}, [3]} rely on frame-level local features (e.g., SIFT [4]}) and aggregation strategies (e.g., Fisher Vector [5]}, [6]}, VLAD [1]}, [8]}, [9]} or explicit feature maps [2]}) for frame-level feature description.  Recent works [11]}, [12]} predominantly employ deep Convolutional Neural Network (CNNs) [13]}, [14]} to extract a feature descriptor from each video frame. Latest work [15]} revisits temporal match kernels [2]} and presents a learnable temporal layer which can further enhance the CNNs based feature descriptor. Subsequently, a video-level representation is typically obtained by directly averaging all frame-level descriptors in the video. Such sum-aggregation strategy disregards the strong temporal correlations among consecutive frames, which may over-emphasize certain long or recurrent shots in the video. We discuss this problem in Section  and show that the redundant information among frames can be effectively suppressed by the feature alignment step.
