Compared with the supervised methods, the task of behavior recognition with unsupervised approaches is much more challenging, and only a few attempts have been reported. Some methods focused on leveraging temporal information of videos to learn visual representation, such as Shuffle & Learn [1]}, OPN [2]}, and [3]}, [4]}, [5]}, [6]}. Other methods utilized the encoder-decoder-based video sequence/frame reconstruction, e.g., RGB-based [7]}, [8]}, [9]}, and skeleton-based LongT GAN [10]} and Predict & Cluster (P&C) [11]}. The problem is that P&C just used the reconstruction loss in an element-wise manner without considering any informative constraint or prior. In addition, P&C employed a fixed-length input scheme which is hard to encode the long-term motion dependencies since the down-sampled sequences may lose essential information.
