As an intuitive substitution of a fully supervised method, the unsupervised model is advantageous since it does not require human-labeled data. In this paper, following the Seq2Seq unsupervised learning routine [1]}, [2]}, we introduce an encoder-decoder model to learn discriminative information of MG (pose or key-points-based) sequences without using labelled data. There are a few key differences between our method and previous unsupervised models: 1) We introduce the mutual information to control characteristics of the representation by matching to a prior distribution adversarially, namely, the Kullback-Leibler (KL) divergence is utilized to act as a measure of non-linear statistical dependence between input sequence and reconstructed one which facilitates the model to learn inherent action/gesture representations. While most of existing methods only rely on traditional element-wise loss, e.g., mean square error (\(L_2\)  distance) [3]} [2]} and mean absolute error (\(L_1\) ) [2]}. 2) Unlike other Seq2Seq type encoder-decoder with a fixed-length scheme [1]} [2]} that only read in parts of the input sequence, we provide a flexible strategy which enables the encoder to read in the whole input sequence, aiming to utilize the complete context information and capture the long-term dynamics in sequences with arbitrary lengthes. To realize the above functions, we extend the preliminary model of sequential variational autoencoder (S-VAE) [8]}, which is a variant of VAE whose encoder-decoder are implemented by bidirectional LSTM (BLSTM). Different to the vanilla VAE, S-VAE can handle sequential data and capture latent patterns from the whole input sequence.
