Cross Reconstruction Transformer for Self-Supervised Time Series Representation Learning
Unsupervised/self-supervised representation learning in time series is
critical since labeled samples are usually scarce in real-world scenarios.
Existing approaches mainly leverage the contrastive learning framework, which
automatically learns to understand the similar and dissimilar data pairs.
Nevertheless, they are restricted to the prior knowledge of constructing pairs,
cumbersome sampling policy, and unstable performances when encountering
sampling bias. Also, few works have focused on effectively modeling across
temporal-spectral relations to extend the capacity of representations. In this
paper, we aim at learning representations for time series from a new
perspective and propose Cross Reconstruction Transformer (CRT) to solve the
aforementioned problems in a unified way. CRT achieves time series
representation learning through a cross-domain dropping-reconstruction task.
Specifically, we transform time series into the frequency domain and randomly
drop certain parts in both time and frequency domains. Dropping can maximally
preserve the global context compared to cropping and masking. Then a
transformer architecture is utilized to adequately capture the cross-domain
correlations between temporal and spectral information through reconstructing
data in both domains, which is called Dropped Temporal-Spectral Modeling. To
discriminate the representations in global latent space, we propose Instance
Discrimination Constraint to reduce the mutual information between different
time series and sharpen the decision boundaries. Additionally, we propose a
specified curriculum learning strategy to optimize the CRT, which progressively
increases the dropping ratio in the training process.