The pre-training model is used to learn a good representation from massive unlabeled data and then use these representations for other downstream tasks.
Recent studies have demonstrated significant performance gains on many natural language processing tasks with the help of the representation extracted from pre-training models [1]}.
Prominent examples are the BERT [2]} and GPT [3]}, which are based on the Transformer encoder and decoder, respectively.
The Transformer architecture is more powerful and more efficient than LSTM architecture [4]}, [5]} and has become the mainstream approach for designing pre-training models.
More recently, Transformer for images has attracted increasing attention because of its powerful performance.
ViT [6]} proposes to split an image into patches and provide the sequence of linear embeddings of these patches as an input to a Transformer, showing impressive performance.
However, ViT needs supervised training, which requires massive labeled data.
On the contrary, MAE [7]} uses self-supervised learning based on the masked autoencoding strategy.
MAE enables us to train large models efficiently and effectively and outperforms supervised pre-training.
Although the pre-training model has made significant progress in natural language processing and computer vision, progress in time series lags behind them. In this paper, we propose a pre-training model (named TSFormer) for time series based on Transformer blocks and improve the performance of the downstream forecasting task.
