(i) Time series information density is lower.
As human-generated signals, each data point in natural language (i.e., a word in a sentence) has rich semantics and is suitable as the data unit for model input.
On the contrary, isolated
data points in time series give less semantic information.
Semantics only arise when we observe at least segment-level data, such as going up or down.
On the other hand, language models are usually trained by predicting only a few missing words per sentence.
However, masked values in time series can often be trivially predicted by simple interpolation [1]}, making the pre-training model only focuses on low-level information.
To address this problem, a simple strategy that works well is to mask a very high portion of the model's input to encourage the learning of high-level semantics, motivated by recent development in computer vision [2]}.
It creates a challenging self-supervised task that forces the model to obtain holistic understanding of time series.
