Positional embeddings.
Another important difference between TSFormer and MAE [1]} and the original Transformer [2]} is the learnable positional embedding.
Therefore, we would like to explore whether TSFormer has learned reasonable positional embeddings.
We compute the cosine similarity between the positional embeddings of 336 patches and get a \(336\times 336\)  heat map, shown in Figure REF (d).
We find that the positional embedding of TSFormer better reflects the multi-periodicity in time series.
This is because, unlike the representation of the encoder, which needs to depend on the input patches, the positional embeddings are completely free to optimize and are less affected by the noise of the input data.
We conjecture that such positional embedding is the key factor for the success of TSFormer since we found that TSFormer could not get meaningful representations if we replace the learnable positional embeddings with the deterministic, sinusoidal ones.
