where \(\mathbf {W}\in \mathbb {R}^{d\times (LC)}\)  and \(\mathbf {b}\in \mathbb {R}^d\)  are learnable parameters, \(\mathbf {U}^i_j\in \mathbb {R}^{d}\)  are the model input vectors, and \(d\)  is the hidden dimension.
For masked patches, we use a shared learnable mask token to indicate the presence of a missing patch to be predicted.
Next, the positional encoding layer is used to add sequential information.
Notably, the positional encoding operates on all patches, although the mask tokens are not used in the encoder.
Moreover, unlike the deterministic, sinusoidal embeddings used in MAE [1]}, we use learnable positional embeddings.
On the one hand, in this work, learnable embeddings significantly outperform sinusoidal ones for all datasets.
On the other hand, we observe that learned positional embeddings are crucial in learning time series' periodic features, which will be demonstrated in Section REF .
Finally, we obtain the latent representations \(\mathbf {H}_j^i\in \mathbf {R}^d\)  through Transformer blocks for all unmasked patches \(j\) .
