The Vanilla Transformer is the same architecture as our proposed Synthesizer Transformer, but uses the original attention mechanism as per Subsection REF . Secondly, long-short term memory models (LSTMs) [1]} are a type of recurrent neural network that are known for their ability to capture long-term dependencies in time series data as well as avoid the vanishing gradient issue [2]}. The full configuration of the networks used as baseline is described in Subsection REF . Finally, we also explore a statistical model often used in time series analysis: Generalized AutoRegressive Conditional Heteroskedasticity, or GARCH [3]}. This model extends the Autoregressive Conditional Heteroskedastic Models (ARCH) model, by including a moving average component (ma) joint with the autoregressive component. This model is often used for volatility prediction, even for Bitcoin  [4]}. As our baseline model, we use GARCH(1,1), which is the first order GARCH model using the ARCH library in Python https://github.com/bashtage/arch.
