Second, we propose a Synthesizer Transformer model to perform the volatility spike prediction. The Transformer architecture has proven to be extremely efficient for a range of tasks related to time series such as text translation [1]}, music generation [2]}, emotion prediction from movies [3]}, and speech synthesis [4]}. In finance, it has been shown to be efficient at stock price [5]}, [6]} and even stock volatility prediction [7]}. In the cryptocurrency markets, we see that it has been used for Dogecoin [8]} and Bitcoin [9]} price prediction. In this work, we expand the existing literature by including CryptoQuant and whale data (plus technical indicators calculated on this data). We then go beyond just building a black-box model, but also explore the influence of these features on volatility prediction through explainable artificial intelligence (XAI) techniques with the Captum library [10]}. Instead of using Vanilla (standard) Transformer architectures, we change the typical dot product self-attention mechanism to Synthesizer attention, which learns synthetic attention weights without token-to-token interactions. By doing so, we optimize the attention span of the model. Recent work has shown that Synthesizer Transformers outperform traditional Transformers. Even a simple Random Synthesizer has shown to be 60% faster than a traditional Transformer [11]}.
In an experiment, we compare our proposed architecture to other configurations and baseline traditional models like GARCH. We show that it is a useful and reliable method for forecasting volatility in cryptocurrencies.
