BERT architecture has inspired many new Transformer models for time-series data [1]}, [2]}, [3]}, [4]}. Creating a BERT style model for time-series data has some challenges compared to NLP tasks. Language data is a standardized type of data that can be used for various tasks, including translation, text summarization, question answering, sentiment analysis, etc. All of these tasks can use the same data for pre-training. However, this is not the case for the time-series tasks. Examples of time-series data include electricity usage [5]}, ambient temperature [6]}, traffic volume [5]}, satellite imagery [1]}, various forms of healthcare data [9]}, and more. With this variety of data types, the pre-training process will have to be different for each task. This task dependent pre-training contrasts with the NLP tasks which can start with the same pre-trained models assuming all tasks are based on the same language semantics and structure.
