Much of the mainstream Transformer applications rely on the training data, which is hand labeled. Labeling data can take a considerable time, rendering the massive amount of unlabeled data unusable. Self-supervised and unsupervised methods seek to improve Transformers by giving them the ability to learn, categorize, and forecast for data that has not been labeled. For example, there is a considerable amount of unlabeled satellite imaging data. Authors in [1]}, proposed a model, SITS-BERT, that learns from unlabeled data to apply region classification from satellite imagery. Other self-supervised or unsupervised learning models include anomaly Transformer [2]} and self-supervised Transformer for Time-Series (STraTS) [3]}.
