<jupyter_start><jupyter_text>created by <NAME> - email: <EMAIL>  TSBERT: Self-Supervised Pretraining of Time Series Models 🤗 This is an unofficial PyTorch implementation created by <NAME> (<EMAIL>) based on:* <NAME>., <NAME>., <NAME>., <NAME>., & <NAME>. (2020). A Transformer-based Framework for Multivariate Time Series Representation Learning. arXiv preprint arXiv:2010.02803v2.. No official implementation available as far as I know (Oct 10th, 2020) `TSBERT` is a self-supervised training method that can be used to pretrain time series models without using any labels. The approach is very similar to BERT.`TSBERT` is performed in 2 steps: 1. Pretrain the selected architecture without any labels. When training is finished, the pretrained model will be automatically saved to the given target_dir/fname.2. Fine-tune or train the same architecture with pretrained=True indicating the weights_path (target_dir/fname).In this notebook we'll use a UCR dataset (LSST) that contains around 2500 training and 2500 validation samples. To analyze the impact of `TSBERT` we'll:1. use supervised learning to set a baseline using 10% or 100% of the labels.2. pretrain a model using 100% of the training dataset without labels.3. fine tune or train using 10% or 100% of the training dataset (with labels). A key difference between `TSBERT` and the original paper is that you can use any architecture of your choice as long as it has a "head" attribute and can take a custom_head kwarg. Architectures finished in Plus in the `tsai` library meet this criteria. To demonstrate how this works, we'll use InceptionTimePlus throughout this notebook.  Results  These results indicate the following: * Pretraining + fine-tuning/ training improves performance when compared to supervised learning (training from scratch).* In this case, there's not much difference between fine-tuning and training a pretrained model. This may be dataset dependent. It'd be good to try both approaches. * The fewer labels available, the better pretraining seems to work.   Import libraries 📚<jupyter_code>stable = True # True: latest version, False: stable version

import sys
ISCOLAB = 'google.colab' in sys.modules
if ISCOLAB:
    if stable: 
        !pip install tsai -q
    else:
        !pip install -Uqq git+https://github.com/timeseriesAI/tsai.git@master
    
import tsai
from tsai.all import *
print('tsai       :', tsai.__version__)
print('fastai     :', fastai.__version__)
print('fastcore   :', fastcore.__version__)
print('torch      :', torch.__version__)<jupyter_output>[K     |████████████████████████████████| 143kB 13.7MB/s 
[K     |████████████████████████████████| 3.2MB 23.0MB/s 
[K     |████████████████████████████████| 5.7MB 22.5MB/s 
[K     |████████████████████████████████| 194kB 58.4MB/s 
[K     |████████████████████████████████| 9.5MB 59.1MB/s 
[K     |████████████████████████████████| 92kB 10.9MB/s 
[K     |████████████████████████████████| 2.5MB 49.7MB/s 
[K     |████████████████████████████████| 296kB 54.4MB/s 
[K     |████████████████████████████████| 22.2MB 1.7MB/s 
[K     |████████████████████████████████| 174kB 64.6MB/s 
[K     |████████████████████████████████| 25.3MB 1.3MB/s 
[K     |████████████████████████████████| 61kB 8.3MB/s 
[K     |████████████████████████████████| 675kB 57.5MB/s 
[K     |████████████████████████████████| 102kB 9.0MB/s 
[K     |████████████████████████████████| 102kB 12.2MB/s 
[?25h  Building wheel for contextvars (setup.py) ... [?25l[?25hdone
[31mERROR: distributed 2021.1.1 has requirement[...]<jupyter_text>Prepare data 🏭 We'll first import the data.<jupyter_code>dsid = 'LSST'
X, y, splits = get_UCR_data(dsid, split_data=False)<jupyter_output><empty_output><jupyter_text>We'll now create 2 dataloaders with 100% of the training and 100% of validation samples.One of them doesn't contain the y (unlabeled). The other one contains the labels. We'll use the unlabeled dataset (udls) to pretrain the model.<jupyter_code># 100% train data
tfms = [None, TSClassification()]
batch_tfms = [TSStandardize(by_sample=True)]
check_data(X, y, splits)
dls100 = get_ts_dls(X, y, splits=splits, tfms=tfms, batch_tfms=batch_tfms)
udls100 = get_ts_dls(X, splits=splits, tfms=tfms, batch_tfms=batch_tfms) # used in pretraining<jupyter_output>X      - shape: [4925 samples x 6 features x 36 timesteps]  type: memmap  dtype:float32  isnan: 0
y      - shape: (4925,)  type: memmap  dtype:<U2  n_classes: 14 (351 samples per class) ['15', '16', '42', '52', '53', '6', '62', '64', '65', '67', '88', '90', '92', '95']  isnan: False
splits - n_splits: 2 shape: [2459, 2466]  overlap: [False]<jupyter_text>We'll also need a labeled dataloaders with 10% of the training and 100% of validation data.<jupyter_code># 10% train data
train_split010 = get_splits(y[splits[0]], valid_size=.1, show_plot=False)[1]
splits010 = (train_split010, splits[1])
check_data(X, y, splits010)
dls010 = get_ts_dls(X, y, splits=splits010, tfms=tfms, batch_tfms=batch_tfms)<jupyter_output>X      - shape: [4925 samples x 6 features x 36 timesteps]  type: memmap  dtype:float32  isnan: 0
y      - shape: (4925,)  type: memmap  dtype:<U2  n_classes: 14 (351 samples per class) ['15', '16', '42', '52', '53', '6', '62', '64', '65', '67', '88', '90', '92', '95']  isnan: False
splits - n_splits: 2 shape: [245, 2466]  overlap: [False]<jupyter_text>Supervised 👀 First we'll train a model in a supervised way to set a baseline. We'll train using 10% and 100% of the training set. We'll run 10 tests.We'll train all models with the same settings (50 epochs with 10% of labels, 20 and 50 epochs with 100% of labels and lr=1e-2) to see the impact of pretraining.<jupyter_code># supervised 10%
n_epochs = 50
n_tests = 10
_result = []
for i in range(n_tests):
    clear_output()
    if i > 0: print(f'{i}/{n_tests} accuracy: {np.mean(_result):.3f} +/- {np.std(_result):.3f}')
    else: print(f'{i}/{n_tests}')
    learn = ts_learner(dls010, InceptionTimePlus, metrics=accuracy)
    learn.fit_one_cycle(n_epochs, 1e-2)
    _result.append(learn.recorder.values[-1][-1])
learn.plot_metrics()
print(f'\naccuracy: {np.mean(_result):.3f} +/- {np.std(_result):.3f} in {n_tests} tests')
# supervised 100%
n_epochs = 50
n_tests = 10
_result = []
for i in range(n_tests):
    clear_output()
    if i > 0: print(f'{i}/{n_tests} accuracy: {np.mean(_result):.3f} +/- {np.std(_result):.3f}')
    else: print(f'{i}/{n_tests}')
    learn = ts_learner(dls100, InceptionTimePlus, metrics=accuracy)
    learn.fit_one_cycle(n_epochs, 1e-2)
    _result.append(learn.recorder.values[-1][-1])
learn.plot_metrics()
print(f'\naccuracy: {np.mean(_result):.3f} +/- {np.std(_result):.3f} in {n_tests} tests')<jupyter_output>9/10 accuracy: 0.713 +/- 0.009<jupyter_text>I've also trained the model with all labels for 20 as it seems to be overfitting with 50 epochs.<jupyter_code># supervised 100%
n_epochs = 20
n_tests = 10
_result = []
for i in range(n_tests):
    clear_output()
    if i > 0: print(f'{i}/{n_tests} accuracy: {np.mean(_result):.3f} +/- {np.std(_result):.3f}')
    else: print(f'{i}/{n_tests}')
    learn = ts_learner(dls100, InceptionTimePlus, metrics=accuracy)
    learn.fit_one_cycle(n_epochs, 1e-2)
    _result.append(learn.recorder.values[-1][-1])
learn.plot_metrics()
print(f'\naccuracy: {np.mean(_result):.3f} +/- {np.std(_result):.3f} in {n_tests} tests')<jupyter_output>9/10 accuracy: 0.716 +/- 0.006<jupyter_text>This is a great result. Just for reference, in a recent review of multivariate time series models (<NAME>., <NAME>., <NAME>., <NAME>., & <NAME>. (2020). The great multivariate time series classification bake off: a review and experimental evaluation of recent algorithmic advances. Data Mining and Knowledge Discovery, 1-49.), the best performing classifier on this dataset is MUSE with an accuracy of **63.62**. Let's see if we can improve our baseline pretraining InceptionTime using `TSBERT`.  Pretrain model  🏋️‍♂️ Now we'll train a model without any labels on the entire training set. To do that we need to use the `TSBERT` callback. You can get more details on this callback visiting [`tsai` documentation](https://timeseriesai.github.io/tsai/callback.TSBERT).<jupyter_code># Unlabeled 100%
learn = ts_learner(udls100, InceptionTimePlus, cbs=[ShowGraph(), TSBERT(target_dir='./data/TSBERT', fname=f'{dsid}_200')])
learn.fit_one_cycle(200, 1e-2)
learn.TSBERT.show_preds(sharey=True)<jupyter_output><empty_output><jupyter_text>Fine-tune 🎻 There are at least 2 options to use the pretrained model weights: 1.   Fine-tune2.   TrainWe'll start by fine-tuning the pretrained model.In this case, we double the base_lr as the training lr will be the base_lr / 2. The only net change of the fine tuning then is just a training of the new head for 10 epochs. The rest of the training will be the same.Before training though, we'll check that when the model is frozen only the last layer is trained:<jupyter_code>learn = ts_learner(dls010, InceptionTimePlus, pretrained=True, weights_path=f'data/TSBERT/{dsid}_200.pth', metrics=accuracy)
for p in learn.model.parameters():
    p.requires_grad=False
print(f'{"trainable params once manually frozen":40}: {count_parameters(learn.model):8}')
learn.freeze()
print(f'{"trainable params after learn.freeze()":40}: {count_parameters(learn.model):8}')
learn.unfreeze()
print(f'{"trainable params learn.unfreeze()":40}: {count_parameters(learn.model):8}')<jupyter_output>weights from data/TSBERT/LSST_200.pth successfully transferred!

trainable params once manually frozen   :        0
trainable params after learn.freeze()   :     3854
trainable params learn.unfreeze()       :   457614<jupyter_text>It seems to be working well.<jupyter_code># self-supervised: fine-tuning with 10% labels
n_epochs = 50
freeze_epochs = 10
n_tests = 10
_result = []
for i in range(n_tests):
    clear_output()
    if i > 0: print(f'{i}/{n_tests} accuracy: {np.mean(_result):.3f} +/- {np.std(_result):.3f}')
    else: print(f'{i}/{n_tests}')
    learn = ts_learner(dls010, InceptionTimePlus, pretrained=True, weights_path=f'data/TSBERT/{dsid}_200.pth', metrics=accuracy)
    learn.fine_tune(n_epochs, base_lr=2e-2, freeze_epochs=freeze_epochs)
    _result.append(learn.recorder.values[-1][-1])
learn.plot_metrics()
print(f'\naccuracy: {np.mean(_result):.3f} +/- {np.std(_result):.3f} in {n_tests} tests')
# self-supervised: fine-tuning with 100% labels
n_epochs = 50
freeze_epochs = 10
n_tests = 10
_result = []
for i in range(n_tests):
    clear_output()
    if i > 0: print(f'{i}/{n_tests} accuracy: {np.mean(_result):.3f} +/- {np.std(_result):.3f}')
    else: print(f'{i}/{n_tests}')
    learn = ts_learner(dls100, InceptionTimePlus, pretrained=True, weights_path=f'data/TSBERT/{dsid}_200.pth', metrics=accuracy)
    learn.fine_tune(n_epochs, base_lr=2e-2, freeze_epochs=freeze_epochs)
    _result.append(learn.recorder.values[-1][-1])
learn.plot_metrics()
print(f'\naccuracy: {np.mean(_result):.3f} +/- {np.std(_result):.3f} in {n_tests} tests')
# self-supervised: fine-tuning with 100% labels
n_epochs = 20
freeze_epochs = 10
n_tests = 10
_result = []
for i in range(n_tests):
    clear_output()
    if i > 0: print(f'{i}/{n_tests} accuracy: {np.mean(_result):.3f} +/- {np.std(_result):.3f}')
    else: print(f'{i}/{n_tests}')
    learn = ts_learner(dls100, InceptionTimePlus, pretrained=True, weights_path=f'data/TSBERT/{dsid}_200.pth', metrics=accuracy)
    learn.fine_tune(n_epochs, base_lr=2e-2, freeze_epochs=freeze_epochs)
    _result.append(learn.recorder.values[-1][-1])
learn.plot_metrics()
print(f'\naccuracy: {np.mean(_result):.3f} +/- {np.std(_result):.3f} in {n_tests} tests')<jupyter_output>9/10 accuracy: 0.717 +/- 0.003
weights from data/TSBERT/LSST_200.pth successfully transferred!<jupyter_text>Train 🏃🏽‍♀️🏃🏽‍♀<jupyter_code># self-supervised: train with 10% labels
n_epochs = 50
n_tests = 10
_result = []
for i in range(n_tests):
    clear_output()
    if i > 0: print(f'{i}/{n_tests} accuracy: {np.mean(_result):.3f} +/- {np.std(_result):.3f}')
    else: print(f'{i}/{n_tests}')
    learn = ts_learner(dls010, InceptionTimePlus, pretrained=True, weights_path=f'data/TSBERT/{dsid}_200.pth', metrics=accuracy)
    learn.fit_one_cycle(n_epochs, 1e-2)
    _result.append(learn.recorder.values[-1][-1])
learn.plot_metrics()
print(f'\naccuracy: {np.mean(_result):.3f} +/- {np.std(_result):.3f} in {n_tests} tests')
# self-supervised: train with 100% labels
n_epochs = 50
n_tests = 10
_result = []
for i in range(n_tests):
    clear_output()
    if i > 0: print(f'{i}/{n_tests} accuracy: {np.mean(_result):.3f} +/- {np.std(_result):.3f}')
    else: print(f'{i}/{n_tests}')
    learn = ts_learner(dls100, InceptionTimePlus, pretrained=True, weights_path=f'data/TSBERT/{dsid}_200.pth', metrics=accuracy)
    learn.fit_one_cycle(n_epochs, 1e-2)
    _result.append(learn.recorder.values[-1][-1])
learn.plot_metrics()
print(f'\naccuracy: {np.mean(_result):.3f} +/- {np.std(_result):.3f} in {n_tests} tests')
# self-supervised 100% + training
n_epochs = 20
n_tests = 10
_result = []
for i in range(n_tests):
    clear_output()
    if i > 0: print(f'{i}/{n_tests} accuracy: {np.mean(_result):.3f} +/- {np.std(_result):.3f}')
    else: print(f'{i}/{n_tests}')
    learn = ts_learner(dls100, InceptionTimePlus, pretrained=True, weights_path=f'data/TSBERT/{dsid}_200.pth', metrics=accuracy)
    learn.fit_one_cycle(n_epochs, 1e-2)
    _result.append(learn.recorder.values[-1][-1])
learn.plot_metrics()
print(f'\naccuracy: {np.mean(_result):.3f} +/- {np.std(_result):.3f} in {n_tests} tests')<jupyter_output>9/10 accuracy: 0.734 +/- 0.005
weights from data/TSBERT/LSST_200.pth successfully transferred!<jupyter_text>Adding data augmentation 🔎🔎 One last thing I'd like to test is the impact of data augmentation when using a pretrained model. I will compare the performance of a model trained from scratch and a pretrained model adding CutMix (CutMix1D in `tsai`). We'll see if the difference in performance still holds.<jupyter_code># self-supervised 100% + training
n_epochs = 20
n_tests = 10
_result = []
for i in range(n_tests):
    clear_output()
    if i > 0: print(f'{i}/{n_tests} accuracy: {np.mean(_result):.3f} +/- {np.std(_result):.3f}')
    else: print(f'{i}/{n_tests}')
    learn = ts_learner(dls100, InceptionTimePlus, metrics=accuracy, cbs=CutMix1D())
    learn.fit_one_cycle(n_epochs, 1e-2)
    _result.append(learn.recorder.values[-1][-1])
learn.plot_metrics()
print(f'\naccuracy: {np.mean(_result):.3f} +/- {np.std(_result):.3f} in {n_tests} tests')
# self-supervised 100% + training with cutmix
n_epochs = 20
n_tests = 10
_result = []
for i in range(n_tests):
    clear_output()
    if i > 0: print(f'{i}/{n_tests} accuracy: {np.mean(_result):.3f} +/- {np.std(_result):.3f}')
    else: print(f'{i}/{n_tests}')
    learn = ts_learner(dls100, InceptionTimePlus, pretrained=True, weights_path=f'data/TSBERT/{dsid}_200.pth', metrics=accuracy, cbs=CutMix1D())
    learn.fit_one_cycle(n_epochs, 1e-2)
    _result.append(learn.recorder.values[-1][-1])
learn.plot_metrics()
print(f'\naccuracy: {np.mean(_result):.3f} +/- {np.std(_result):.3f} in {n_tests} tests')<jupyter_output>9/10 accuracy: 0.751 +/- 0.006
weights from data/TSBERT/LSST_200.pth successfully transferred!
{'fraction_non_alphanumeric': 0.1727648952243316, 'fraction_numerical': 0.04473494055048282, 'mean_word_length': 4.944552909019914, 'pattern_counts': {'":': 3, '<': 52, '<?xml version=': 0, '>': 61, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '336088', 'license': 'Apache-2.0', 'n_tokens_mistral': 6453, 'n_tokens_neox': 5643, 'n_words': 1440, 'path': 'tutorial_nbs/08_Self_Supervised_TSBERT.ipynb', 'repo_name': 'sjdlloyd/tsai', 'size': '15314'}