<jupyter_start><jupyter_text>Transformer > Training a Timesformer model for UCR video classif.Tbaks to <NAME> (@lucidrains) we have a bunch of attention based models to trian:- `Is Space-Time Attention All You Need for Video Understanding?`: This paper looks pretty cool, as it is the first full attention model for video. The training is complicated without pretrained models, but this will be solved soon. The model code comes from LucidRains [implementation](https://github.com/lucidrains/TimeSformer-pytorch)- `STAM (Space Time Attention Model)`: This is a ddiferent type of joint space-time attention. Code [here](https://github.com/lucidrains/STAM-pytorch)- `ViVit` : Our google friends made this model that I have not tried yet, and thanks to Phil and rishikksh20 we have some code [here](https://github.com/rishikksh20)All this models are difficult to train, so if you find ways to train them from scratch, or you make loading ViT weights on the image encoder part let me know.<jupyter_code>from fastai.vision.all import *
from action_recognition.core import *
from action_recognition.models import *
torch.cuda.set_device(0)
torch.cuda.get_device_name()
data_path = Path.home()/'.fastai/data/UCF101-frames'
instances = get_instances(data_path)
seq_len = 20
step=5
image_size = 128
bs = 16<jupyter_output><empty_output><jupyter_text>you could put this split on a text file:<jupyter_code>dls = get_action_dataloaders(instances, bs=bs, image_size=image_size, seq_len=seq_len, step=step).cuda()<jupyter_output><empty_output><jupyter_text>TimesFormer<jupyter_code>model = TimeSformer(
    dim = 128,
    image_size = 128,
    patch_size = 16,
    num_frames = 20,
    num_classes = dls.c,
    depth = 12,
    heads = 8,
    dim_head =  64,
    attn_dropout = 0.1,
    ff_dropout = 0.1
)
model = model.cuda()
model(x).shape
learn = Learner(dls, model, metrics=[accuracy, top_k_accuracy], wd=0.1, opt_func=ranger).to_fp16()
learn.lr_find()
learn.fit_flat_cos(10, 1e-3)<jupyter_output><empty_output><jupyter_text>It needs ImageNet pretraining of the encoder. It is documented on the paper.> First, we attempted to train TimeSformer on video datasetsdirectly, without ImageNet pretraining. For these exper-iments, we followed the training-from-scratch protocolof Touvron et al. (2020) and we also evaluated some vari-ants of it. However, the model failed to learn meaningfulfeatures.<jupyter_code>learn.show_results()<jupyter_output><empty_output><jupyter_text>STAM<jupyter_code>model = STAM(
    dim = 256,
    image_size = 128,     # size of image
    patch_size = 16,      # patch size
    num_frames = 20,       # number of image frames, selected out of video
    space_depth = 6,     # depth of vision transformer
    space_heads = 8,      # heads of vision transformer
    space_mlp_dim = 512, # feedforward hidden dimension of vision transformer
    time_depth = 6,       # depth of time transformer (in paper, it was shallower, 6)
    time_heads = 4,       # heads of time transformer
    time_mlp_dim = 512,  # feedforward hidden dimension of time transformer
    num_classes = dls.c,    # number of output classes
    space_dim_head = 64,  # space transformer head dimension
    time_dim_head = 64,   # time transformer head dimension
    dropout = 0.1,         # dropout
    emb_dropout = 0.1      # embedding dropout
)
learn = Learner(dls, model, metrics=[accuracy, top_k_accuracy], wd=0.1, opt_func=ranger).to_fp16()
learn.lr_find()
learn.fit_flat_cos(10, 1e-4)
learn.show_results()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08907801418439716, 'fraction_numerical': 0.023829787234042554, 'mean_word_length': 4.475155279503106, 'pattern_counts': {'":': 0, '<': 22, '<?xml version=': 0, '>': 24, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '23548', 'license': 'Apache-2.0', 'n_tokens_mistral': 1213, 'n_tokens_neox': 1130, 'n_words': 359, 'path': '03_train_transformer.ipynb', 'repo_name': 'tcapelle/action_recognition', 'size': '3528'}