<jupyter_start><jupyter_text>Using VideoGPTThis is a notebook demonstrating how to use VideoGPT and any pretrained models, Make sure that it is a GPU instance: **Change Runtime Type -> GPU**  InstallationFirst, we install the necessary packages<jupyter_code>! pip install git+https://github.com/wilson1yan/VideoGPT.git
! pip install scikit-video av
%matplotlib inline

from matplotlib import pyplot as plt
from matplotlib import animation
from IPython.display import HTML

import os
import torch
from torchvision.io import read_video, read_video_timestamps

from videogpt import download, load_vqvae
from videogpt.data import preprocess

VIDEOS = {
    'breakdancing': '1OZBnG235-J9LgB_qHv-waHZ4tjofiDgj',
    'bear': '16nIaqq2vbPh-WMo_7hs9feVSe0jWVXLF',
    'jaywalking': '1UxKCVrbyXhvMz_H7dI4w5hjPpRGCAApy',
    'cartoon': '1ONcTMSEuGuLYIDbX-KeFqd390vbTIH9d'
}<jupyter_output><empty_output><jupyter_text>Downloading the ModelThere are four pretrained models available: `bair_stride4x2x2`, `ucf101_stride4x4x4`, `kinetics_stride4x4x4`, and `kinetics_stride2x4x4`. BAIR was trained on 64 x 64 video, and the rest on 128 x 128. The `stride` component represents the THW downsampling the VQ-VAE performs on the video tensor.<jupyter_code>device = torch.device('cuda')
vqvae = load_vqvae('kinetics_stride2x4x4', device=device).to(device)<jupyter_output><empty_output><jupyter_text>Video Loading and PreprocessingThe code below downloads, loads, and preprocesses a given `mp4` file.<jupyter_code>video_name = 'jaywalking'
# `resolution` must be divisible by the encoder image stride
# `sequence_length` must be divisible by the encoder temporal stride
resolution, sequence_length = vqvae.hparams.resolution, 16

video_filename = download(VIDEOS[video_name], f'{video_name}.mp4')
pts = read_video_timestamps(video_filename, pts_unit='sec')[0]
video = read_video(video_filename, pts_unit='sec', start_pts=pts[0], end_pts=pts[sequence_length - 1])[0]
video = preprocess(video, resolution, sequence_length).unsqueeze(0).to(device)<jupyter_output><empty_output><jupyter_text>VQ-VAE Encoding and DecodingNow, we can encode the video through the `encode` function. The `encode` function also has an optional input `including_embeddings` (default `False`) which will also return the embedding versions of the encodings.<jupyter_code>with torch.no_grad():
    encodings = vqvae.encode(video)
    video_recon = vqvae.decode(encodings)
    video_recon = torch.clamp(video_recon, -0.5, 0.5)<jupyter_output><empty_output><jupyter_text>Visualizing Reconstructions<jupyter_code>videos = torch.cat((video, video_recon), dim=-1)
videos = videos[0].permute(1, 2, 3, 0) # CTHW -> THWC
videos = ((videos + 0.5) * 255).cpu().numpy().astype('uint8')

fig = plt.figure()
plt.title('real (left), reconstruction (right)')
plt.axis('off')
im = plt.imshow(videos[0, :, :, :])
plt.close()

def init():
    im.set_data(videos[0, :, :, :])

def animate(i):
    im.set_data(videos[i, :, :, :])
    return im

anim = animation.FuncAnimation(fig, animate, init_func=init, frames=videos.shape[0], interval=200) # 200ms = 5 fps
HTML(anim.to_html5_video())<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11443102352193261, 'fraction_numerical': 0.02733630006357279, 'mean_word_length': 6.675609756097561, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 23, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '577295', 'license': 'MIT', 'n_tokens_mistral': 1200, 'n_tokens_neox': 1131, 'n_words': 265, 'path': 'notebooks/Using_VideoGPT.ipynb', 'repo_name': 'zzyunzhi/VideoGPT', 'size': '3146'}