title nextframe video prediction with convolutional lstms amogh joshihttps github comamogh7joshi date created 20210602 last modified 20210605 description how to build and train a convolutional lstm model for nextframe video prediction accelerator gpu introduction the convolutional lstmhttps papers nips ccpaper2015file07563a3fe3bbe7e3ba84431ad9d055afpaper pdf architectures bring together time series processing and computer vision by introducing a convolutional recurrent cell in a lstm layer in this example we will explore the convolutional lstm model in an application to nextframe prediction the process of predicting what video frames come next given a series of past frames setup dataset construction for this example we will be using the moving mnisthttp www cs toronto edunitishunsupervisedvideo dataset we will download the dataset and then construct and preprocess training and validation sets for nextframe prediction our model will be using a previous frame which we ll call fn to predict a new frame called fn 1 to allow the model to create these predictions we ll need to process the data such that we have shifted inputs and outputs where the input data is frame xn being used to predict frame yn 1 download and load the dataset swap the axes representing the number of frames and number of data samples we ll pick out 1000 of the 10000 total examples and use those add a channel dimension since the images are grayscale split into train and validation sets using indexing to optimize memory normalize the data to the 01 range we ll define a helper function to shift the frames where x is frames 0 to n 1 and y is frames 1 to n apply the processing function to the datasets inspect the dataset data visualization our data consists of sequences of frames each of which are used to predict the upcoming frame let s take a look at some of these sequential frames construct a figure on which we will visualize the images plot each of the sequential images for one random data example print information and display the figure model construction to build a convolutional lstm model we will use the convlstm2d layer which will accept inputs of shape batchsize numframes width height channels and return a prediction movie of the same shape construct the input layer with no definite frame size we will construct 3 convlstm2d layers with batch normalization followed by a conv3d layer for the spatiotemporal outputs next we will build the complete model and compile it model training with our model and data constructed we can now train the model define some callbacks to improve training define modifiable training hyperparameters fit the model to the training data frame prediction visualizations with our model now constructed and trained we can generate some example frame predictions based on a new video we ll pick a random example from the validation set and then choose the first ten frames from them from there we can allow the model to predict 10 new frames which we can compare to the ground truth frame predictions select a random example from the validation dataset pick the firstlast ten frames from the example predict a new set of 10 frames extract the model s prediction and postprocess it extend the set of prediction frames construct a figure for the original and new frames plot the original frames plot the new frames display the figure predicted videos finally we ll pick a few examples from the validation set and construct some gifs with them to see the model s predicted videos you can use the trained model hosted on hugging face hubhttps huggingface cokerasioconvlstm and try the demo on hugging face spaceshttps huggingface cospaceskerasioconvlstm select a few random examples from the dataset iterate over the examples and predict the frames pick the firstlast ten frames from the example predict a new set of 10 frames extract the model s prediction and postprocess it extend the set of prediction frames create and save gifs for each of the ground truthprediction images construct a gif from the selected video frames construct a gif from the frames display the videos construct and display an hbox with the ground truth and prediction introduction the convolutional lstm https papers nips cc paper 2015 file 07563a3fe3bbe7e3ba84431ad9d055af paper pdf architectures bring together time series processing and computer vision by introducing a convolutional recurrent cell in a lstm layer in this example we will explore the convolutional lstm model in an application to next frame prediction the process of predicting what video frames come next given a series of past frames setup dataset construction for this example we will be using the moving mnist http www cs toronto edu nitish unsupervised_video dataset we will download the dataset and then construct and preprocess training and validation sets for next frame prediction our model will be using a previous frame which we ll call f_n to predict a new frame called f_ n 1 to allow the model to create these predictions we ll need to process the data such that we have shifted inputs and outputs where the input data is frame x_n being used to predict frame y_ n 1 download and load the dataset swap the axes representing the number of frames and number of data samples we ll pick out 1000 of the 10000 total examples and use those add a channel dimension since the images are grayscale split into train and validation sets using indexing to optimize memory normalize the data to the 0 1 range we ll define a helper function to shift the frames where x is frames 0 to n 1 and y is frames 1 to n apply the processing function to the datasets inspect the dataset data visualization our data consists of sequences of frames each of which are used to predict the upcoming frame let s take a look at some of these sequential frames construct a figure on which we will visualize the images plot each of the sequential images for one random data example print information and display the figure model construction to build a convolutional lstm model we will use the convlstm2d layer which will accept inputs of shape batch_size num_frames width height channels and return a prediction movie of the same shape construct the input layer with no definite frame size we will construct 3 convlstm2d layers with batch normalization followed by a conv3d layer for the spatiotemporal outputs next we will build the complete model and compile it model training with our model and data constructed we can now train the model define some callbacks to improve training define modifiable training hyperparameters fit the model to the training data frame prediction visualizations with our model now constructed and trained we can generate some example frame predictions based on a new video we ll pick a random example from the validation set and then choose the first ten frames from them from there we can allow the model to predict 10 new frames which we can compare to the ground truth frame predictions select a random example from the validation dataset pick the first last ten frames from the example predict a new set of 10 frames extract the model s prediction and post process it extend the set of prediction frames construct a figure for the original and new frames plot the original frames plot the new frames display the figure predicted videos finally we ll pick a few examples from the validation set and construct some gifs with them to see the model s predicted videos you can use the trained model hosted on hugging face hub https huggingface co keras io conv lstm and try the demo on hugging face spaces https huggingface co spaces keras io conv lstm select a few random examples from the dataset iterate over the examples and predict the frames pick the first last ten frames from the example predict a new set of 10 frames extract the model s prediction and post process it extend the set of prediction frames create and save gifs for each of the ground truth prediction images construct a gif from the selected video frames construct a gif from the frames display the videos construct and display an hbox with the ground truth and prediction
import numpy as np
import matplotlib.pyplot as plt

import keras
from keras import layers

import io
import imageio
from IPython.display import Image, display
from ipywidgets import widgets, Layout, HBox




fpath = keras.utils.get_file(
    "moving_mnist.npy",
    "http://www.cs.toronto.edu/~nitish/unsupervised_video/mnist_test_seq.npy",
)
dataset = np.load(fpath)


dataset = np.swapaxes(dataset, 0, 1)

dataset = dataset[:1000, ...]

dataset = np.expand_dims(dataset, axis=-1)


indexes = np.arange(dataset.shape[0])
np.random.shuffle(indexes)
train_index = indexes[: int(0.9 * dataset.shape[0])]
val_index = indexes[int(0.9 * dataset.shape[0]) :]
train_dataset = dataset[train_index]
val_dataset = dataset[val_index]


train_dataset = train_dataset / 255
val_dataset = val_dataset / 255




def create_shifted_frames(data):
    x = data[:, 0 : data.shape[1] - 1, :, :]
    y = data[:, 1 : data.shape[1], :, :]
    return x, y



x_train, y_train = create_shifted_frames(train_dataset)
x_val, y_val = create_shifted_frames(val_dataset)


print(
    "Training Dataset Shapes: " + str(x_train.shape) + ", " + str(y_train.shape)
)
print(
    "Validation Dataset Shapes: " + str(x_val.shape) + ", " + str(y_val.shape)
)




fig, axes = plt.subplots(4, 5, figsize=(10, 8))


data_choice = np.random.choice(range(len(train_dataset)), size=1)[0]
for idx, ax in enumerate(axes.flat):
    ax.imshow(np.squeeze(train_dataset[data_choice][idx]), cmap="gray")
    ax.set_title(f"Frame {idx + 1}")
    ax.axis("off")


print(f"Displaying frames for example {data_choice}.")
plt.show()




inp = layers.Input(shape=(None, *x_train.shape[2:]))



x = layers.ConvLSTM2D(
    filters=64,
    kernel_size=(5, 5),
    padding="same",
    return_sequences=True,
    activation="relu",
)(inp)
x = layers.BatchNormalization()(x)
x = layers.ConvLSTM2D(
    filters=64,
    kernel_size=(3, 3),
    padding="same",
    return_sequences=True,
    activation="relu",
)(x)
x = layers.BatchNormalization()(x)
x = layers.ConvLSTM2D(
    filters=64,
    kernel_size=(1, 1),
    padding="same",
    return_sequences=True,
    activation="relu",
)(x)
x = layers.Conv3D(
    filters=1, kernel_size=(3, 3, 3), activation="sigmoid", padding="same"
)(x)


model = keras.models.Model(inp, x)
model.compile(
    loss=keras.losses.binary_crossentropy,
    optimizer=keras.optimizers.Adam(),
)




early_stopping = keras.callbacks.EarlyStopping(monitor="val_loss", patience=10)
reduce_lr = keras.callbacks.ReduceLROnPlateau(monitor="val_loss", patience=5)


epochs = 20
batch_size = 5


model.fit(
    x_train,
    y_train,
    batch_size=batch_size,
    epochs=epochs,
    validation_data=(x_val, y_val),
    callbacks=[early_stopping, reduce_lr],
)




example = val_dataset[np.random.choice(range(len(val_dataset)), size=1)[0]]


frames = example[:10, ...]
original_frames = example[10:, ...]


for _ in range(10):
    
    new_prediction = model.predict(np.expand_dims(frames, axis=0))
    new_prediction = np.squeeze(new_prediction, axis=0)
    predicted_frame = np.expand_dims(new_prediction[-1, ...], axis=0)

    
    frames = np.concatenate((frames, predicted_frame), axis=0)


fig, axes = plt.subplots(2, 10, figsize=(20, 4))


for idx, ax in enumerate(axes[0]):
    ax.imshow(np.squeeze(original_frames[idx]), cmap="gray")
    ax.set_title(f"Frame {idx + 11}")
    ax.axis("off")


new_frames = frames[10:, ...]
for idx, ax in enumerate(axes[1]):
    ax.imshow(np.squeeze(new_frames[idx]), cmap="gray")
    ax.set_title(f"Frame {idx + 11}")
    ax.axis("off")


plt.show()




examples = val_dataset[np.random.choice(range(len(val_dataset)), size=5)]


predicted_videos = []
for example in examples:
    
    frames = example[:10, ...]
    original_frames = example[10:, ...]
    new_predictions = np.zeros(shape=(10, *frames[0].shape))

    
    for i in range(10):
        
        frames = example[: 10 + i + 1, ...]
        new_prediction = model.predict(np.expand_dims(frames, axis=0))
        new_prediction = np.squeeze(new_prediction, axis=0)
        predicted_frame = np.expand_dims(new_prediction[-1, ...], axis=0)

        
        new_predictions[i] = predicted_frame

    
    for frame_set in [original_frames, new_predictions]:
        
        current_frames = np.squeeze(frame_set)
        current_frames = current_frames[..., np.newaxis] * np.ones(3)
        current_frames = (current_frames * 255).astype(np.uint8)
        current_frames = list(current_frames)

        
        with io.BytesIO() as gif:
            imageio.mimsave(gif, current_frames, "GIF", duration=200)
            predicted_videos.append(gif.getvalue())


print(" Truth\tPrediction")
for i in range(0, len(predicted_videos), 2):
    
    box = HBox(
        [
            widgets.Image(value=predicted_videos[i]),
            widgets.Image(value=predicted_videos[i + 1]),
        ]
    )
    display(box)