Pretraining takes place on Scriptbase
[1]}, a dataset which consists of
\(\sim \) 1,100 full-length screenplays (approx. 140K scenes). We adapt, a
self-supervised task, namely Contrastive Predictive Coding (CPC;
[2]}) to our setting: given a
(contextualized) scene representation, we learn to predict a future
representation in the movie. We consider a context window of several
future scenes, rather than just one. This is an attempt to account for
non-linearities in the movie, which can occur because of unrelated
intervening events and subplots. Given the representation of an
anchor scene \(g_i\) , a positive future representation \(c_i^{+}\)  and a
set of negative examples \(\lbrace c_{i1}^{-}, \dots , c_{i(N-1)}^{-}\rbrace \) , we
compute the InfoNCE [2]} loss:
\(\hspace{-10.0pt}\mathcal {L}_\mathit {self} = - \frac{1}{N} \sum _{j=1}^{N} \log \frac{e^{s(g_i, c_i^{+})/\tau }}{e^{s(g_i, c_i^{+})/\tau } + \sum _{k=1}^{N-1}e^{s(g_i, c_{ik}^{-})/\tau }} \) 
