Moreover, for each shot we extract visual and audio features. We
consider three different types of visual features: (1) We sample one
key frame per shot and extract features using ResNeXt-101
[1]} pre-trained for object recognition on ImageNet
[2]}. (2) We sample frames with a frequency of 1
out of every 10 frames (we increase this time interval for shots with
larger duration since we face memory issues) and extract motion
features using the two-stream I3D network pre-trained on Kinetics
[3]}. (3) We use Faster-RCNN [4]}
implemented in Detectron2 [5]} to detect person
instances in every key frame and keep the top four bounding boxes per
shot which have the highest confidence alongside with the respective
regional representations.
We first project all individual representations to the same lower
dimension and perform L2-normalization. Next, we consider the visual
shot representation as the sum of the individual vectors.
For the audio modality, we use YAMNet pre-trained on the
AudioSet-YouTube corpus [6]} for classifying audio
segments into 521 audio classes (e.g., tools, music, explosion); for
each audio segment contained in the scene, we extract features from
the penultimate layer. Finally, we extract textual features
[7]} from subtitles and screenplay scenes using
the Universal Sentence Encoder (USE; [8]}).
