Previous work
[1]} evaluates the performance of TP
identification models in terms of three metrics: Total Agreement (TA),
i.e., the percentage of TP scenes that are correctly identified,
Partial Agreement (PA), i.e., the percentage of TP events for which at
least one gold-standard scene is identified, and Distance (D), i.e.,
the minimum distance in number of scenes between the predicted and
gold-standard set of scenes for a given TP, normalized by the
screenplay length. We report results with the partial agreement
metric. We can no longer use total agreement, since we evaluate
against silver standard (rather than gold) labels for shots (rather
than scenes) and as a result consider all shots within a scene equally
important. We do not use the distance metric either since it yields
very similar outcomes and does not help discriminate among model
variants.
