1. A non-transitory computer-readable medium embodying a program that, when executed by at least one computing device, causes the at least one computing device to at least: obtain data indicating a plurality of cast members known to appear in a video program; subsequent to obtaining the data indicating the plurality of cast members known to appear in the video program, sample a first frame from the video program; detect a face in the first frame; recognize the detected face in the first frame as being one of the plurality of cast members known to appear in the video program based at least in part on a plurality of reference images corresponding to the plurality of cast members, wherein recognizing the detected face in the first frame is restricted to a set of faces recognized within a predefined time period in response to determining that a quantity of detected faces in the first frame meets a maximum threshold; generate a first association between the one of the plurality of cast members and the first frame when the detected face is recognized as being the one of the plurality of cast members; and generate a second association between the one of the plurality of cast members and a second frame of the video program based at least in part on the first association, a third association between the one of the plurality of cast members and a third frame of the video program, and a temporal smoothing factor.