1. In a computing environment, a method performed at least in part on at least one processor, comprising, receiving face detection data corresponding to a face detected in an input video frame, building face galleries, including grouping faces detected in input video frames into candidate groups based upon similarity data, filtering at least some faces from a candidate group, adding remaining faces to one of the face galleries, and labeling each face gallery with the face identification data, matching the face detection data against face identification data maintained in a face gallery among a plurality of face galleries to recognize the face in the input video frame, and generating metadata that associates the video frame and the face with the face identification data.