1. A method of face tracking by an imaging device, comprising: receiving image data associated with a frame; receiving one or more face detection windows constrained to correspond to actual faces; determining a face list based on a lifetime value that is based on at least a detection confidence level for each face detection window, and a degree of confidence that each face in the face list is an actual face, a higher degree of confidence being based in part on a higher number of neighboring face detection windows for the same actual face; sorting the face list based on at least one lifetime value; and tracking one or more of the actual faces based on the sorted face list.