1. A method performed by data processing apparatus, the method comprising: receiving, by the data processing apparatus, data identifying images determined to be responsive to a query, each image having an associated search score that is a measure of responsiveness to the query, and the images ranked according to a first order based on the search scores, and in response to receiving the data identifying the images: for each image in a first set of the images, detecting, by the data processing apparatus, a face depicted in the image and generating a face template from the depicted face by processing the image using a facial template generation process that identifies machine-identifiable properties of structures of a face, and where each face template is a temporary face template that is not persisted to memory; clustering, by the data processing apparatus, the images in the first set of images into a plurality of clusters based on similarity of the facial templates to each other, each cluster including at least one of the images, and each image belonging to only one cluster; determining, by the data processing apparatus, a quantity of images belonging to each cluster; selecting, by the data processing apparatus, the cluster with the highest quantity relative to the quantity of other clusters as an inlier cluster; for each image in the first set of images, determining, by the data processing apparatus, an inlier score for the image that is a measure of similarity of the facial template of the image to the facial templates of the images that belong to the inlier cluster; and re-ranking, by the data processing apparatus, the images determined to be responsive to a query based on the inlier scores of the set of images; wherein the determining the inlier score comprises, for images that belong to the inlier cluster that are determined to be duplicate images of each other, using only one of the images from the images determined to be duplicate images of each other to determine the inlier score.