1. A method performed by data processing apparatus, the method comprising: receiving data identifying images determined to be responsive to a query, each image having an associated search score that is a measure of responsiveness to the query, and the images ranked according to a first order based on the search scores, and in response to receiving the data identifying the images: for each image in a first set of the images, detecting a face depicted in the image and generating a face template from the depicted face; clustering the images in the first set of images into a plurality of clusters based on similarity of the facial templates to each other, each cluster including at least one of the images, and each image belonging to only one cluster; determining a quantity of images belonging to each cluster; selecting the cluster with the highest quantity relative to the quantity of other clusters as an inlier cluster; for each image in the first set of images, determining an inlier score for the image that is a measure of similarity of the facial template of the image to the facial templates of the images that belong to the inlier cluster; and re-ranking the images determined to be responsive to a query based on the inlier scores of the set of images.