1. A method performed by a data processing apparatus, the method comprising: accessing, by the data processing apparatus, images stored in an image data store, the images being associated with respective sets of labels, each label describing content depicted in an image and having a respective confidence score that is a measure of confidence that the label accurately describes the content depicted in the image; accessing, by the data processing apparatus, one or more proximity metric values associated with a first image of the images; accessing, by the data processing apparatus, one or more proximity metric values associated with a second image of the images; determining, by the data processing apparatus, a proximity score for the second image based on a comparison of the proximity metric values of the first image and corresponding proximity metric values of the second image, the proximity score being a measure of a relatedness of the second image to the first image; identifying, by the data processing apparatus, a second label associated with the second image, the second label having a second confidence score that is a measure of confidence that the second label accurately describes the content depicted in the second image; and generating, by the data processing apparatus, a first label for the first image based on the second label, the proximity score for the second image, and the second confidence score for the second label; and determining, for the first label, a first confidence score based on the proximity score for the second image and the second confidence score for the second label, the first confidence score being a measure of confidence that the first label accurately describes the content depicted in the first image.