1. A method implemented by data processing apparatus, the method comprising: receiving a non-image query that is a query that does not include a query image provided by input at a user device; receiving, separate from the non-image query, a query image from the user device; identifying a result image responsive to the non-image query, the result image identified from a set of images that do not include the query image and being an image that is different from the query image; in response to identifying a result image responsive to the non-image query, determining that an object depicted in a sub-portion of the query image is similar to the result image, the determining comprising comparing visual features of the result image to visual features of the query image; and in response to determining that the object depicted in the sub-portion of the query image is similar to the result image: providing annotation data to the user device that causes the user device to display an annotation with the query image, the annotation specifying the sub-portion of the query image that includes the object that is similar to the result image.