1. A computer-implemented method for searching a collection of content, comprising: under control of one or more computer systems configured with executable instructions, identifying a visually significant subset of categories of a category tree that categorizes the collection of content, the collection of content corresponding to a plurality of items each associated with one or more images, the visually significant subset including the categories of the category tree that are each associated with a respective set of images having a respective size that is greater than a lower size threshold and less than an upper size threshold; receiving a search request associated with a query image; identifying, among the visually significant subset, a candidate set of categories for the query image based at least in part on the search request, wherein at least one parent category is automatically excluded from the candidate set when the candidate set includes at least one child category of the at least one parent category; determining feature vectors for the query image and cluster descriptors for images of items associated with the candidate set of categories; determining visual similarity scores quantifying visual similarity between each of the images of items associated with the candidate set of categories and the query image based at least in part on the feature vectors of the query image and the cluster descriptors for the images of items associated with the candidate set of categories, a respective visual similarity score of each image of an item with respect to the query image being weighted based at least in part on a position of a category of the item in the category tree; selecting a result set from among the images of items associated with the categories in the candidate set based at least in part on the visual similarity scores; and providing information corresponding to the result set for presentation.