1. A computer-implemented method for determining face similarity, the method comprising: using one or more processors to perform steps comprising: detecting a plurality of persons collectively depicted in a collection of images; performing image analysis for individual persons that appear in one or more images in the collection, including determining a plurality of visual features and determining gender, age, or racial classifications for at least some of the persons depicted in the collection from performance of the image analysis, wherein the plurality of visual features comprise skin tone, hair color, hair length, or face type; storing information determined from performing image analysis for the individual persons; receiving one or more inputs from the user that (i) include an image of a person, and (ii) identify one or more gender, age, or racial classifications; and performing a similarity search using said one or more inputs, in order to identify images of other persons depicted in one or more images of the collection that are both (i) similar in appearance to the person in the image input by the user, and (ii) match the gender, age, or racial classifications identified by the one or more inputs; wherein performing the similarity search includes determining similarities between a plurality of visual features of the person depicted in the image input by the user and the plurality of visual features of persons depicted in the one or more images of the collection, wherein the determined similarities are weighted based upon the gender, age, or racial classifications identified by the one or more inputs.