1. An image processing apparatus comprising: a subject information storage unit configured to store feature quantities and attributes relating to a plurality of subjects; a subject detecting unit configured to detect a subject included in an image; a scaling unit configured to scale the subject detected in the image to a scale of an image corresponding to the feature quantities and attributes stored in the subject information storage unit; an attribute determining unit configured to determine the attributes of said detected subject; a feature quantity extracting unit configured to extract a feature quantity relating to said detected subject; and a similarity calculating unit configured to select one or a plurality of feature quantities from feature quantities relating to a plurality of subjects stored in said subject information storage unit based on said determined attributes to calculate similarity between a subject according to said selected feature quantities, and said detected subject, based on said selected feature quantities and said extracted feature quantity.