1. An image search apparatus that searches an image similar to an inputted query image from registered images, the image search apparatus comprising: an extraction unit configured to acquire a pair of feature points corresponding to each other based on a feature amount of feature points in the query image and to extract a symmetric partial image based on the acquired pair of feature points, wherein among a straight line group passing through two points selected from a midpoint group obtained from the plurality of sets of pairs of feature points acquired from the query image, said extraction unit is configured to set a straight line, in which an accumulated distance from the midpoint group is minimum, as a symmetry axis and to extract an area including the pair of feature points with the midpoint near the symmetry axis as the symmetric partial image; a decision unit configured to decide two partial areas, which are obtained by dividing the partial image at a symmetry axis of the partial image, as a first area and a second area based on the tendency of image features in the partial image; a setting unit configured to extract, based on the feature amount, at least two sets of corresponding pairs of feature points from the feature points in partial areas decided as the first area of the query image and a registered comparison destination image and to set a coordinate transformation coefficient for transforming the feature points of one of the two images so that the coordinates of the extracted pairs of feature points match; and a comparison unit configured, in order to determine the similarity between the two images, to extract pairs of feature points corresponding to each other from the feature points in the two images based on the feature amount to transform coordinates of the feature points extracted from the one of the images by a coordinate transformation process using the coordinate transformation coefficient, and to compare the coordinates of the feature points after the transformation with coordinates of the feature points in the other image of the two images, wherein at least one processor functions as at least one of the units.