1. An image processing apparatus comprising: a comparison unit adapted to select a pixel of interest in a processing image, and compare magnitudes of luminance value of the pixel of interest and luminance value of each of a plurality of neighboring pixels having a predetermined positional relationship with the pixel of interest; and a calculation unit adapted to calculate a feature amount of the pixel of interest based on said predetermined positional relationship between the pixel of interest and each of the plurality of neighboring pixels and a comparison result obtained by said comparison unit, wherein for two neighboring pixels at positions which are point symmetrical with respect to the pixel of interest, said comparison unit sets that only one of the two neighboring pixels has the predetermined positional relationship.