1. An image processing apparatus comprising: a comparison unit configured to select a pixel of interest in a processing image, and compare magnitudes of luminance value of the pixel of interest and luminance value of each of a plurality of neighboring pixels located at predetermined positions relative to the pixel of interest; and a calculation unit configured to calculate a feature amount of the pixel of interest by setting different weight values to respective neighboring pixels located at the predetermined positions depending on their positions relative to the pixel of interest, multiplying a weight value set to a respective one of the neighboring pixels located at the predetermined positions and a value representing a comparison result of comparing the luminance value of the pixel of interest and the luminance value of the respective one of the neighboring pixels located at the predetermined positions, and summing up results of multiplication, wherein for two neighboring pixels at positions which are point symmetrical with respect to the pixel of interest, said comparison unit sets that only one of the two neighboring pixels belongs to the neighboring pixels located at the predetermined positions.