1. An image processing apparatus comprising: a first memory configured to store frequency values of respective feature amounts; a second memory configured to store some of the frequency values stored in the first memory, wherein the second memory has smaller capacity than the first memory and is accessible at higher speed than the first memory; a generation unit configured to generate a histogram of feature amounts of respective pixels of an image by receiving the feature amount of each pixel of the image and, if a frequency value of the feature amount is registered in the second memory, increasing the frequency value in the second memory, and, if the frequency value is not registered in the second memory, reading out the frequency value from the first memory into the second memory, and increasing the frequency value in the second memory; and a rearranging unit configured to rearrange bins of the histogram so that bins with high frequency values are close to each other in the histogram.