1. An image processing apparatus comprising: a spatial information calculation unit configured to calculate spatial information of a subject, the spatial information being information of an area in which the subject in an image is predicted to be present; a first area setting unit configured to set a first area in the image based on the spatial information; a second area setting unit configured to set a second area outside the first area; a first feature amount calculation unit configured to calculate a first feature amount of the first area; a second feature amount calculation unit configured to calculate a second feature amount of the second area, the second feature amount being a feature amount of the same type as the first feature amount; and an saliency calculation unit configured to calculate a degree of visual saliency of the subject based on a difference between the first feature amount and the second feature amount.