1. An image processing apparatus comprising: an image capturing unit configured to capture a plurality of images having different in-focus positions; a defocus amount calculating unit configured to calculate a defocus amount of an object image in a second image among the plurality of images relative to the object image in a first image among the plurality of images; a filtering unit configured to perform filtering on the first image according to the defocus amount calculated by the defocus amount calculating unit, wherein the filtering unit makes equal the defocus amounts of the object image in the second image and the object image in the first image; and a displacement amount detecting unit configured to detect a displacement amount between the second image and the first image on which filtering was performed by the filtering unit.