1. An image processing apparatus comprising: an input unit which inputs an image obtained by an image pickup unit in which pixels that receive light passing through different pupil areas of a focusing optical system are arranged; a parallactic image generation unit which generates a plurality of parallactic images having a relationship in which a magnitude of an amount of image shift between the parallactic images increases with an increase in a magnitude of a defocus amount; a parallactic image series generation unit which, through a plurality of selections of the generated parallactic images permitting a repeated same selection, generates a series of parallactic images in which the selected parallactic images are arranged in a selection order; an interpolation processing unit which repeatedly applies interpolation processing to each pair of adjacent parallactic images included in the series of parallactic images to generate a plurality of interpolation images and arranges the generated interpolation images between the adjacent parallactic images to output the interpolated images; and an output unit which outputs an output of the interpolation processing unit as an output image of a moving image.