1. An image processing apparatus for generating a time-lapse moving image by using time series parallax images picked up with different viewpoints for a plurality of frames, the image processing apparatus comprising: a specification unit configured to specify a focusing position for one frame of a plurality of frames; an image generation unit configured to generate a refocus image focused at a virtual focusing position in one frame by relatively shifting and combining the first and second parallax images corresponding to the one frame; and a moving image generation unit configured to generate a time-lapse moving image of the plurality of frames including in the frame the refocus image generated by the image generation unit, wherein, in the case where a focusing position in the first frame is specified by the specification unit, the image generation unit relatively shifts the first and second parallax images corresponding to the first frame with a first shift amount corresponding to the focusing position specified by the specification unit, generates a first refocus image by combining the shifted first and second parallax images, relatively shifts a third and fourth parallax images corresponding to a second frame prior to the first frame with a second shift amount smaller than the first shift amount, and generates a second refocus image by combining the shifted third and fourth parallax images, wherein the moving image generation unit generates a time-lapse moving image including the first and second refocus images.