1. An image processing apparatus capable of generating a plurality of output images having different focus positions by reconstructing an input image, the image processing apparatus comprising: a storage unit configured to store image pickup condition information of the input image; and an image processing unit configured to generate the output image from the input image using the image pickup condition information, wherein the image processing unit is configured to: obtain the input image that is information of an object space viewed from a plurality of viewpoints that is obtained via an imaging optical system and an image pickup element having a plurality of pixels; calculate an average pixel value of a pixel group of the input image of the same region of the object space, and substitute each pixel value of the pixel group by the average pixel value; and perform combination such that the pixels of the pixel group substituted by the average pixel value are shifted from each other to generate the output image.