1. An image processing apparatus capable of generating a plurality of output images having different focus positions by processing an input image, the image processing apparatus comprising: a non-transitory memory configured to store image pickup condition information of the input image; and an image processor configured to generate the output image from the input image using the image pickup condition information, wherein the image processor is configured to: obtain the input image including information of an object space viewed from a plurality of viewpoints that is obtained via an imaging optical system and an image pickup element having a plurality of pixels including a group of distinct pixels that have imaged the same region of the object space during an image pickup; calculate an average of pixel values of the distinct pixels in the group of distinct pixels that have imaged the same region of the object space, the imaged same region of the object space corresponding in size to a size of each distinct pixel included in the group of distinct pixels, and substitute each of the pixel values of the distinct pixels in the group of distinct pixels with the calculated average of pixel values of the distinct pixels in the group of distinct pixels, the calculated average of pixel values being the average of pixel values of the distinct pixels in the group of distinct pixels obtained by the image pickup; and perform combination such that the distinct pixels in the group of distinct pixels having pixel values substituted with the calculated average of pixel values are shifted from each other to generate the output image.