1. An image processing apparatus, comprising: an image generating unit, executed by a processor, that generates, from output data of a first receive unit and a second receive unit, and from output data of a third receive unit and a fourth receive unit among a plurality of receive units that is disposed at each of a plurality of micro-lenses and outputs the output data to generate an image on an arbitrary focal plane, the image on the focal plane that is constituted with more pixels than the plurality of micro-lenses, the first receive unit being disposed at a predetermined micro-lens and used regardless of the focal plane of the image to be generated, the second receive unit being disposed near at the predetermined micro-lens and changed in accordance with change of the focal plane of the image to be generated, the third receive unit being different from the first receive unit, disposed at the predetermined micro-lens and used regardless of the focal plane of the image to be generated, and the fourth receive unit being disposed near at the predetermined micro-lens and changed in accordance with change of the focal plane of the image to be generated, wherein the first receive unit, the second receive unit, the third receive unit, an the fourth receive unit each comprises image-capturing pixels.