1. An image processing device that performs processing on a plurality of pieces of photographed image data photographed by a camera array in which cameras mounting any of two or more types of filter set are arranged in an array, the image processing device comprising: a first color information calculation unit configured to calculate, from pixel values of the plurality of pieces of photographed image data, first color information that is color information in a common color space; a distance information calculation unit configured to calculate a corresponding point between images indicated by the plurality of pieces of photographed image data by using the calculated first color information, to thereby calculate information on a depth of a subject in the image indicated by the photographed image data; a second color information calculation unit configured to calculate, from the pixel values of the plurality of pieces of photographed image data, second color information that is color information used for reproducing a color of the subject; and a combination unit configured to combine the plurality of pieces of photographed image data by using the calculated depth information and second color information, wherein each of the plurality of pieces of photographed image data is photographed by a camera mounting any of filter sets obtained by dividing a six-color color filter into two sets of a three-color color filter.