1. An image processing device for generating an output image as a color image wherein each pixel has color information, using a first image as a sparsely color sampled image wherein each pixel has monochromatic information, which is obtained by an image sensor which has multiple pixels that photoelectrically convert light of a plurality of colors, the image processing device comprising: a simulation unit configured to generate a second image as an estimated image of the sparsely color sampled image by simulating imaging using the image sensor based on the first image as the color image; a comparison unit configured to compare the second image with the first image, and to generate first information which shows a comparison result; a converting unit configured to convert the first information into second information which corresponds to the color image; a penalty computing unit configured to compute a correction amount of the first image based on an image penalty which is determined corresponding to the first image; and a correction unit configured to generate a third image as the color image, by correcting the first image based on at least one of the second information and the correction amount; wherein the third image which is generated by repeating the processing of the correction unit as the input image of the third image, becomes the output image.