1. An image processing device that converts a first image captured via an optical system in which one pixel has at least one of a plurality of color components and MTF characteristics are different between at least two color components at an imaging plane, into a second image in which MTF characteristics are matched, comprising: an MTF correction unit that performs smoothing or sharpening processing to one of the at least two color components of the first image to correct MTF thereof; a comparison unit that performs a plurality of ways of processing for correcting the MTF and compares color responses therebetween; a decision unit that decides one MTF correction processing out of the plurality of ways of processing for correcting the MTF based on a result of comparing the color responses by the comparison unit; and an image conversion unit that matches MTF characteristics of one color component out of the at least two color components to MTF characteristics of the other color component based on a result of performing the one MTF correction processing decided by the decision unit.