1. An image processing apparatus comprising: an interpolation unit configured to perform an interpolation process in each of a plurality of defined directions on a mosaic image signal generated by an image pickup device which outputs an image signal corresponding to one of colors of a plurality of color filters for each pixel; an evaluation unit configured to acquire an evaluation value representing a correlation for each of the defined directions for each of pixels of the image signal having undergone the interpolation process by the interpolation unit; and an output unit configured to correct the evaluation values for at least one pixel of interest based on the evaluation values for peripheral pixels about the at least one pixel of interest if the at least one pixel of interest satisfies a predetermined condition, and to output an image signal after the at least one pixel of interest is interpolated based on the corrected evaluation values, wherein the predetermined condition includes a condition that the at least one pixel of interest is an isolated point.