1. An image processing apparatus comprising: a processor configured to perform operations including: obtaining reference image data and parallax image data generated by image-capturing the same scene, the parallax image data having parallax with respect to a subject image of the reference image data and including right parallax image data that is shifted to a right direction with respect to the reference image data and left parallax image data that is shifted to a left direction with respect to the reference image data, calculating an amount of disparity that corresponds to parallax between the same subject images respectively of the reference image data and the parallax image data, and generating image data by implanting, based on the amount of disparity, at least a part of primary color information for generating a color image of the reference image data to different locations within an image for the right parallax image data and the left parallax image data.