1. An image processing apparatus comprising: a class tap selection unit that selects pixel values related to pixels corresponding to a plurality of color components as a class tap, from a first image formed by image signals which are output from a single-plate type pixel portion where the pixels respectively corresponding to the plurality of color components are regularly disposed on a plane; a class sorting unit that sorts out target pixels of second images as any one of a plurality of classes on the basis of a feature amount obtained from the pixel values of the plurality of color components of the class tap, each of the second images being formed by pixels of only a single color component of the plurality of color components; a coefficient reading unit that reads a coefficient stored in advance on the basis of a result of the class sorting; a prediction tap selection unit that selects pixel values related to pixels corresponding to a first color component of the plurality of color components as a prediction tap from the first image; and a product-sum operation unit that uses the pixel values of the first color component of the prediction tap as variables, and operates pixel values of the second images through a product-sum operation using the read coefficient, wherein the second image formed by only the first color component is generated from an image formed by the plurality of color components, and wherein, when the second image formed by only a second color component different from the first color component is generated from the image formed by the plurality of color components, the prediction tap is acquired from the second image formed by only the first color component.