1. An image processing apparatus which inputs an image obtained by sensing an optical image by an imaging system, compensates an omitted color component, and outputs a color image, comprising: a weight setting unit which sets a weight Wk in each direction Sk ( k is an integer from 1 to n ) of a plurality of directions S 1 , S 2 ,. .. , Sn ( n is an integer of not less than 1) starting from a pixel of interest in a predetermined neighborhood of the pixel of interest; an average value calculation unit which calculates a weighted average of pixel values Vk of pixels having a specific color component and located on each direction Sk ( k is an integer from 1 to n ) near the predetermined size by using weights W 1 to Wn of the pixel values V 1 to Vn; and a restoration unit which causes the average calculation unit to calculate weighted averages of a plurality of types of color components and restores a value of an omitted color component of the pixel of interest on the basis of the weighted average result and the pixel value of the pixel of interest.