1. An image processing apparatus for generating a signal value of a pixel of interest in an image, which is captured by an image sensor having a color filter, from signal values of a plurality of peripheral pixels of the pixel of interest, comprising: a determination unit adapted to output a determination signal indicating a degree to which the pixel of interest is of an achromatic color based on a chroma signal corresponding to the pixel of interest; an achromatic color direction determination unit adapted to determine a direction, in which a pixel highly correlated with the signal value of the pixel of interest exists, from the signal values of a plurality of pixels of a different color from the pixel of interest among the plurality of peripheral pixels of the pixel of interest; a first correction value generation unit adapted to generate the signal value of the pixel of interest by using a signal value of a pixel that exists in the direction determined by the achromatic color direction determination unit among the plurality of peripheral pixels of the pixel of interest; a chromatic color direction determination unit adapted to determine a direction, in which a pixel highly correlated with the signal value of the pixel of interest exists, from signal values of a plurality of pixels of the same color as the pixel of interest and signal values of a plurality of pixels of a different color from the pixel of interest among the plurality of peripheral pixels of the pixel of interest; a second correction value generation unit adapted to generate the signal value of the pixel of interest by using a signal value of a pixel that exists in the direction determined by the chromatic color direction determination unit among the plurality of peripheral pixels of the pixel of interest; and a weighted addition unit adapted to perform weighted addition of the signal value generated by the first correction value generation unit and the signal value generated by the second correction value generation unit such that a weight of the signal value generated by the first correction value generation unit is larger and a weight of the signal value generated by the second correction value generation unit is smaller as the degree to which the pixel of interest is of an achromatic color indicated by the determination signal is higher, and calculating a final signal value of the pixel of interest.