1. A signal processing apparatus for performing signal processing on pixel signals output from an imaging element, the imaging element having a plurality of pixels for photoelectrically converting light from a subject, the pixel signal corresponding to respective pixels, the signal processing apparatus comprising: a determination section for determining brightness or saturation at one or a plurality of pixels for each pixel; and an interpolation processing section for performing interpolation processing of interpolating a pixel signal of a target pixel using pixel signals of peripheral pixels positioned in the periphery of the target pixel based on the determined brightness or saturation of the pixel, wherein the interpolation processing section performs adaptive interpolation processing of selecting, for each pixel, interpolation processing suitable for the determined brightness or saturation of the target pixel, from among at least two or more types of interpolation processing, and interpolating a pixel signal of the target pixel by the selected interpolation processing.