1. Imaging apparatus, comprising: a mosaic image sensor, which is configured to generate a stream of input pixel values belonging to a plurality of input sub-images, each sub-image responsive to light of a different, respective color that is incident on the mosaic image sensor; an image restoration circuit, which is coupled to receive and digitally filter the input pixel values in each of the input sub-images so as to generate a corresponding plurality of enhanced output sub-images; and an image signal processor (ISP), which is coupled to receive and combine the plurality of the output sub-images in order to generate a color video output image.