1. Imaging apparatus, comprising: a mosaic image sensor, which is configured to generate a stream of input pixel values belonging to a plurality of input sub-images, each input sub-image responsive to light of a different, respective color that is incident on the mosaic image sensor, wherein the input sub-images have an input blur caused by a point spread function (PSF) of a set of objective optics; an image restoration circuit, which is coupled to receive and digitally filter the input pixel values in each of the input sub-images so as to generate a corresponding plurality of enhanced output sub-images, the image restoration circuit comprising a deconvolution filter (DCF) having a filter kernel, the DCF undoing effects of the PSF to cause the output sub-images to have an output blur that is less than the input blur; and an image signal processor (ISP), which is coupled to receive and process the plurality of output sub-images in order to generate a color output image.