TECHNICAL FIELD

The disclosure relates to video processing.

BACKGROUND

Image “noise” is characterized by random variations in color and/or intensity not present in the image itself. In image processing, noise is introduced as unwanted electrical fluctuations that manifest themselves as perceptible (and imperceptible) flaws in the resulting image (whether static or video). Noise can be caused, for example, by poor illumination, analog-to-digital converter errors, or simply random statistical fluctuations in the number of photons detected during image exposure.

Noise in digital images can potentially be reduced using digital image processing systems. Some existing approaches to noise reduction involve performing a fast Fourier transform to convert a digital image into the frequency domain so that a high-frequency filter can be applied to the image. Such operations can require substantial computational and storage resources. The original image gets transformed into a complex function of frequency, with each frequency component requiring a complex number to represent amplitude and phase at that frequency. Due to the amount of processing time and data involved, image processing algorithms are not well suited to real-time video applications, such as live news and sports broadcasting, autonomous vehicle control systems, high-throughput film and video productions, and some consumer electronics like cell phone cameras or video baby monitors. Thus, improvements in noise reduction in both video and static image processing are needed.

SUMMARY

The invention provides noise reduction methods in an image processing pipeline. Methods of the invention are particularly well-suited to video processing. In a preferred embodiment, noise reduction comprises applying a discrete cosine transform (DCT) to pixel blocks, quantizing the resulting DCT coefficients, and performing an inverse quantization and DCT prior to further processing. Because noise reduction is performed quickly on small blocks of pixels, it can be applied to real-time video processing. In particular, noise reduction methods of the invention are applicable to high-dynamic-range video processing. Unlike standard data compression technologies, quantized DCT coefficients are not subject to compression or entropy encoding. Instead, the full and complete set of quantized DCT coefficients is simply passed back through the inverse DCT and through a video processing algorithm. Noise reduction methods of the invention can occur on a chip as a self-contained loop, with image pixels flowing in and essentially similar but “cleaned up” image pixels flowing out, and thus can be added to a digital video processing pipeline without otherwise disrupting the design and logic of the pipeline.

Pixels flow through the noise reduction step in real-time. In some embodiments, the noise reduction step operates on 8×8 blocks. There is no requirement to accumulate, or save, or operate on an entire frame of data (e.g., no need to park 1920×1080 pixels on the chip anywhere). Because the pixels flow through the noise reduction step in real-time, the noise reduction can be implemented in real-time video processing systems. The noise reduction can be performed on video or still images and can be implemented in any suitable camera such as a general purpose digital camera or a camera on a phone, or on a specialty camera such as a multi-sensor, high dynamic range (HDR) camera. Preferred embodiments that include the noise reduction step within a real-time HDR video camera are described in greater detail herein.

In some embodiments, the noise reduction step is implemented by employing frequency domain processing of image data. Accordingly, block-based DCT and quantization of image data represented in the frequency domain may be combined with an inverse of the DCT to create a live video stream that can be displayed immediately on a video screen with no lag and can also be suitable for broadcast. The block-based steps (DCT, quantization, and inverse-DCT) can easily be implemented in firmware. Thus the invention provides a noise reduction method that comprises a block-based DCT, data quantization, inverse data quantization, and an inverse DCT as a novel method of image noise reduction. This method is especially useful for treatment of HDR video data and even more useful in a pipeline for real-time image processing. In most preferred embodiments, the noise reduction step is block-based and is applied to only a very small block of an entire image at a time (as opposed to performing the operations on an entire image all at once, which would require reading the entire image into the pipeline first).

Thus, the invention comprises the application of a DCT to an image in order to effect noise reduction. Traditionally, DCT has been used for data compression and is not used in connection with image-processing or noise-reduction. Additionally, frequency domain filtering has not been used for noise reduction in a “block-based” method. The existing paradigm has been to use DCT and quantization within, for example, a JPEG compression algorithm in order to obtain a quantization matrix that is amenable to compression by Huffman encoding. It has not been suggested to reverse the DCT to arrive at an uncompressed (but quantized) image, because to do so contradicts the purpose for which the JPEG process is applied. Performing the inverse DCT (e.g., without any Huffman encoding) maintains essentially the original volume of data, but loses some of the original data. That lost data is image noise, so performing the noise reduction step as described provides better images that would be obtained otherwise and can be implemented in a real-time image processing pipeline and thus can be used in live streaming digital video.

Other noise-reduction methods, such as 3D or spatial-based noise reduction are memory- and hardware-intensive, and too slow to really be considered “real-time”. It may be found that methods of this disclosure produce better images than those other 3D noise reduction techniques. Additionally, methods of the present disclosure stand in contrast to other emerging methods of noise reduction which tend to be more- and overly-complicated likely because more computing power is available now than before. In contrast, methods of the present disclosure are simple and effective.

In certain aspects, the invention provides a method of removing noise from images. The method includes obtaining data from at least one image sensor and performing a noise reduction step. The noise reduction step comprises (and preferably consists of) performing a discrete cosine transform (DCT) on the data to get DCT values, quantizing the DCT values, and performing an inverse of the DCT on the DCT values to yield image data. The image data is provided for displaying, broadcasting, or storing as a digital image.

The method may include performing the noise reduction step on an N×N block of pixels for 2<N<16. In preferred embodiments, N=8.

Quantizing the DCT values may be done by dividing elements of the DCT values by corresponding elements in a quantization matrix and rounding the result.

The noise reduction step may be performed on a processing device such as a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC) in a video camera.

In other aspects, the invention provides a method for removing noise from a real-time stream of digital image data. The real-time stream can be from a still digital image capture device or a digital video camera that is capturing multiple frames per second of scene using one or more image sensors. Data are streamed from the one or more sensors into a novel noise reduction pipeline. The pipeline can be implemented in a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), or a discrete circuit module.

The noise reduction pipeline is configured to be seamlessly inserted into any image acquisition and processing flow. The pipeline includes an interface to receive image data in any color space and convert it into a color space amenable for noise reduction. In one implementation, the pipeline's interface coverts and RGB stream of pixel values into a YCbCr color space. The pipeline processes the image via distinct channels of noise reduction processing. Specifically, the pipeline processes the luminance and each of the chroma domains of the digital image in parallel but separately. Advantageously, the pipeline is able to perform noise reduction steps that are unique to each domain as described herein. For instance, the pipeline downsamples each of the chroma components (i.e., Cb and Cr) of the digital image such that a first portion of noise is removed from the digital image.

The pipeline then performs frequency domain processing to remove a second portion of the noise. First, the image data is converted into the frequency domain such that a block (e.g., an 8×8 block) of pixel values is represented by a matrix of frequency coefficients where each coefficient represents a weight of a particular frequency waveform's contribution to the entire block of pixel values. The pipeline identifies those frequencies that are associated with noise and then removes those frequencies. The pipeline then converts the image data as defined by the remaining set of frequencies to yield noise reduced image data. This data can then be used for displaying, broadcasting, or storing as a digital image.

In some aspects, the pipeline may include one or more filters, such as a high-pass filter, low-pass filter, or band-bass filter. Each of the filters is configured to process image data that is represented in the frequency domain such that it removes a third portion of the noise. In particular, the one or more filters can be selected to filter noise that is specific to a particular digital image acquisition device and environmental conditions at the time of image acquisition. For instance, each filter can operate with particular frequency pass-through settings. In other words, each filter is configured to remove frequencies that are either below and/or above a particular range of frequencies. For example, a low-pass filter can be specifically selected to remove noise that is known to be introduced due to specific characteristics of a particular digital image acquisition device and/or environmental conditions at the time of image acquisition. Those environmental conditions can be any one of or combination of the following factors: lighting conditions, temperature, real-time fluctuations of light and/or temperature, motion of objects in a scene to be captured and/or motion of the digital acquisition device.

In certain HDR video embodiments, the video camera comprises a plurality of image sensors that include the at least one sensor, the plurality of image sensors coupled to the processing device. Preferably, the video camera is configured to stream pixel values from each of the plurality of image sensors in a frame independent manner through a pipeline on the processing device, wherein the pipeline includes a kernel operation that identifies saturated pixel values and a merge module to merge the pixel values to produce a high-dynamic range (HDR) image. Each of the sensors may include a Bayer filter. The pipeline may perform the following steps (e.g., in the recited order): synchronize, the kernel operation, a tone-mapping operations, and the noise reduction step. Preferably, the plurality of image sensors are each positioned with respect to at least one beamsplitter and a lens of the video camera such that incoming light is split onto the plurality of image sensors so that each image sensor senses an image that is identical but for light level.

The noise reduction step may be performed in any suitable color space. For example, in some embodiments, the method includes transforming the data from the at least one image sensor from an RGB color space into a YCBCRcolor space and performing the noise reduction step in the YCBCRcolor space.

In some smartphone and tablet embodiments, the noise reduction step is performed by a mobile personal device and the at least one sensor is part of a digital camera on the mobile personal device. Optionally, providing the image data for storing as the digital image can include converting the image data into a JPEG and storing the JPEG on a computer-readable storage medium.

