TECHNICAL FIELD

This disclosure relates to real-time production and broadcast of video for display on Low Dynamic Range and High Dynamic Range display devices.

BACKGROUND

The human visual system is capable of identifying and processing visual features with High Dynamic Range. For example, real-world scenes with contrast ratios of 1,000,000:1 or greater can be accurately processed by the human visual cortex. However, most image acquisition devices are only capable of reproducing or capturing Low Dynamic Range, resulting in a loss of image accuracy. The problem is ever more significant in video imaging.

There are examples of creating High Dynamic Range images by post processing images from multiple sensors, each of different exposures. The resulting “blended” image is intended to capture a broader dynamic range than would be possible from a single sensor without a post-processing operation. However, the required post-processing is time-intensive and produces an HDR video signal that requires an expensive HDR playback device and display device.

The requirement for intensive post-processing and uncommon equipment leads to HDR video systems that are not well suited to how many people watch television. For example, live broadcasts are an important part of television. Many people follow the news or their favorite sports teams by watching events live, whereas existing HDR video production requires filming followed by post-production processing and then display. Additionally, some HDR video systems are not backwards-compatible with the existing installed base of users who have low dynamic range (LDR) televisions.

SUMMARY

The invention provides methods for broadcasting video in a dual HDR/LDR format such that the video can be displayed in real time by both LDR and HDR display devices. Methods and devices of the invention process streams of pixels from multiple sensors in a frame-independent manner to produce High Dynamic Range (HDR) video signals in real time. The real-time aspect of the invention is accomplished by analyzing streams of pixels from the various sensors without reference to the frame to which those pixels belong. Thus, the frame-independent nature of the invention means that there is no need to wait for an entire frame of data to be read from a sensor before processing pixel data. The result is an HDR video that has a dynamic range greater than the range that can be obtained using a single image sensor, typically 8 bits in depth. That HDR video signal is then tone-mapped to produce an LDR video signal, the LDR signal is subtracted from the HDR signal to calculate a residual signal, and the LDR signal and the residual signal are merged into a combined signal that is broadcast via a communications network. The combined signal is backwards compatible with existing LDR display devices because those devices simply use the LDR signal of the combined signal. The combined signal is also displayable by HDR displays by merging the residuals back with the LDR signal to re-create the HDR video signal. The HDR video starts with the multiple sensors that are exposed to identical scenes with different light levels and that each produce an ordered stream of pixel values. The pixel values are processed in real-time and independent of frame in which they will reside. The video processing pipeline used to produce HDR images includes kernel and merge operations that identify saturated pixel values and merge streams of pixel values. The merging operation includes replacing saturated pixel values with corresponding pixel values originating from a different sensor. The merged signal is an HDR signal and is tone mapped to produce the LDR signal and the residuals. The residuals are combined with the LDR signal into a combined signal that is broadcast for LDR and HDR display by LDR and HDR devices, respectively.

The dual LDR/HDR video broadcast is provided in real time by streaming the pixel values through the kernel and merge operations in the pipeline. Preferably, a synchronization module in the pipeline synchronizes the streams of pixel values arriving from the sensors. This means that when, for example, the 60thpixel from a first sensor enters the kernel operation, the 60thpixel from each of the other sensors is also simultaneously entering the kernel operation. As a result, pixel values from corresponding pixels on different sensors flow through the pipeline synchronously. This allows two things. First, the synchronization module can correct small phase discrepancies in data arrival times to the system from multiple sensors. Second, the synchronization allows the kernel operation to consider—for a given pixel value from a specific pixel on one of the image sensors—values from the neighborhood of surrounding pixels on that sensor and also consider values from a corresponding neighborhood of pixels on another of the image sensors. This allows the kernel operation to create an estimated value for a saturated pixel from one sensor based on a pattern of values from the surrounding neighborhood on the same or another sensor.

The pipeline may optionally correct for differences in spectral characteristics of each of the multiple sensors. Optical components such as beamsplitters, lenses, or filters—even if purported to be spectrally neutral—may have slight wavelength-dependent differences in the amounts of light transmitted. That is, each image sensor may be said to have its own “color correction space” whereby images from that sensor need to be corrected out of that color correction space to true color. The optical system can be calibrated (e.g., by taking a picture of a calibration card) and a color correction matrix can be determined and stored for each image sensor. The HDR video pipeline can then perform the counter-intuitive step of adjusting the pixel values from one sensor toward the color correction space of another sensor—which may in some cases involve nudging the colors away from true color. This may be accomplished by multiplying a vector of RGB values from the one sensor by the inverse of the color correction matrix of the other sensor. After this color correction to the second sensor, the streams are merged, and the resulting HDR video signal is color corrected to true color (e.g., by multiplying the RGB vectors by the applicable color correction matrix). This operation accounts for spectral differences of each image sensor.

A preferred pipeline includes other processing modules as described in detail in the Detailed Description of the invention below.

In certain aspects, the invention provides methods for streaming video for high dynamic range (HDR) and low dynamic range (LDR) display. The methods include receiving pixel values from multiple image sensors on an HDR video camera and producing an HDR signal, processing the HDR signal to produces an output signal comprising an LDR signal and a residual signal, and transmitting the output signal for HDR and LDR display with less than one frame of delay between the receiving and the transmitting. Processing the HDR signal may include tone-mapping the HDR video to produce the LDR signal and subtracting the LDR signal from the HDR signal to produce the residual signal. The LDR video may be compressed, e.g., using an MPEG encoder. The output signal is displayable as an 8-bit video by an LDR display, and displayable as HDR video by an HDR display.

In certain embodiments, the HDR signal is produced from the pixel values by a pipeline that includes a kernel operation that identifies saturated pixel values and a merge operation that merges the pixel values to produce the HDR signal. Preferably, the multiple image sensors all capture images simultaneously through a single lens. The method may include receiving incoming light through the lens and splitting the light via at least one beamsplitter onto the multiple image sensors, wherein at least 95% of the light gathered by the lens is captured by the multiple image sensors.

The multiple image sensors may include at least a high exposure (HE) sensor and a middle exposure (ME) sensor, and merging the sequences may include using HE pixel values that are not saturated and ME pixel values corresponding to the saturated pixel values. The multiple image sensors may further include a low-exposure (LE) sensor. In preferred embodiments, the multiple image sensors capture images that are optically identical except for light level.

The method operates in real-time and may include beginning to merge portions of the pixel values while still streaming later-arriving pixel values through the kernel operation.

The pipeline may be provided by a processing device (e.g., a field-programmable gate array or an application-specific integrated circuit) and each of the image sensors may include a color filter array. In some embodiments, the methods include demosaicing the HDR signal after the merging.

