1. Field of the Invention
The present invention relates to image restoration techniques for use with image sensors and in particular to signal processing methods for luminance estimation for use with matrix array image sensors of the type commonly incorporated in video cameras.
2. Description of Related Art
The development of low-cost video cameras is driven by the existing Closed Circuit Television (CCTV) camera marketplace and by expected new markets, such as, for example, video conferencing. A major cost of these cameras is the image sensor chip whose cost is a monotonic function of sensor area. Two major goals of sensor designers are to improve sensitivity by increasing pixel area and improve resolution by increasing pixel count. Both of these steps have the disadvantage of increasing sensor area and hence sensor cost. If the sensor area is to remain constant then, until now, image resolution could only be increased at the cost of lowered sensitivity of the sensor and sensitivity could only be increased at the cost of lowered resolution.
In one known type of video camera a colour image is formed on a single image sensor device which is spatially divided into many (typically 100,000 or more) pixels and each pixel is covered with a colour filter corresponding to one of a number of desired spectral components. These may typically be red, green and blue components or, for example, cyan, magenta and yellow. Further colour sets are also sometimes used.
Many different types of colour matrix patterns are possible. The signal outputs of all the pixels are combined by means of a reading operation at the sensor device output. Many image restoration techniques have been developed for combining the signals from individual pixels in the matrix array into a composite image representation which presents a xe2x80x9crealxe2x80x9d -looking image to the human eye. One known technique involves superimposing a highpass-filtered or xe2x80x9cedge-extractedxe2x80x9d luminance signal onto lowpass-filtered or xe2x80x9cspatially smoothedxe2x80x9d component colour signals (e.g. red, green, blue) obtained from the pixel matrix. Such image restoration techniques can enhance the perceived resolution or xe2x80x9csharpnessxe2x80x9d of the video camera image.
One problem associated with image restoration is obtaining an accurate luminance signal from the sensor. It is known that the green spectral component is closely related to the luminance response of the human eye. It has therefore been known to use the signal outputs of green pixels, or alternatively another colour of pixel whose colour is representative of luminance, to obtain the necessary luminance signal. As the chosen colour of pixels (e.g. green) only account for some of the pixels in a colour image sensor array, however, it is desirable to estimate luminance at the remaining pixel sites.
It is an object of the present invention to provide a signal processing method for accurately estimating luminance for use in matrix array colour image sensors.
U.S. Pat. No. 4,630,307 describes a signal processing method for estimating luminance for. use in matrix array sensors having luminance-sensing pixels arranged in a chequerboard pattern. The method uses signals obtained from luminance-sensing pixels to estimate luminance values at pixels in the matrix array which are not capable of sensing luminance.
In the following text the term xe2x80x9crowxe2x80x9d is defined as a horizontal row and the term xe2x80x9ccolumnxe2x80x9d is defined as a vertical row.
According to a first aspect, the present invention comprises a signal processing method for estimating luminance, for use with matrix patterned image sensors where elected sensor pixels from which luminance-representative signals are taken only occupy alternate horizontal and vertical positions in the matrix pattern, the method comprising identifying each pixel site which is at the centre of both a row and a column of five pixels and which is not the site of a said elected sensor pixel, and for each such identified site processing the signals output from the immediately adjacent four pixels in said row and column to establish both signal amplitude and pixel position and evaluating a luminance value for the pixel of the identified site from the amplitude values of the median pair of said signal amplitudes in accordance with an algorithm where decision taking is based upon the location of the pixels from which the two signals of highest amplitude emanate, the algorithm being such that when the two signals of highest amplitude emanate from pixels in the same row or column a further signal processing step is undertaken to identify, in said row and column, the signal outputs from the two respective next adjacent pixels to the pixels having the highest and lowest signal amplitudes, and the signal output from the pixel at the identified site, to enable an amplitude comparison to be made according to the result of which one of the said median pair is selected.
One advantage of the signal processing method according to the first aspect of the invention is that it utilizes signal outputs from pixels in said row and column which pixels are-next adjacent to the highest and lowest amplitude ones of said four pixels immediately adjacent the identified site.
Preferably, the selected one of said median pair is the higher amplitude one where the signal output from the next adjacent pixel to said pixel having the highest signal amplitude is closer in amplitude to the signal output from the pixel at the identified site than the signal output from the next adjacent pixel to said pixel having the lowest signal amplitude. Preferably, also, the selected one is the lower amplitude one of said median pair where the signal output from the next adjacent pixel to said pixel having the lowest signal amplitude is closer in amplitude to the signal output from the pixel at the identified site than the signal output from the next adjacent pixel to said pixel having the highest signal amplitude.
According to a second aspect the present invention comprises a signal processing method for estimating luminance for use with matrix array colour image sensors in which elected luminance-sensing pixels capable of producing signals representative of luminance are arranged in a chequerboard pattern with colour sensing pixels which are capable of producing signals representative of colour, the method comprising the steps of: ranking the signal values representative of luminance obtained from a cluster of four luminance-sensing pixels surrounding a first colour-sensing pixel, analyzing the spatial distribution of said ranked signal values so as to detect the presence of any diagonal luminance contour edge, horizontal luminance contour stripe or vertical luminance contour stripe at said first colour-sensing pixel, and determining an estimated discrete signal value representative of luminance for said first colour-sensing pixel based on one or more middle ranked signal values in a colour-enhancing fashion dependent upon, firstly, whether any diagonal luminance contour edge is detected and, secondly, whether any horizontal or vertical luminance contour stripe which is detected is interpreted as a dark stripe on a light background or a light stripe on a dark background, wherein when a horizontal or vertical luminance contour stripe is detected, the signal values representative of colour obtained from a plurality of said colour-sensing pixels in the vicinity of said cluster of four luminance-sensing pixels are compared so as to determine whether said detected horizontal or vertical luminance contour stripe is interpreted as a dark stripe on a light background or a light stripe on a dark background.
One advantage of the signal processing method according to the second aspect of the invention is that it utilizes colour signal values from pixels other than said elected luminance-sensing pixels to influence the estimated luminance signal value for said first colour-sensing pixel.
Preferably, said plurality of colour-sensing pixels in the vicinity of said cluster of four luminance-sensing pixels includes said first colour-sensing pixel, a second colour-sensing pixel disposed within the detected luminance contour stripe and a third colour-sensing pixel disposed outwith the detected luminance contour stripe. This increases the likelihood of accurately detecting and enhancing horizontal and vertical luminance contour stripes occurring in the image. The method thus enables real-time images obtained from the image sensor to be enhanced so as to improve the perceived resolution of the sensor to the human eye.
The estimated discrete signal value representative of luminance is preferably based on the mean of the middle ranking signal values when a diagonal contour edge is detected at first colour-sensing pixel.
Advantageously, said estimated discrete value is based on a corresponding one of the two middle ranking values when a stripe detected at said first colour-sensing pixel is interpreted as a dark stripe on a light background or as a light stripe on a dark background. Preferably said corresponding one of the two middle ranking signal values is the lower ranked one when a dark stripe on a light background is detected and is the higher ranked one when a light stripe on a dark background is detected.
A diagonal contour edge is preferably detected when the two highest ranking signal values are obtained from luminance-sensing pixels disposed diagonally adjacent one another.
A horizontal or vertical contour stripe is preferably detected when the two highest ranking signal values are obtained from luminance-sensing pixels disposed horizontally or vertically opposite one another.
Where a horizontal or vertical stripe is detected, the signal value obtained from said first colour-sensing pixel is preferably compared with the signal values obtained from second and third colour-sensing pixels disposed respectively vertically and horizontally opposite said first colour-sensing pixel and immediately adjacent to the luminance-sensing pixels having the highest and lowest ranked signal values. Conveniently, if the signal value of said first colour-sensing pixel is closest in value to the signal value of the pixel immediately adjacent said highest ranked signal value pixel the stripe is interpreted as a light stripe on a dark background and if the signal value of the first colour-sensing pixel is closest in value to the signal value pixel of the pixel immediately adjacent said lowest ranked signal value pixel the stripe is interpreted as a dark stripe on a light background.
Said elected luminance-sensing pixels may be pixels which are capable of producing signals representative of both luminance and colour.
The signal processing steps of the method according to the second aspect of the invention are preferably repeated for every colour-sensing pixel in the matrix array sensor which is not also an elected luminance-sensing pixel and which is at the centre of both a row and a column of five pixels. In this manner a respective estimated discrete signal value representative of luminance is determined for each such colour-sensing pixel in the matrix array.
The estimated luminance signal value obtained from each such colour-sensing pixel in the array may provide a luminance signal for a luminance channel of a camera incorporating the image sensor array. Said estimated luminance channel signal may be highpass-filtered to obtain an xe2x80x9cedge-extractedxe2x80x9d luminance signal. Alternatively, said estimated luminance signal may be lowpass-filtered and further processed to produce a signal which approximates to an xe2x80x9cedge-extractedxe2x80x9d luminance signal.
Signals representative of colour obtained from colour-sensing pixels in the array may be processed to obtain lowpass-filtered colour channel signals in a colour camera which incorporates the sensor array. Where said elected luminance-sensing pixels also sense colour, the signals from the elected luminance-sensing pixels in the array may also be processed to obtain a lowpass-filtered signal for an additional colour channel. The xe2x80x9cedge-extractedxe2x80x9d luminance signal may be superimposed on said colour channel signals to obtain full colour and luminance information for every pixel which is used to produce a contour-enhanced (i.e. improved resolution) image to the human-eye.
The elected luminance-sensing pixels are preferably green pixels. The colour-sensing pixels are preferably red and blue pixels. Alternatively, the matrix array is comprised of magenta, cyan and yellow pixels. Other colour sets are also possible.
Where the matrix array is comprised of red, green and blue pixels, the pixel signals are read into respective red, green and blue colour channels. The three colour channel signals are preferably processed to obtain lowpass-filtered, xe2x80x9cspatially-smoothedxe2x80x9d respective red, green and blue channel signals. In this way, red, green and blue colour information is obtained for every pixel in the array.
Where green pixels are the elected luminance-sensing pixels, the green signal values are preferably used in raw, uncorrected colour form.
The matrix array image sensor may, alternatively, comprise a mixed colour and monochrome array in which monochrome, luminance-sensing pixels are arranged in a chequerboard pattern with colour-sensing pixels.
According to a third aspect, the invention comprises a signal processing method for estimating luminance, for use with matrix patterned sensors where elected sensor pixels from which luminance-representative signals are taken only occupy alternate horizontal and vertical matrix element positions in the matrix pattern, the method comprising
identifying each matrix element site which is at the centre of both a row and a column of seven matrix elements and which is not the site of a said elected sensor pixel, and for each such identified site processing the signals output from the immediately adjacent four pixels in said row and column to establish both signal amplitude and pixel position and evaluating a luminance value for the matrix element of the identified site from the amplitude values of the median pair of said signal amplitudes in accordance with an algorithm where decision taking is based upon the location of the pixels from which the two signals of highest amplitude emanate, the algorithm being such that when the two signals of highest amplitude emanate from pixels in the same row or column a further signal processing step is undertaken to identify outputs in said row and column from the two pixels respectively next adjacent to the pixels having the highest and lowest signal amplitudes, and to identify the mean of the signal outputs from the eight pixels diagonally adjacent to, and arranged around, said four pixels in said row and column, to enable an amplitude comparison to be made according to the result of which one of the said median pair is selected.
According to a fourth aspect the invention comprises a signal processing method for estimating luminance for use in a matrix array monochrome image sensor in which elected luminance-sensing pixels capable of producing signals representative of luminance are arranged in a chequerboard pattern with non-luminance-representative matrix array elements which do not produce signals, the method comprising the steps of: ranking the signal values representative of luminance obtained from a cluster of four luminance-sensing pixels surrounding a first non-luminance-representative matrix element, analyzing the spatial distribution of said ranked signal values so as to detect the presence of any diagonal luminance contour edge, horizontal luminance contour stripe or vertical luminance contour stripe at said first non-luminance-representative matrix element, and determining an estimated discrete signal value representative of luminance for said non-luminance-representative matrix element based on one or more middle ranked signal values in a contour-enhancing fashion dependent upon, firstly, whether any diagonal luminance contour edge is detected and, secondly, whether any horizontal or vertical luminance contour stripe which is detected is interpreted as a dark stripe on a light background or a light stripe on a dark background, wherein, when a horizontal or vertical luminance contour stripe is detected the signal values representative of luminance obtained from at least second and third luminance-sensing pixels disposed respectively within and outwith the detected luminance contour stripe are compared with the mean of the signal values obtained from a plurality of further luminance-sensing pixels in the vicinity of said cluster of four luminance-sensing pixels so as to determine whether said horizontal or vertical stripe is interpreted as a dark stripe on a light background or a light stripe on a dark background.
An advantage of this fourth method is that the signal value from at least one pixel disposed within the detected luminance contour stripe is compared with signal values from pixels disposed outwith said stripe so as to determine whether the stripe should be interpreted as a dark stripe on a light background or a light stripe on a dark background.
Preferably said plurality of further luminance-sensing pixels comprise eight pixels, each pixel being disposed diagonally adjacent one of said cluster of four luminance-sensing pixels surrounding said first non-luminance representative matrix element.
Advantageously said second and third luminance-sensing pixels are disposed horizontally and vertically opposite said first non-luminance representative matrix element and adjacent to the two of the cluster of four luminance-sensing pixels which have the highest and lowest ranked signal values.
The steps of the signal processing method according to the fourth aspect of the invention may be repeated for each non-luminance representative matrix element in the matrix array which is at the centre of both a row and a column containing seven matrix elements.
According to a fifth aspect, the invention provides a camera comprising a matrix array image sensor and signal processing means for carrying out the signal processing method according to any of the above-mentioned first, second, third and fourth aspects of the invention. The image sensor may be a Charge Coupled Device (CCD) or CMOS array.