Vision systems for detecting, identifying, and tracking targets must acquire and process large volumes of video data in real time. Most applications requiring these tasks, however, are characterized by imagery where targets and other objects of interest occupy only a small fraction of the total scene. Hence, traditional imaging systems that employ uniform and constant spatial resolution throughout the entire field of view acquire much irrelevant information and thus burden valuable data processing and communication resources in the system. As a result, such systems are slow and do not operate at their full potential due to a flood of unimportant video information. Building vision systems with multiple cameras, several processors, high-bandwidth communication links, and specialized hardware will increase performance, but results in equipment that is complex, expensive, large, heavy, high power consuming, and prone to failures. These constraints make constant resolution vision systems difficult or impossible to use where space, speed of response, and reliability are critical considerations, for instance, in defense applications.
Humans and other vertebrates have foveal vision that allows them to concurrently perform several tasks: survey a wide field of view at a low resolution for situational awareness and detection of features or targets of interest; track moving targets with great accuracy; scan at high resolution these multiple targets of interest; and communicate over channels with limited bandwidth (neurons) the information of interest to the computer (brain). Because high-resolution imaging is limited to the fovea, which is fixed in the center of the retina, the tracking of targets involves movement of the eyes and the head.
The vision system of the present invention operates according to a concept termed Dynamically Reconfigurable Vision, or DRV, which is inspired by the biologically proven concept of foveal vision, but is further extended for practicality in a machine vision implementation. DRV is adaptive image sensing driven by a computer or human operator's response to changing scenery. One goal of this system is to reduce the amount of irrelevant video information sensed and thus achieve more effective bandwidth and computational resource utilization than traditional vision systems. This is achieved by a system controlled by a computer or microprocessor and a photodetector array for imaging a frame of a scene through multiple independently controllable, time-correlated, overlapping photodetector array windows, which are dynamically reconfigurable in real time, and where such windows in a frame are capable of having varying resolution.
Several vision systems or imaging photodetector arrays have been developed which do not enable DRV. U.S. Pat. No. 5,541,654 describes a focal plane array imaging device having sensor windowing, variable integration time, and snapshot mode. U.S. Pat. No. 5,990,469 describes a control system for an image array sensors providing multiple windows. U.S. Pat. No. 5,973,311 provides a pixel array imager with a high and low resolution mode. U.S. Pat. No. 5,949,483 describes a multiresolution image sensor that contains an electronic shutter, can form multiple pixel windows, and possesses a multiresolution imaging circuit for averaging pixels into groups of superpixels. U.S. Pat. No. 5,493,335 provides a color CCD sensor video camera that is adapted for processing images of different resolution to provide a user selectable image size. U.S. Pat. No. 5,926,208 describes a reconfigurable camera with low-bandwidth transmission capability using a conventional image sensor, processor, and memory that can hold several image compression algorithms. These vision systems and sensors do not provide a reduction in the amount of irrelevant information sensed through the use of a computer-controlled system for imaging a scene through multiple independently controllable, time-correlated, overlapping sensor windows.
In a publication by NASA's Jet Propulsion Laboratory, a photodetector array circuitry is described that performs snapshot imaging and pixel averaging, see F B. Pain and X. Zheng, Active Pixel Sensor with Photosites in Substrates, NASA Tech Brief, vol. 23, no. 10, October 1999 (from JPL New Technology Report NPO-20534). This circuitry requires the host computer to send control signals to the sensor for the extraction of the signal from each superpixel, and like the other prior art system, do not provide imaging a scene through a multiple independently controllable, time-correlated, overlapping sensor windows.
In U.S. Pat. No. 5,262,871, a multiresolution image sensor is described that inputs data representing a superpixel to a computer that controls the size of that superpixel. The computer extracts video data from the photodetector array one superpixel at a time. This is time consuming and reduces bandwidth by increasing the amount of interaction between the computer and the camera. Furthermore, the values reporting the level of illumination of the superpixels generated by the system are a function of pixel size in which this value is equal to the sum of the comprising pixels. The superpixel values are normalized so that the image processing algorithms do not erroneously interpret the superpixel to represent a bright scene region. This operation requires extra time and system memory, thus slowing down the image processing. Larger pixel values also require a wider dynamic range in the video communication circuits. In addition, the photodetector array described in U.S. Pat. No. 5,262,871 does not operate in a snapshot mode, so that the exposure time can be different for different pixels after the system has been reconfigured, requiring the computer to carry out additional normalization of pixel values in order to avoid a mistake in interpreting the pixel values in the context of the image. Moreover, because the pixels in this patent are exposed at different times, any motion in the field of view (due to camera and/or target motion) will introduce artifacts, such as target warping, artifacts that reduce the accuracy of the target classification. Overexposed pixels in an image will appear brighter than the properly exposed or underexposed pixels in the same image.
Other developed vision systems that use fixed geometry multiresolution have a two-dimensional photodetector array that has small size pixels in the center and larger size pixels in the periphery, closely simulating the anatomical structure of the retina. Such arrays are not reconfigurable, do not operate in a closed loop fashion with the vision processor, and require a pointing mechanism for gazing. Mechanical pointing suffers from instabilities and is slow. Further, because the topology of the array in a fixed geometry system cannot be changed on demand, some relevant regions of the scene may not be resolved adequately, as a result sacrificing system reliability and usefulness, while the irrelevant regions may be resolved too finely, hence reducing the efficiency of system resource utilization.
In still other systems, referred to as pyramidal machine vision systems, targets are detected using low resolution; these targets are then scanned using localized high-resolution windows. However, the pixels that form the high resolution windows are not combined into superpixels directly in the pixel array, but the information from individual pixels is combined in the computer so as to mimic superpixels. These techniques require the processing of video from a uniform resolution camera to generate the pyramid data structure. As a result, the communications bandwidth and the video processing resources are not used any more efficiently than in a conventional uniform resolution camera.