1. Field of the Invention
The present invention relates to environment recognition systems, and more particularly, to an environment recognition system that recognizes a surrounding environment by detecting objects in a taken image.
2. Description of the Related Art
In general, in order to measure the distance to an object existing in a surrounding environment with a stereo camera, a pair of images are taken by a pair of right and left cameras that are mounted at the same height, and one of the taken images used for reference (hereinafter referred to as a reference image T0) is compared with the other image (hereinafter referred to as a comparative image Tc). By comparison, a difference between corresponding positions of the same object in the images, that is, a parallax is calculated, and the distance to the object is calculated from the parallax. The positions in the reference image and the comparative image where an image of the same object is included are typically located by stereo matching (for example, see Japanese Unexamined Patent Application Publication Nos. 10-283461 and 10-283477).
In stereo matching, as shown in FIG. 17, a reference image T0 is divided into small regions (hereinafter referred to as reference pixel blocks PB0) each defined by a predetermined number of pixels, such as 3 by 3 pixels or 4 by 4 pixels. An epipolar line EPL is set in the vertical position in the comparative image Tc corresponding to each reference pixel block PB0, and a brightness pattern of the reference pixel block PB0 is compared with a brightness pattern of a comparative pixel block PBc that exists on the epipolar line EPL and that has the same shape as that of the reference pixel block PB0.
In this case, for example, a SAD (Sum of Absolute Difference) value is calculated as a difference in the brightness pattern according to the following Expression (1):
                    SAD        =                              ∑                          s              ,              t                                ⁢                                                                p                ⁢                                                                  ⁢                1                ⁢                st                            -                              p                ⁢                                                                  ⁢                2                ⁢                st                                                                                    (        1        )            where p1st represents the brightness of the pixel in the reference pixel block PB0 in the reference image T0, and p2st represents the brightness of the pixel in the comparative pixel block PBc in the comparative image Tc. Of SAD values that are less than or equal to a preset threshold value, a comparative pixel block PBc that provides the smallest SAD value is specified as a comparative pixel block in the comparative image Tc that includes an image of the same object as that included in the reference pixel block PB0.
A parallax dp between the comparative pixel block PBc specified in the comparative image Tc and the original reference pixel block PB0 in the reference image T0 is calculated, and a distance Z to the object at the reference pixel block PB0 is calculated on the basis of the parallax dp according to the principle of triangulation. On the basis of the calculated distance Z, the object is detected from the surrounding environment.
It is confirmed that this object detection method that calculates the parallax dp by stereo matching of the reference image T0 and the comparative image Tc and calculates the distance Z to the object functions without any trouble in a normal image taking environment and can effectively detect the object from the surrounding environment, as disclosed in the above-described publications.
However, for example, when the stereo camera is placed in a backlit environment, a reference image T0 shown in FIG. 18A is bright because backlight enters the image, while a comparative image Tc shown in FIG. 18B taken in the same scene is totally darker than the reference image T0 because backlight is blocked by a building or the like and much backlight does not enter the image.
When the brightness balance between a pair of cameras is thus disturbed, the difference between the brightness p1st of the pixel in the reference pixel block PB0 in the reference image T0 and the brightness p2st of the pixel in the comparative pixel block PBc in the comparative image Tc in Expression (1) described above generally increases. Therefore, the calculated SAD value increases above the above-described threshold value. In this case, the parallax dp is not effectively calculated, and the number of reference pixel blocks PB0 to be rejected increases.
For this reason, in data image (hereinafter referred to as a distance image Tz) formed by assigning calculated parallaxes dp to the pixel blocks PB0 in the reference image T0, little data on effective parallaxes dp is provided, as shown in FIG. 19. In this case, it is sometimes difficult to detect objects, and reliability of the object detection result decreases. In the worst case, little data on effective parallaxes dp is included in the obtained distance image Tz, and it is completely impossible to detect objects.
In this case, for example, a reference edge image TE0 shown in FIG. 20A is formed by calculating differences in the brightness p1ij between pixels belonging to the reference image T0 shown in FIG. 18A and pixels adjacent on the right or left side. Similarly, a comparative edge image TEc shown in FIG. 20B is formed from the comparative image Tc shown in FIG. 18B. The edge image TE0 and the comparative edge image TEc can be subjected to stereo matching.
By subjecting the reference edge image TE0 and the comparative edge image TEc thus formed to stereo matching, a distance image TEz in which a relatively large amount of data on effective parallaxes dp are included (hereinafter a distance image based on the edge images is referred to as an edge distance image) is obtained, as shown in FIG. 21. In this case, objects can sometimes be effectively detected even when it is difficult to effectively detect the objects by directly conducting stereo matching on the original reference and comparative images T0 and Tc. FIGS. 20A, 20B, and 21 illustrate parts of the reference edge image TE0, the comparative edge image TEc, and the edge distance image TEz.
However, edge processing has a problem in that much information is lost when obtaining the differences in brightness between the adjacent pixels. That is, when the difference in brightness is 30 in 256 brightness levels, it is unclear whether the difference of 30 indicates a difference between 50 and 80 or between 200 and 230. Moreover, this amplifies noise components in the reference image T0 and the comparative image Tc.
Further, since the difference in brightness only in a width corresponding to one or several pixels is found, information about low-frequency components in the frequency components in the reference image T0 and the comparative image Tc is lost. Therefore, mismatching easily occurs. Further, it is difficult to obtain effective information, for example, about a wall that is not characteristic in structure and pattern and an asphalt road surface.
In this way, although edge processing is effective, as described above, it should be avoided to always detect objects only on the basis of an edge distance image TEz that is formed by stereo matching of a reference edge image TE0 and a comparative edge image TEc obtained by subjecting a reference image T0 and a comparative image Tc to edge processing.
When objects are detected only on the basis of the distance image Tz obtained from the reference image T0 and the comparative image Tc, objects can be effectively and accurately detected from the surrounding environment in a normal image taking condition, as described above. However, it is undeniable that object detection is difficult in the above-described special condition.