TECHNICAL FIELD

The present disclosure relates to ambient environment recognition which is performed with a plurality of cameras installed in a vehicle.

BACKGROUND

In recent years, a large number of technologies have been developed in which a plurality of cameras installed in a vehicle are used so as to perform ambient environment recognition on the vehicle and in which thus a brake is automatically applied to the vehicle depending on the situation, with the result that the driving assistance and the automated driving of vehicles proceed. In the technologies described above, since not only the present vehicle but also people therearound, other vehicles and the like are moved, with consideration given to feedback from the ambient environment recognition to control on alarms, braking and the like, images shot with the cameras need to be processed in real time. However, when a processor is provided in each of the cameras, the cost of the vehicle is significantly increased.

In order to solve such a problem, Patent Document 1 discloses that in a configuration in which a plurality of stereo cameras are individually connected to I/Os and in which a CPU (image processing IC) uses a RAM so as to perform, as necessary, image processing on image data shot, a situation judgment/image selection portion determines which one of the images shot with the stereo cameras is output to an image processing portion in accordance with a support mode, and that thus the image processing portion (a preprocessing portion and the subsequent portions) performs the image processing. In this way, it is possible to avoid the need to use a computation device whose processing capability is high and the like, and thus it is possible to reduce the cost.

Likewise, Patent Document 2 discloses that a CPU (computation portion) capable of performing front recognition processing for the image information of a front camera input to an input portion and rear recognition processing for the image information of a rear camera selects and performs processing executed as the front recognition processing and the rear recognition processing, respectively, based on the state information of the present vehicle (information indicating a forward movement and a backward movement and a travelling state). In this way, as compared with a case where CPUs are separately provided for the front recognition processing and the rear recognition processing, it is possible to simply its configuration.

Moreover, Patent Document 3 discloses that part of captured images obtained by imaging the ambient environment of a vehicle is selected as a target region according to conditions of the vehicle including the travelling environment and the travelling state of the vehicle, that image processing is performed on the selected target region and thus as compared with a case where image processing is performed on the entire captured images, a processing burden can be reduced.

CITATION LIST

Patent Literature

Patent Document 1: JP2013-93013APatent Document 2: JP2018-95067APatent Document 3: JP2015-210584A

SUMMARY

However, in Patent Document 1, in order for the image selection to be performed with the situation judgment/image selection portion, all the image data generated by shooting with the individual stereo cameras is temporarily input to the CPU. Hence, the CPU needs to perform processing for capturing all the image data in a stage preceding the image selection described above, and an operating time allocated to the image processing is reduced accordingly. Likewise, in Patent Document 2, the CPU receives inputs of all the image information of the front camera and the rear camera, performs correction processing (distortion correction) on all the image information which is input and thereafter performs the selection described above. Hence, the CPU needs to perform, for example, processing for capturing all the image data and processing on image information which is not selected, and thus an operating time allocated to the image processing is reduced accordingly. As described above, when the amount of processing performed by a processor for the image processing is increased, in order to process shot images in real time, the processor is required to have higher processing capability, with the result that it is likely that the cost of the vehicle cannot be sufficiently reduced.

In view of the foregoing situation, an object of at least one embodiment of the present invention is to provide a vehicle image processing device which can more rapidly perform, while reducing the cost of a vehicle, image processing on images shot with a plurality of cameras.

(1) A vehicle image processing device according to at least one embodiment of the present invention includes: a plurality of buffers configured to accumulate pieces of image data input individually and sequentially from a plurality of cameras installed in a vehicle so as to associate the pieces of image data with the cameras; a processor configured to select the buffer based on the state information of the vehicle and acquire the piece of image data from the selected buffer so as to perform image processing thereon; a signal line for transferring the pieces of image data in the buffers to the processor; and a transfer controller configured to output the piece of image data in the buffer required from the processor to the signal line.

In the configuration of (1) described above, the processor for the image processing does not perform any processing on the image data until the image data is transferred with the transfer controller from the buffer in which the image data shot with each of the cameras is stored, and performs the image processing on only the image data acquired by the transfer. In this way, it is possible to reduce the burden of the processor, and thus it is possible to allocate a larger amount of operating time to the image processing. Moreover, for example, the processor determines, from the image data of the individual cameras accumulated in the buffers, based on the state information of the vehicle such as the vehicle speed and shift position, the buffers which need to be set to the target for the image processing so as to narrow the cameras set to the target for the image processing to at least part thereof, and thus it is possible to reduce the number of pieces of image data which needs to be processed per unit time.

As described above, the processor for the image processing sets the image data of the cameras determined based on the state information of the vehicle to the target for the image processing, and does not perform processing until the image data set to the target is acquired, with the result that it is possible to more rapidly (more quickly) perform environment recognition (the detection of a person and an article) around the vehicle through the cameras. Hence, it is possible to avoid the provision of the processor for each of the cameras and the use of an expensive processor having more excellent processing capability, and thus it is possible to reduce the cost of the vehicle.

(2) In some embodiments, in the configuration of (1) described above, the processor determines, based on the state information, at least one buffer of the buffers as a selection target, and sets a selection frequency of the at least one buffer included in the selection target higher than selection frequencies of the other buffers so as to sequentially select the at least one buffer included in the selection target.

In the configuration of (2) described above, the processor determines, from the cameras, one or more cameras (one or more buffers) which are set to the target for the image processing, sequentially acquires the image data from the one or more buffers corresponding to the one or more cameras determined so as to perform the image processing and thereby performs the image processing on a predetermined number of pieces of image data per unit time. As described above, the cameras which are set to the target for the image processing are narrowed to at least part of the cameras, and thus it is possible to reduce the number of pieces of image data which need to be processed per unit time, with the result that it is possible to perform environment recognition (detection of movement of a person and an article) around the vehicle through the cameras in a satisfactory real-time manner.

(3) In some embodiments, in the configuration of (1) and (2) described above, the vehicle image processing device further includes: a capture unit having the buffers and the transfer controller provided therein; a work buffer configured to hold the piece of image data output by the transfer controller to the signal line and subjected to the image processing performed by the processor; and a processor unit having the processor and the work buffer provided therein, and the signal line is configured to be able to connect the buffers in the capture unit and the work buffer in the processor unit to each other.

In the configuration of (3) described above, the pieces of image data of the individual cameras managed with the buffers are passed through the signal line for connecting the capture unit and the processor unit, are transferred to the work buffer provided in the processor unit and are subjected to the image processing in the processor. The management and the transfer processing of the buffers in the capture unit is separated with the image processing in the processor unit in terms of hardware in this way, and thus as described above, it is possible to perform environment recognition (the detection of a person and an article) around the vehicle through the cameras in a satisfactory real-time manner.

(4) In some embodiments, in the configuration of (1) to (3) described above, the state information includes a vehicle speed, and the processor performs the selection of the buffer based on the comparison of the vehicle speed and threshold values.

In general, there is an upper limit on the number of pieces of image data on which a processor can perform image processing per unit time. Hence, when the pieces of image data of the cameras are sequentially processed, as the number of cameras (buffers) is increased, the number of pieces of image data per camera subjected to image processing at a unit time is decreased. As this number is decreased, a longer delay occurs in the image processing on the pieces of image data which are sequentially fed from each of the cameras, and thus a time lag in the environment recognition through the image processing of the cameras is increased.

In the configuration of (4) described above, based on the comparison of the vehicle speed and the threshold values, the processor makes a selection such as by determining the buffers which are set to the target for the image processing. As the vehicle speed is increased, the time lag in the environment recognition through the image processing is more likely to be fatal whereas as the vehicle speed is decreased, the time lag is more likely to serve as an allowable range. Hence, as the vehicle speed is decreased, even when a larger number of cameras are set to the target for the image processing, it is possible to appropriately perform driving assistance for the vehicle. Therefore, the cameras which are set to the target for the image processing are selected by the relationship with the vehicle speed, and thus even the processor whose processing capability is relatively low can appropriately perform the driving assistance for the vehicle through the environment recognition using the cameras in a satisfactory real-time manner.

(5) In some embodiments, in the configuration of (4) described above, the cameras include one or more of the cameras for shooting the surrounding of the vehicle, the threshold values include a first threshold value and when the vehicle speed is less than the first threshold value, the processor selects the buffer in which the piece of image data shot with the one or more of the cameras for shooting the entire surrounding of the vehicle is accumulated.

For example, when the vehicle is stopped before being moved or in a state (state where travelling is started) where the vehicle is moved at a very low speed due to a creep phenomenon or the like, the vehicle can be moved to various positions depending on a steering angle. Hence, with the near-field cameras which have a relatively wide angle of view so as to be able to appropriately shoot a near field or the like, it is necessary to monitor the entire surrounding of the present vehicle (at least the entire orientation in a horizontal direction).

In the configuration of (5) described above, when the state (state where travelling is started) as described above is detected with the first threshold value, the processor sets images shot with one or more of the cameras (such as one or more of the near-field cameras) necessary for obtaining images around the vehicle (the entire orientation) to the target for the image processing. In this way, it is possible to appropriately perform driving assistance corresponding to conditions of the vehicle through monitoring with the cameras in a satisfactory real-time manner.

For example, when the cameras such as a total of four cameras for shooting the front, the back, the left and the right of the vehicle are installed in order to obtain the entire surrounding, the number of cameras set to the target for the image processing is more likely to be increased. Hence, when the images of the cameras are sequentially switched one by one, and then the image processing is performed, as the number of cameras is increased, the processing cycle of the image data in the individual cameras (individual buffers) is prolonged. However, since the vehicle is moved at a very low speed, as long as the image processing is allowed (is not problematic) as the driving assistance for the vehicle, the image processing can be performed on the images shot with the individual cameras.

(6) In some embodiments, in the configuration of (4) and (5) described above, the cameras include a camera for shooting an area ahead of the vehicle and a camera for shooting an area behind the vehicle, the state information further includes a shift position of a gear of the vehicle, the threshold values include the first threshold value and a second threshold value which is more than the first threshold value and when the vehicle speed is equal to or more than the first threshold value and less than the second threshold value, and the shift position indicates a forward movement, the processor selects only the buffer in which the piece of image data shot with the camera for shooting the area ahead of the vehicle is accumulated whereas when the shift position indicates a backward movement, the processor selects only the buffer in which the piece of image data shot with the camera for shooting the area behind the vehicle is accumulated.

For example, in a state (low-speed travelling state) where the vehicle can be rapidly moved in a lateral direction with respect to the direction of travelling depending on the steering angle, it is necessary to use the near-field cameras (described above) and the like so as to extensively monitor an area in the direction of travelling of the present vehicle.

In the configuration of (6) described above, when the state (low-speed travelling state) as described above is detected with the first threshold value and the second threshold value, the processor sets images shot with the cameras for shooting images in the direction of travelling of the vehicle to the target for the image processing. In this way, it is possible to appropriately perform driving assistance corresponding to conditions of the vehicle through monitoring with the cameras in a satisfactory real-time manner.

(7) In some embodiments, in the configuration of (4) to (6) described above, the cameras include a far-field camera which shoots a far-field area in the direction of travelling of the vehicle and a near-field camera which shoots a near-field area in the direction of travelling of the vehicle and which has a wider angle of view than the far-field camera, the threshold values include the second threshold value and when the vehicle speed is equal to or more than the second threshold value, the processor selects the buffer in which the piece of image data shot with the far-field camera is accumulated whereas when the vehicle speed is less than the second threshold value, the processor selects the buffer in which the piece of image data shot with the near-field camera is accumulated.

In a travelling state where the vehicle speed is relatively so high that the vehicle cannot be rapidly turned in direction by steering, the time allowed for feedback (for example, automatic braking in the vehicle or a warning for the driver) from the environment recognition (the detection of a person and an article) through the image processing to the driver of the vehicle or the like is more shortened. Hence, it is necessary to monitor a distant area from the vehicle is monitored so as to detect a target (a person and an article) in a distant area in an earlier stage.

In the configuration of (7) described above, when the travelling state as described above is detected with the second threshold value, the processor sets images shot with the far-field cameras for shooting a distant area in the direction of travelling of the vehicle to the target for the image processing. As described above, for example, images shot with a smaller number of cameras such as one camera are set to the target for the image processing, and thus it is possible to appropriately perform driving assistance corresponding to conditions of the vehicle through monitoring with the cameras in a satisfactory real-time manner.

(8) In some embodiments, in the configuration of (4) to (7) described above, the threshold values include a switching start threshold value and a switching completion threshold value in which a difference with the switching start threshold value is a first value, the processor is configured to perform the image processing on a predetermined number of pieces of the image data per unit time and when the vehicle speed is between the switching start threshold value and the switching completion threshold value, the processor switches the buffers to be selected such that, as the vehicle speed approaches the switching completion threshold value from the switching start threshold value, the predetermined number of pieces of image data on which the image processing is performed per the unit time before the vehicle speed reaches the switching start threshold value is replaced with the predetermined number of pieces of image data after the vehicle speed reaches the switching completion threshold value.

When the cameras (buffers) which are set to the target for the image processing are exactly switched at a predetermined threshold value, for example, if the vehicle speed obtained from the vehicle includes an error, appropriate buffers corresponding to the vehicle speed are not selected around the threshold value, with the result that it is likely that the appropriate environment recognition is not performed.

In the configuration of (8) described above, the cameras (buffers) which are set to the target for the image processing of the processor are gradually switched from the preceding stage so as to be completely switched at the predetermined threshold values (such as the first threshold value and the second threshold value described above). In this way, even when the vehicle speed obtained from the vehicle has an error, the ambient environment recognition on the vehicle can be appropriately performed.

(9) In some embodiments, in the configuration of (8) described above, when the vehicle speed reaches the switching start threshold value, the processor first switches the camera corresponding to the buffer which needs to be selected when the vehicle speed reaches the switching completion threshold value and the buffer corresponding to the camera which is least associated with a direction of shooting.

In the configuration of (9) described above, even when the vehicle speed obtained from the vehicle has an error, the ambient environment recognition on the vehicle can be appropriately performed.

(10) In some embodiments, in the configuration of (1) to (9) described above, the vehicle image processing device further includes: an acquisition portion acquiring characteristic information including an instruction for the monitoring direction in a specific place, and the processor selects the buffers based on the position of travelling and the characteristic information.

In the configuration of (10) described above, the processor sets the image data of the camera for shooting the area in the monitoring direction indicated by the characteristic information to the target for the image processing. Although in the vehicle which travels in various places, it may be necessary to monitor the specific direction in a focused manner such as a place in which accidents often occur, the cameras which are set to the target for the image processing are switched based on the characteristic information, and thus it is possible to perform the driving assistance which is safer.

(11) In some embodiments, in the configuration of (1) to (10) described above, the vehicle is an industrial vehicle, the state information includes the vehicle speed and the steering angle, the processor determines, based on the state information, whether turning of the vehicle is rightward turning or leftward turning and when the vehicle is turned rightward, the processor selects the buffer in which the piece of image data obtained by shooting at least an area on a right side in a forward direction and an area on a left side in a backward direction with respect to the direction of travelling is accumulated whereas when the vehicle is turned leftward, the processor selects the buffer in which the piece of image data obtained by shooting at least an area on the left side in the forward direction and an area on the right side in the backward direction with respect to the direction of travelling is accumulated.

For example, since an industrial vehicle such as a forklift has a short wheel base and a large steering angle, it is necessary to pay sufficient attention to the lateral direction as compared with the forward direction with respect to the direction of travelling when travelling is started in a state where a steering wheel is turned. Specifically, it is necessary to simultaneously monitor not only the right side in the forward direction with respect to the direction of travelling but also the left side in the backward direction with respect thereto at the time of rightward turning and not only the left side in the forward direction with respect to the direction of travelling but also the right side in the backward direction with respect thereto at the time of leftward turning.

In the configuration of (11) described above, the processor determines the direction of turning based on the vehicle speed and the steering angle, and selects, based on the result of the determination, the cameras which are set to the target for the image processing. As described above, according to the direction of turning (steering angle) of the vehicle, the image data of not only one side such as the right side but also the opposite side (left side) is set to the target for the image processing, and thus at the time of lateral turning, while attention is being paid to the direction opposite to the direction of travelling, for example, the detection of a person and an article is performed, with the result that it is possible to perform the driving assistance which is safer.

(12) In some embodiments, in the configuration of (11) described above, the cameras include a right-side shooting camera for shooting an area on the right side of the vehicle and a left-side shooting camera for shooting an area on the left side thereof, and when the vehicle is turned rightward, the processor performs the image processing on the region of part on a left side of the pieces of image data individually shot with the right-side shooting camera and the left-side shooting camera whereas when the vehicle is turned leftward, the processor performs the image processing on the region of part on a right side of the pieces of image data individually shot with the right-side shooting camera and the left-side shooting camera.

According to the direction of turning (steering angle) of the vehicle, the image data of not only one side such as the right side but also the opposite side (left side) is set to the target for the image processing, and thus the number of cameras set to the target for the image processing is increased, with the result that the frequency at which the image processing is performed with the processor on the image data accumulated in each of the buffers (cameras) is decreased (the cycle is decreased).

In the configuration of (12) described above, the processor performs the image processing on the region of part of each piece of image data acquired when the vehicle is turned. For example, according to the steering angle, the camera on the right side at the time of rightward turning processes the region (the left side of the image) of only the front side of the vehicle, and the camera on the left side processes the region (the left side of the image) of only the back side of the vehicle in a limited manner, that is, the image processing is performed on only the regions of parts, with the result that it is possible to reduce the prolongation of the acquisition cycle of the image data from each of the buffers. In this way, even when the number of cameras is increased, it is possible to perform driving assistance which is safer in a satisfactory real-time manner.

(13) In some embodiments, in the configuration of (1) to (12) described above, the state information includes the steering angle and the shift position of the gear of the vehicle, and the processor performs, based on the steering angle and the shift position the image processing on a region of part included in the pieces of image data acquired from the buffers.

In the configuration of (13) described above, based on the steering angle and the shift position, part of each piece of image data is extracted so as to be set to the target for the image processing. In this way, in each piece of image data, the image processing on the partial region which does not include the travelling route of the vehicle predicted based on the steering angle and the shift position can be omitted. Hence, it is possible to reduce the burden of the image processing on each piece of image data, and thus it is possible to increase the number of pieces of image data on which the image processing can be performed per unit time. Consequently, it is possible to cope with a performance time which is required in a high vehicle speed or the like.

(14) In some embodiments, in the configuration of (1) to (13) described above, the cameras include a first camera for shooting an area in a first direction of the vehicle and a second camera for shooting an area in a direction different from the first direction, and the vehicle image processing device further includes an adjustment portion adjusting, based on the brightness of the pieces of image data individually shot with the first camera and the second camera, the shooting parameter for each of the cameras.

In the configuration of (14) described above, the frontlight and backlight of the cameras which are set to the target for the image processing are detected such that the shooting parameters for the cameras are adjusted, and thus it is possible to minimize influence caused by variations in brightness produced by sunlight, with the result that it is possible to perform stable environment recognition.

(15) A vehicle image processing method according to at least one embodiment of the present invention is performed by a processor of a computer, the computer includes: a plurality of buffers configured to accumulate pieces of image data input individually and sequentially from a plurality of cameras installed in a vehicle so as to associate the pieces of image data with the cameras; the processor configured to select the buffer based on the state information of the vehicle including a vehicle speed and acquire the piece of image data from the selected buffer so as to perform image processing thereon; a signal line for transferring the pieces of image data in the buffers to the processor; and a transfer controller configured to output the piece of image data in the buffer required from the processor to the signal line and the vehicle image processing method include a step of performing the selection of the buffer based on comparison of the vehicle speed and threshold values.

In the configuration of (15) described above, the same effects as in (1) described above are achieved.

(16) A vehicle image processing program according to at least one embodiment of the present invention instructs a processor of a computer to perform a step of performing the selection of a buffer based on the comparison of a vehicle speed and threshold values, and the computer includes: a plurality of buffers configured to accumulate pieces of image data input individually and sequentially from a plurality of cameras installed in a vehicle so as to associate the pieces of image data with the cameras; the processor configured to select the buffer based on state information of the vehicle including the vehicle speed and acquire the piece of image data from the selected buffer so as to perform image processing thereon; a signal line for transferring the pieces of image data in the buffers to the processor; and a transfer controller configured to output the piece of image data in the buffer required from the processor to the signal line.

In the configuration of (16) described above, the same effects as in (1) described above are achieved.

(17) A storage medium according to at least one embodiment of the present invention is a computer-readable storage medium storing the vehicle image processing program discussed in (16) described above.

In the configuration of (17) described above, the vehicle image processing program stored in the storage medium is executed, and thus the same effects as in (1) described above are achieved.

According to at least one embodiment of the present invention, a vehicle image processing device is provided which can more rapidly perform, while reducing the cost of a vehicle, image processing on images shot with a plurality of cameras.

