BACKGROUND

Geometric Camera Parameters

A camera is an optical instrument to project three-dimensional (3D) world images onto a flat surface referred to as an image plane. Camera images may or may not be within the visible light spectrum. A camera-based vision system characterizes its environment as a two-dimensional image. A camera can be used to infer the depth or range of detected objects. Optical images captured at camera pixels in the image plane are transformed to real world depth measurements to analyze certain aspects of a scene, such as the size or position of objects. However, a camera generally must be calibrated periodically with geometric parameters, also referred to as extrinsic parameters, that provide precise camera location and camera pose information required to accurately infer spatial relations from its images. In stereo camera systems, this transformation from pixel coordinates to depth information can be done with disparity computation between the camera sensors after basic calibration, and in monocular systems this typically can be done with prior scene parameters/rules or with machine learning methods known in the art. In a mono camera system, for example, camera height and angle relative to a ground plane must be known (or calibrated for) precisely in order to infer high quality depth maps. In a multi-camera system, for example, the relative distance and pose of the cameras with respect to one another must be known (or calibrated for) precisely in order to infer high quality depth maps.

Calibration of a multi-camera system typically is primarily to estimate a mathematical matrix that maps between a 3D world and each individual camera view. The matrix mapping includes geometric parameters that provide translation distances and relative angles between cameras in a multi-camera system. Calibration of geometric parameters has six degrees of freedom in terms of translation and rotation between two cameras. A mapping or reprojection between camera views also can be inferred from these matrices. A perfect calibration results in zero reprojection error among the camera views. Often, geometric calibration is performed using a checkerboard pattern placed at some known distance. In general, however, any 3D world features with some prior knowledge of the pattern can be used, e.g., size of the squares in a checkerboard pattern.

FIGS. 1A-1Bare illustrative drawings showing an example stereo camera system including a left camera102and a right camera104with cameras horizontally aligned parallel in an x-axis (FIG. 1A) and showing an example stereo camera system with a relative rotation of the right camera104such that the left and right cameras102,104are misaligned in the x-axis (FIG. 1B). In both the aligned and misaligned examples, an image of an object106in a 3D world is projected on a left camera image plane108and a right camera image plane110. For the horizontally aligned cameras example inFIG. 1A, a z-distance of the object from the cameras can be calculated as,
Z=BF/(xL−xR)  (1)
where Z is the distance from the object computed based upon geometric parameters, B is distance between the cameras, F is focal length, xL is x-axis offset of the object image within the left camera, and xR is offset of the object image within the right camera. For the misaligned camera image planes example inFIG. 1B, an unknown, angular offset in the misaligned (right) cameras example will correspond to a shift (dx) of the object in the (right) image planeFIG. 1B. In this misaligned stereo camera example, without camera calibration, if we assume zero rotation between the cameras, the z-distance of the object from the cameras will be calculated as,
Z_hat=BF/(xL−xR−dx)  (2)
Taking into consideration the offset on the right side as xR+dx, Z_hat is the computed distance of the object based on the uncalibrated, zero rotation geometric parameters. In this example Z_hat<Z. Thus, a small, unknown (uncalibrated) rotation of the cameras with respect to each other influences the accuracy of the z-direction measurement of the cameras. In order to resolve this inaccuracy in depth estimation, one must properly calibrate to determine the precise pose of the cameras with respect to each other.

Unfortunately, in actual use in the field, various geometric factors such as vibrations, mechanical shock, large temperature variations, stress/strain on fixture, for example, can frequently and repeatedly throw off camera calibration, which can necessitate more frequent calibration. Moreover, actual distances from 3D world objects in the field generally are unknown and estimated distance using a camera optics generally is accurate only to a camera scale factor. Therefore, there exists a need to accurately and quickly adjust geometric parameters used to calibrate the pose of camera systems in the field where actual distances are unknown.

Photometric Camera Parameters

A vision system whether comprised of a single camera or multiple cameras can require photometric calibration to achieve acceptable image quality. For a mono-camera the photometric calibration can involve setting photometric parameters (sometimes referred to as ‘intrinsic’ parameters) such as auto-exposure (AE), auto-white balance (AWB), and auto-focus (AF), (the ‘3A’ parameters), gain, and HDR, for example, to achieve visibility of the scene in varied conditions including mixed lightening and challenging scenarios (such as WDR, low light, direct sun-light). For a multi-camera system a calibration also can involve photometric calibration across the camera views to achieve a uniform visual experience in 3A, gain/tone mapping and HDR. In low light or dark conditions, a multi-camera system is usually coupled with one or more illuminators (visible and/or IR) and in these conditions a continuous photometric correction is usually needed between the illuminator power and the camera's photometric calibration settings to avoid saturation, avoid noisy images and maintain uniformity across views.

SUMMARY

In one aspect, an imaging system is provided that includes a memory storing instructions which, when executed by processing circuitry, cause the processing circuitry to perform operations. The operations include using a camera to receive image information projected at an image plane of the first camera from a 3D world. The operations include determining multiple image regions of interest (ROIs) based upon the image information, wherein individual ones of the multiple image ROIs correspond to one or more three-dimensional (3D) world object images projected at the image plane. The operations include using a radar unit to receive radar information indicating one or more 3D world objects. The operations include determining multiple respective radar ROIs are determined that correspond to one or more 3D world objects. The operations include determining 3D world distances that correspond to the radar ROIs. The operations include determining multiple co-registered ROI pairs by co-registering individual image ROIs with individual radar ROIs corresponding to common 3D world objects. The operations include adjusting one or more geometric parameters associated with the camera based upon the co-registered ROI pairs.

In another aspect, an imaging system is provided that includes a memory storing instructions which, when executed by processing circuitry, cause the processing circuitry to perform operations. The operations include receiving an image of an object using a camera calibrated using a group of intrinsic parameters. The operations include determining luminance information using the camera. The operations include, receiving, using a radar unit, radar information indicating an object. The operations include, determining based upon the radar information, a speed to associate with the object. The operations include adjusting one or more intrinsic parameters of the group of intrinsic parameters based upon the determined luminance information and the tracked speed of the object.

