CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a U.S. National Stage Application under 35 U.S.C. § 371 of International Patent Application No. PCT/IB2019/052926, filed Apr. 9, 2019, which claims priority to European Patent Application No. 18169993.5, filed Apr. 28, 2018 and Portugal Patent Application No. 110671, filed Apr. 9, 2018, the contents of which are each hereby incorporated by reference in their respective entireties.

TECHNICAL FIELD

The present disclosure relates to a hybrid imaging system with particular application for underwater robotic operations.

BACKGROUND

The developments in the field of visual perception have contributed to the appearance of new and sophisticated autonomous robots. Acoustic sensors are often used by underwater applications to conduct bathymetric surveys, obstacle avoidance and long-range navigation. However, they have severe limitations related with the cost and the inability to provide texture information and low sampling frequencies. On the contrary, optical systems can provide dense and accurate texture information updated at high speed, which is crucial for tasks such as, underwater manipulation, oceanographic studies and marine biology, shipwreck surveys and object identification. Although optical systems can increase the feasibility of missions that are be carried out by AUVs, they need to be specifically designed to overcome the photometric limitations of sub-sea conditions namely, poor visibility due to light attenuation and scattering and caused by suspended particles in the medium or abundance of marine life, non-uniform lighting and colour filtering.

Conventional systems for 3D reconstruction are usually classified in passive or active imaging techniques.

Underwater passive imaging uses ambient sunlight at shallow waters or light sources (i.e., spotlights and LEDs) at deeper waters just to ensure a sufficient lighting of the scene. This category is based on photogrammetry where information of the scene taken from different viewpoints is used for underwater 3D acquisition. A stereoscopic system estimates the stereo correspondence from images obtained by different viewpoints and, this technique is often used for close and medium range acquisition. Usually, conventional stereoscopic setups are formed by two industrial cameras and the estimation of the point cloud is conducted by a SAD (Standard Absolute Differences correlation method) which provides an acceptable quality. These systems do not provide additional imaging methodologies to accommodate the photometric limitations of underwater environments.

Oleari et al. (2015) [1] demonstrates a stereoscopic setup formed by two industrial cameras, but this systems does not accommodate the photometric limitations of underwater environments. Experiments conducted in a water pool with different light conditions (sunny, cloudy and night using vehicle lights) but without turbidity constraints have demonstrated the challenges of perceiving the underwater environment, in particular, the balance between the illumination of scene and the camera shutter. An advanced vision system [2], formed by two stereoscopic cameras (Bumblebee sensors) enhances the maneuverability and manipulation capability of underwater robots. This system is called Fugu-f, has a depth rate of 250 m and can be adaptable to different vehicles. The results demonstrated that this system can be used for different tasks (such as, object detection and grasping) however, a quantitative analysis of the accuracy of Fugu-f for 3D mapping is missing.

Underwater active imaging uses the projected pattern of light to determine the distance between the device and the illuminated target by means of time-of-flight or triangulation principle. There was been a recent interest regarding to underwater active sensors since they are typically employed for long and medium range acquisitions and enhance the quality of images perceived by the receiver by controlling a light pattern. A laser-based range sensor for underwater environments formed by one camera and two lasers projecting continuous vertical lines is presented in [3]. The authors have reported a calibration error of 5% and results obtained in a controlled tested scenario (with very clear water) showed a set of measurements with a drift close to 0.04 m.

The work presented in Bruno et al. (2011) [4] concludes that the structured light can be used in underwater environments and the authors have reported good results for a 3D reconstruction in low turbidity waters. The solution presented requires a camera, projector and a physical structure capable of protecting the entire system when it is submerged in water. The presence of a projector in this kind of applications makes it possible to conduct range measurements with a well-defined pattern that is normally projected in the scene however, the electric power required by this device is non-negligible since it reduces the autonomy of a robotic vehicle. The work of Amin and Brian (2016) [5], extends a structured light technique based on a camera below water and a projector fixed above water. Thus, the scene is illuminated with spatially varying intensity patterns to allow the extraction of the 3D surface of objects. Despite of its accuracy, this technique can only be used in specific case-scenarios where the observer is standing still. Advances in remote visual sensing technology have been followed closely by advances in the artificial illumination.

Currently, the limited visual perception capability of AUVs restricts the use of such vehicles to medium/long range missions. Therefore, the development of visual methods for enhancing the perception of elements in underwater could be extremely useful for robotic operations in murky and in medium and highly turbid underwater environments.

REFERENCES

[1] Oleari, F., Kallasi, F., Rizzini, D. L., Aleotti, J., Caselli, S., May 2015. An underwater stereo vision system: From design to deployment and dataset acquisition. In: OCEANS 2015—Genova. pp. 1-6.[2] Bonin-Font, F., Oliver, G., Wirth, S., Massot, M., luis Negre, P., Beltran, J.-P., 2015. Visual sensing for autonomous underwater exploration and intervention tasks. Ocean Engineering 93, 25-44.[3] Hansen, N., Nielsen, M. C., Christensen, D. J., Blanke, M., 2015. Short-range sensor for underwater robot navigation using line-lasers and vision. IFAC-PapersOnLine 48 (16), 113-120, 10th IFAC Conference on[4] Bruno, F., Bianco, G., Muzzupappa, M., Barone, S., Razionale, A., 2011. Experimentation of structured light and stereo vision for underwater 3d reconstruction. ISPRS Journal of Photogrammetry and Remote Sensing 66 (4), 508518.[5] Sarafraz, A., Haus, B. K., 2016. A structured light method for underwater surface reconstruction. ISPRS Journal of Photogrammetry and Remote Sensing 114, 40-52.[6] Hirschmuller, H., 2008. Efficient belief propagation for early vision. IEEE transactions on pattern analysis and machine intelligence 30 (2), 328-341.

GENERAL DESCRIPTION

It is disclosed a visual imaging system that is able to provide dense and accurate 3D information in hash underwater environments using both active and passive imaging methods: multiple light stripe range and a photometric stereo technique, respectively. This hybrid approach combines information from these imaging techniques to extend the measurement range and to produce high density 3D estimations in dynamic underwater environments. Moreover, the disclosed system can synchronize and match the acquisition of images with sub-sea phenomena which leads to clear pictures (with a high signal-to-noise ratio). This hybrid system is driven by a gating timing approach to reduce the impact of several photometric issues related to the underwater environments such as, diffuse reflection, water turbidity and non-uniform illumination.

It is therefore an object of the present invention to provide a hybrid imaging system that is capable of perceiving 3D structures in underwater scenarios. The advantages of the disclosure, when compared to conventional visual sensors, includes reducing the photometric limitations that usually affect imaging solutions operating in underwater scenes (mitigates the underwater effects related to the light propagation in deep waters) and increases the density, robustness and accuracy of 3D points extracted by the disclosure. In particular, by having the same image sensors for both structured and isotropic light using a switched approach, an improved 3D reconstruction is obtained, the system is less complex and also less energy consuming.

The light interacts with the water medium through two phenomena: absorption (loss of power which depends on the index of refraction of the medium) and scattering (deflection from a straight-line propagation path). Thus, the irradiance E(r) at position r can be described as a function of the absorption and scattering coefficients of the medium. Under these terms, a simple analysis can be conducted by considering the Lambert-Beer empirical law along with the assumption of a homogeneous and isotropic water medium. Equation 1 demonstrates how both components (scattering and absorption) decrease the contrast and attenuate the light intensity of images.
E(r)=E(0)e−are−br;  (1)
where a and b are the absorption and scattering coefficients. Considering the image formation model of Jaffie-McGlamery [6], an underwater image can be expressed as the linear superposition of the direct component (Ed), the forward-scattered component (Ef) and the backscatter component (Eb), see equation 2.
ET=Ed+Ef+Eb;  (2)

where ETis the total irradiance. The Ebrepresents the light that does not contain information about the object since it was reflected by the medium. The Ebadds a noise component and reduces the image contrast. The Efrepresents the amount of light reflected by the object which is forward scattered at small angles. Its practical evidence is an image blur. The majority of research works available in the literature are usually focused on proposing post-processing methods to mitigate Efand Ebcomponents. This research work takes a different path by incorporating a set of imaging principles and techniques directly into an optical system. These innovations aim to increase the robustness of the image formation process by minimizing the influence of Eb(and Ef).

The advantage of the disclosed imaging system when compared to conventional visual sensors relies on the fact that it was developed with set of features to reduce the photometric limitations that usually affect imaging solutions operating in underwater scenes. The disclosed imaging system combines a hybrid-based visual approach controlled by a range-gated imaging scheme with a data-driven formulation. The internal architecture of the disclosed imaging system is formed by three main modules according to an embodiment: multiple Light Stripe Range (LSR), Photometric Stereo (PS) and the Range-Gated Synchronizer. The 3D information received from the LSR and the PS modules are combined and filtered in a fourth module named, 3D fitting and texture. These modules are organized in a functional structure, seeFIG. 2.

The hardware of disclosed imaging system is formed by two cameras (left and right), two laser stripes (red and green wavelengths), a triggering system, and the illumination module (high intensity LEDs).FIG. 1presents different view-points for the concept design of the disclosed system, in particular, the physical layout of the entire system.

The disclosed imaging system and method are particularly advantageous for obtaining underwater texture data, as combining both structured-light and spatially non-coherent light, while filtering out closer-than-target incoming light, allows full texture reconstitution of larger and smaller 3D features, while avoiding unnecessary short-distance backscatter particle noise which impairs for said texture retrieval.

All components are preferably interconnected with software modules that extract the 3D information from visual data. The architecture implemented in the disclosed system according to an embodiment is depicted inFIG. 2. Outputs of the system according to an embodiment comprise a pair of images from the PS, point clouds retrieved by each 3D imaging technique (4 point clouds from the LSR and a dense point cloud from the PS). These point clouds can be combined further by applying 3D filtering techniques

It is disclosed a hybrid imaging system for 3D imaging of an underwater target, comprising:two optical image sensors for stereoscopic imaging of the target;a switchable structured-light emitter having different wavelengths for illuminating the target;a switchable spatially non-coherent light source for illuminating the target;a data processor connected to the image sensors, structured-light emitter and spatially non-coherent light source, configured for alternating between operating modes which comprise:a first mode wherein the structured-light emitter is activated, the spatially non-coherent light source is deactivated and the image sensors are activated to capture reflected light from the structured-light emitter, anda second mode wherein the structured-light emitter is deactivated, the spatially non-coherent light source is activated and the image sensors are activated to capture reflected light from the light source;wherein the data processor is configured for delaying image sensor capture, on the activation of the structured-light emitter and on the activation of the spatially non-coherent light source, for a predetermined amount of time such that light reflected from points closer than the target is not captured.

It is also disclosed a hybrid imaging system for 3D imaging of an underwater target by alternating structured-light and spatially non-coherent light, comprising:two optical image sensors for stereoscopic imaging of the target;a switchable structured-light emitter having different wavelengths for illuminating the target;a switchable spatially non-coherent light source for illuminating the target;a data processor connected to the image sensors, structured-light emitter and spatially non-coherent light source,wherein the data processor is configured for delaying image sensor capture, on the activation of the structured-light emitter and on the activation of the spatially non-coherent light source, for a predetermined amount of time such that first light reflected from points closer than the target is not captured.

In an embodiment, the data processor is configured for alternating between operating modes which comprise:a first mode wherein the structured-light emitter is activated, the spatially non-coherent light source is deactivated and the image sensors are activated to capture reflected light from the structured-light emitter, anda second mode wherein the structured-light emitter is deactivated, the spatially non-coherent light source is activated and the image sensors are activated to capture reflected light from the light source.

Both capturing structured light and stereoscopic images benefits from having switched lighting, because by using the time of travel of light as light begins its course, after being switched on, one can exclude the first light reflected back to image sensors. This way, one can remove the mostly back-scattered light from underwater turbidity, which is reflected back to the image sensor from the proximity of the sensor. This improves the quality of the image remarkably.

When using switched lighting, more powerful light sources and emitters are normally used. Commonly speaking, the effect can be similar to a photographic flash repetition, wherein short bursts of intense light are emitted quickly and repeatedly.

Thus, by having switched lighting, it should be ensured that the structured light (normally, an intense laser light) does not obfuscate the image sensors during image capture of the lighting by common light—loosing image sharpness. The sharpness of the captured images is important for stereoscopic computer vision analysis of the common lighting images.

Conversely, it should also be ensured that the lighting by common light does not impair the image sensors during image capture of the lighting by structured-light—laser lighting is not so effective against darker surfaces and the broad common light will remove contrast from the captured images. This contrast is important for 3D computer vision analysis of the structured lighting images.

In an embodiment, a data processor is further configured for processing both captured structured-light images and light source images to obtain merged 3D imaging data. In an embodiment, the data processor is configured to, previous to said processing, to extract 3D data from the captured structured-light images and from the captured light source images. In an embodiment, the data processor is configured to extract 3D data from the captured light source images by rectifying the captured images according to a pre-calibration, by estimating spatial disparity between the captured images from the two image sensors, and by calculating a 3D point cloud from the estimated spatial disparity. In an embodiment, the data processor is configured to extract 3D data from the captured structured-light images by correcting for distortion according to a pre-calibration, by triangulating the 3D locations of the reflected structured-light, and by calculating a 3D point cloud from the triangulated 3D locations.

In an embodiment, said merger of 3D data is carried out periodically and with a frequency of at least 1 Hz, preferably of at least 10 Hz, more preferably of at least 30 Hz, more preferably of at least 60 Hz, more preferably of at least 100 Hz. By periodically combining the 3D merged data obtained between structured-light and spatially non-coherent light, the underwater vehicle can avoid obstacles in real-time. Also, a robotic underwater manipulator can use in real-time the merged 3D texture data obtained between structured-light and spatially non-coherent light. For example, surfaces covered in biofilm or rusty surfaces have specific 3D textures; information that a robotic arm can make use of in real-time.

An embodiment is further configured for segmenting the reflected structured-light in the captured structured-light images by using color space, in particular the YcrCb color space.

In an embodiment, the 3D imaging data comprises 3D texture data. In an embodiment, the data processor is configured to maintain the structured-light emitter activated during image sensor exposure time for capturing the reflected light from the structured-light emitter. In an embodiment, the data processor is configured to maintain the light source activated during image sensor exposure time for capturing the reflected light from the light source.

By having the light remaining switched on during substantially all the exposure period of the cameras, this increases the accuracy of the 3D imaging, reduces the power consumption of the system and enables higher speed acquisitions.

An embodiment comprises additional optical image sensors for 3D imaging of the target. Additional image sensors are useful for improving the image capture in particular the 3D resolution.

In an embodiment, the structured light emitter is a dual laser stripe. In an embodiment, the laser stripe is substantially parallel to a movement path of the imaging system. In an embodiment, the two lasers have a different wavelength. In an embodiment, the structured light emitter comprises a red laser stripe and a green laser stripe. A laser stripe is advantageous for underwater use, especially when used along a movement path.

In an embodiment, the light source comprises LEDs. In an embodiment, the light source is substantially isotropic in illumination of said target. LEDs are efficient light sources suitable for flooding a considerable area with spatially non-coherent light.

In an embodiment, the optical image sensors are cameras. Cameras are efficient cost-effective image sensors and are able to capture both reflected light from the structured-light emitter and reflected light from a common spatially non-coherent light source.

The hybrid imaging approach disclosed brings together active and passive imaging principles: multiple Light Stripe Range (LSR) and a Photometric Stereo (PS) technique.

The LSR is an active technique that comprises a set of stripes of light generated by laser diodes that are projected in the scene. The 3D information is recovered from these laser stripes by means of triangulation. Laser technology has a strong advantage in underwater scenes since emitters concentrate an intense light over a very narrow area which makes the light to propagate longer distances.

The segmentation of the laser stripe can be conducted in the YCrCb because this space is more photometric invariant to light changes and the chroma components of this space are less intercorrelated. Then, a straight-forward segmentation procedure takes into consideration the chroma values (Cr and Cb) and confidence intervals. Afterwards, the preliminary segmentation results are redefined by a watershed approach that groups pixels with similar spatial position and color. Morphological operators increase the spatial connectivity of the segmentation result.

Before extracting the 3D information, a calibration must be performed to estimate the spatial arrangement of each laser according to the camera frame. Thus, the triangulation determines the three-dimensional points by intercepting 2D points (image plane because the camera matrix is known) from the segmentation with the plane equations available, see equations 3 to 5:

Z=-(a*x+b*y+dc)(3)X=Z*x(4)Y=Z*y(5)
where x and y are the 2D points described in the image plane and, the a; b; c and d define a plane that characterizes the laser-camera setup. The X; Y and Z are 3D coordinates of a point represented in the camera frame. This triangulation process is repeated for each pair camera-laser.

The biggest advantage of an active imaging approach is the high accuracy and robustness of data that can be retrieved from the scene. Usually, the propagation of light in water is seriously compromised by the scatter and absorption of light caused by the medium which reduce the image contrast and intensity [1]. The LSR calculates distances and object sizes with a high degree of precision, minimizes the backscatter and increases the image contrast.

The PS is a passive technique that reconstructs the 3D surfaces based on different viewpoints. It provides point clouds having a high density of 3D data when compared to LSR. The knowledge of the relative position and orientation of both cameras (separated by a rigid baseline) makes possible the extraction of the three-dimensional surface through the stereoscopy principle: 3D points generated by 2D point pairs. Conventional stereo vision could be performed in two distinct ways namely, sparse (a set of point pairs) or dense (for every pixel). Both ways can be used in the disclosed system, depending on the application. A key element of the PS module is the estimation of the disparity of images acquired in different viewpoints. This is often called as the correspondence problem (the challenge of associating 2D point pairs). The perception of depth arises from disparity by means of triangulation (depth is inversely proportional to disparity). The set of 3D points (also known as a point cloud) can be more or less dense depending on the amount of valid stereo correspondences. In practice, the PS module is suitable for environments having dynamic elements because it provides data that is richer in 3D information.

The algorithm 2 presents high-level procedures for extracting 3D information based on a stereo rig. The implementation is straightforward in computer vision: the calibration of the stereo rig should is conducted in aerial and underwater scenarios, the disparity estimation follows [6], and 2D points with no disparity must be removed.

Algorithm 2 PS module: Extraction of 3D data from 2D point pairs.load_StereoRig_Setup();Spatial conf.procedure EVENT_INCOME_IMAGES()Itleft, Itright= rectify_Images(Itleft, Itright);disparity = calculate_Disparity(Itleft, Itright);ptcloud = 3D_Reconstruction(disparity);PointCloud_RGBXYZ = remove_Invalid_3D_Points(ptcloud);return PointCloud_RGBXYZDense_Pointcloud & Itleft

The disclosed imaging system also has the ability to control the formation of image in both cameras (e.g., the starting instant and duration). The range-gated imaging concept lays on the principle that the reflected light source pulsed from the LEDs or lasers is accumulated by an optical sensor and over a specified time, i.e., the camera shutter only opens after a short period of time. Thus, the camera sensor receives the light reflected from the object and blocks the back-scattered photons. The exposure time selects the depth range where the light reflected is perceived at the sensor location. This approach is one of the most effective optical solution for visual perception in turbid waters since it increases the image contrast by reducing the backscattering component.

The backscattering component affects the underwater imaging sensors: a fraction of light that passes through water is scattered back to the camera before reaching the target. Conventional optical systems are severely affected by Eb, which is mainly caused by particles suspended along the water column. These particles increase the water turbidity, and reduce both the visibility and the measurement range of optical systems. In this context, the range-gated feature available on the disclosed imaging system delays the image acquisition in a short period of time after the scene be illuminated by a light source: the artificial illumination module or the laser light from the LSR. An active-illumination functionality is implemented based on high intensity LEDs (light emitting diodes) to create a close-to-isotropic light source to illuminate the underwater scenario when the system is operating in total darkness. This active-illumination light is crucial at deeper environments because several light sources produce an uniform illuminated area which enhances the texture content of data perceived by the sensor.

The disclosed imaging system generates several triggering signals (PSTrigand LSRTrig) that are sent to cameras with external triggering to control: the PSTrigactivates the artificial illumination and the PS modules, and the signal LSRTrigactivates the LSR module. These periodic signals are generated by an external trigger control unit called Range-Gated Synchronizer whose main objective is to ensure that images are acquired in the correct instant of time. Triggering signals PSTrigand LSRTrigare only sent to cameras with after a delay period of Tstartof these signals were received by the LSR and the active-illumination unit.

In this context, the camera shutter waits for the time that light takes to propagate from the emitter to the target. Ideally, Tstartis controlled according to the distance and the exposure time should be adjusted relatively to the Tstop−Tstart. The goal of this approach is to reduce the amount of backscattering component that is perceived by the optical sensors which is mainly caused by suspended particles close to the observer. Therefore, the disclosed imaging system incorporates a range-gated feature that synchronizes the acquisition of image with sub-sea phenomena. The gated timing approach limits the exposure time of the optical system to the returning time of a light pulse (laser or LED) scattered by the target at a defined distance. A practical advantage of this range-gated approach relates with the robustness of the optical system to other sources of light or external interferences (exposure time versus the diaphragm's aperture). Although the influence of Ebcan be reduced by a range-gated approach (synchronization of the light source and camera), the Efcannot be quantitatively reduced by this method since this component reaches the image detector approximately at the same time than Ed.

The data-driven formulation proposed in this research follows a time discrimination pattern to aggregate the 3D information generated by the LSR and PS approaches. In this way, each image captures the light returned from a single light pulse that could be driven by the LSR or by the PS module. In practice it means that, the optical components of the disclosed imaging system are sequentially gated by the LSR (without the active-illumination source) and by the PS (with the active-illumination source) to minimize the amount of backscattering light that reaches the optical sensors, as well as, to mitigate the motion blur that arrives when the system is installed on the AUV or ROV.

A data-driven scheme synchronizes the acquisition instant of multi-cameras based on distinct time events. These time events are represented by signals PSTrigand LSRTrig, which control the camera exposure delayed by Tstart/2. Signals describing the ON period of the active-illumination module or lasers anticipate (by Tstart/2) the time events: LSR ON and PS ON. In this way, the light sources (LSR or active-illumination) remain ON during the exposure period of the multi-cameras which increases the accuracy (real-time) of the synchronization process, reduces the power consumption and enables high speed acquisitions.

Therefore, the disclosed imaging system follows a time discrimination pattern to create at least two operating modes: LSR ON and PS ON, seeFIG. 7(b). Optical sensors are able to capture the laser and LED light in distinct time slots. The PS ON mode shows that the disclosed imaging system acquires images that are analyzed by the PS module: estimates the 3D surface of the scenario illuminated by the active-illumination module. Overall, this module artificially illuminates the scene based on a rapidly variation of light pulses (strobe) which has several advantages: it is more energy-efficient than continuously emitted light sources and reduces the motion blur. This solution is, therefore, suitable for autonomous vehicles. The LSR ON mode recovers the three-dimensional structure of the environment from multiple laser-camera triangulations, conducted in the LSR module. In this mode, the active-illumination module remains OFF to decrease the amount of external light interferences during the segmentation of the laser stripes.

In practice, the disclosed imaging system combines multiple laser-based range-gated imaging with a stereo-illuminated range-gated imaging system, in a watertight enclosure. High intensity LEDs (Light Emitting Diodes) forms the active illumination module that turns ON during the exposure period of cameras (in the PS ON mode). A set of pulse-gated laser stripes are processed by a LSR module (during the LSR ON mode) which extends the optimal range of the disclosed system to retrieve the three-dimensional characteristic of underwater scenarios. This gating timing scheme makes it possible to create images with high signal-to-noise ratio.

