Non-line-of-sight (NLOS) imaging of objects not visible to either the camera or illumination source is a challenging task with vital applications including surveillance and robotics. Recent NLOS reconstruction advances have been achieved using time-resolved measurements which requires expensive and specialized detectors and laser sources. In contrast, we propose a data-driven approach for NLOS 3D localization requiring only a conventional camera and projector. We achieve an average identification of 79 object identification for three classes of objects, and localization of the NLOS object's centroid for a mean-squared error (MSE) of 2.89cm in the occluded region for real data taken from a hardware prototype. To generalize to line-of-sight (LOS) scenes with non-planar surfaces, we introduce an adaptive lighting algorithm. This algorithm, based on radiosity, identifies and illuminates scene patches in the LOS which most contribute to the NLOS light paths, and can factor in system power constraints. We further improve our average NLOS object identification to 87.8 accuracy and localization to 1.94cm MSE on a complex LOS scene using adaptive lighting for real data, demonstrating the advantage of combining the physics of light transport with active illumination for data-driven NLOS imaging.
introduced looking around corners" using 5D time-resolved light transport @cite_15 . This was later realized using streak cameras and a femto-second laser to perform elliptic backpropagation @cite_40 @cite_5 . Researchers demonstrated NLOS reconstruction using continuous-wave time-of-flight sensors via optimization @cite_14 and NLOS tracking using array signal processing @cite_1 . Single-photon avalanche diodes (SPADS) @cite_36 have been used for NLOS reconstruction @cite_37 @cite_29 . Advances in reconstruction algorithms for SPADs include accounting for partial occluders and surface normals @cite_30 , confocal NLOS reconstruction @cite_29 , geometric modeling @cite_46 , backpropagation @cite_18 @cite_11 and space-carving @cite_10 . Higher level applications such as detection and tracking have been shown @cite_25 @cite_27 . Recently, neural networks have been applied to time-resolved measurements for NLOS imaging @cite_9 @cite_32 .
@cite_30 @cite_37 @cite_14 @cite_18 @cite_36 @cite_46 @cite_29 @cite_9 @cite_1 @cite_32 @cite_40 @cite_27 @cite_5 @cite_15 @cite_10 @cite_25 @cite_11
2752118498 2952225226 2755257996 2082344075
Recent works have demonstrated non-line of sight (NLOS) reconstruction by using the time-resolved signal from multiply scattered light. These works combine ultrafast imaging systems with computation, which back-projects the recorded space-time signal to build a probabilistic map of the hidden geometry. Unfortunately, this computation is slow, becoming a bottleneck as the imaging technology improves. In this work, we propose a new back-projection technique for NLOS reconstruction, which is up to a thousand times faster than previous work, with almost no quality loss. We base on the observation that the hidden geometry probability map can be built as the intersection of the three-bounce space-time manifolds defined by the light illuminating the hidden geometry and the visible point receiving the scattered light from such hidden geometry. This allows us to pose the reconstruction of the hidden geometry as the voxelization of these space-time manifolds, which has lower theoretic complexity and is easily implementable in the GPU. We demonstrate the efficiency and quality of our technique compared against previous methods in both captured and synthetic data. Imaging objects obscured by occluders is a significant challenge for many applications. A camera that could “see around corners” could help improve navigation and mapping capabilities of autonomous vehicles or make search and rescue missions more effective. Time-resolved single-photon imaging systems have recently been demonstrated to record optical information of a scene that can lead to an estimation of the shape and reflectance of objects hidden from the line of sight of a camera. However, existing non-line-of-sight (NLOS) reconstruction algorithms have been constrained in the types of light transport effects they model for the hidden scene parts. We introduce a factored NLOS light transport representation that accounts for partial occlusions and surface normals. Based on this model, we develop a factorization approach for inverse time-resolved light transport and demonstrate high-fidelity NLOS reconstructions for challenging scenes both in simulation and with an experimental NLOS imaging system. Non-line-of-sight (NLOS) imaging utilizes the full 5D light transient measurements to reconstruct scenes beyond the cameras field of view. Mathematically, this requires solving an elliptical tomography problem that unmixes the shape and albedo from spatially-multiplexed measurements of the NLOS scene. In this paper, we propose a new approach for NLOS imaging by studying the properties of first-returning photons from three-bounce light paths. We show that the times of flight of first-returning photons are dependent only on the geometry of the NLOS scene and each observation is almost always generated from a single NLOS scene point. Exploiting these properties, we derive a space carving algorithm for NLOS scenes. In addition, by assuming local planarity, we derive an algorithm to localize NLOS scene points in 3D and estimate their surface normals. Our methods do not require either the full transient measurements or solving the hard elliptical tomography problem. We demonstrate the effectiveness of our methods through simulations as well as real data captured from a SPAD sensor. Tomographic image reconstruction, such as the reconstruction of computed tomography projection values, of tomosynthesis data, positron emission tomography or SPECT events, and of magnetic resonance imaging data is computationally very demanding. One of the most time-consuming steps is the backprojection. Recently, a novel general purpose architecture optimized for distributed computing became available: the cell broadband engine (CBE). To maximize image reconstruction speed we modified our parallel-beam backprojection algorithm [two dimensional (2D)] and our perspective backprojection algorithm [three dimensional (3D), cone beam for flat–panel detectors] and optimized the code for the CBE. The algorithms are pixel or voxel driven, run with floating point accuracy and use linear (LI) or nearest neighbor (NN) interpolation between detector elements. For the parallel-beam case, 512 projections per half rotation, 1024 detector channels, and an image of size 512 2 was used. The cone-beam backprojection performance was assessed by backprojecting a full circle scan of 512 projections of size 1024 2 into a volume of size 512 3 voxels. The field of view was chosen to completely lie within the field of measurement and the pixel or voxel size was set to correspond to the detector element size projected to the center of rotation divided by 2 . Both the PC and the CBE were clocked at 3 GHz . For the parallel backprojection of 512 projections into a 512 2 image, a throughput of 11 fps (LI) and 15 fps (NN) was measured on the PC, whereas the CBE achieved 126 fps (LI) and 165 fps (NN), respectively. The cone-beam backprojection of 512 projections into the 512 3 volume took 3.2 min on the PC and is as fast as 13.6 s on the cell. Thereby, the cell greatly outperforms today’s top-notch backprojections based on graphical processing units. Using both CBEs of our dual cell-based blade (Mercury Computer Systems) allows to 2D backproject 330 images s and one can complete the 3D cone-beam backprojection in 6.8 s .