Abstract This work is concerned with camera pose estimation from correspondences of 3D 2D lines, i. e. with the Perspective-n-Line (PnL) problem. We focus on large line sets, which can be efficiently solved by methods using linear formulation of PnL. We propose a novel method “DLT-Combined-Lines” based on the Direct Linear Transformation (DLT) algorithm, which benefits from a new combination of two existing DLT methods for pose estimation. The method represents 2D structure by lines, and 3D structure by both points and lines. The redundant 3D information reduces the minimum required line correspondences to 5. A cornerstone of the method is a combined projection matrix estimated by the DLT algorithm. It contains multiple estimates of camera rotation and translation, which can be recovered after enforcing constraints of the matrix. Multiplicity of the estimates is exploited to improve the accuracy of the proposed method. For large line sets (10 and more), the method is comparable to the state-of-the-art in accuracy of orientation estimation. It achieves state-of-the-art accuracy in estimation of camera position and it yields the smallest reprojection error under strong image noise. The method achieves top-3 results on real world data. The proposed method is also highly computationally effective, estimating the pose of 1000 lines in 12 ms on a desktop computer.
The first DLT method for solving PnL is the method of [ ] hartley2004multiple . Following the terminology of @cite_1 , we call the method . It does not act directly on 3D lines, but rather on 3D lying on 3D lines (for example line endpoints). It exploits the fact that if a 3D line and a 3D point coincide, their projections also must coincide. The DLT-Lines method requires at least 6 line correspondences.
@cite_1
2329631920
In this paper we approach the problem of in- situ camera calibration using an auxiliary mobile color-depth camera. The calibration is based on image lines and the Direct Linear Transformation (DLT). Calibration includes intrinsic and extrinsic parameters, and radial distortion. Traditionally, camera calibration methods based on DLT combine 2D and 3D points. Using 2D image lines allows adding simple image processing methods for fine tuning the calibration data (lines). Experimental comparison of points and lines based calibration shows that the automatic fine tuning of the lines is beneficial for the calibration process in noisy conditions. Experiments on both synthetic and real setups show that despite the low computational complexity of the DLT, the proposed calibration methodology yields promising accurate results. I. I NTRODUCTION In this paper we deal with the camera pose estimation problem from a set of 2D 3D line correspondences, which is also known as PnL (Perspective-n-Line) problem. We carry out our study by comparing PnL with the well-studied PnP (Perspective-n-Point) problem, and our contributions are three-fold: (1) We provide a complete 3D configuration analysis for P3L, which includes the well-known P3P problem as well as several existing analyses as special cases. (2) By exploring the similarity between PnL and PnP, we propose a new subset-based PnL approach as well as a series of linear-formulation-based PnL approaches inspired by their PnP counterparts. (3) The proposed linear-formulation-based methods can be easily extended to deal with the line and point features simultaneously. Correspondences between 3D lines and their 2D images captured by a camera are often used to determine position and orientation of the camera in space. In this work, we propose a novel algebraic algorithm to estimate the camera pose. We parameterize 3D lines using Plucker coordinates that allow linear projection of the lines into the image. A line projection matrix is estimated using Linear Least Squares and the camera pose is then extracted from the matrix. An algebraic approach to handle mismatched line correspondences is also included. The proposed algorithm is an order of magnitude faster yet comparably accurate and robust to the state-of-the-art, it does not require initialization, and it yields only one solution. The described method requires at least 9 lines and is particularly suitable for scenarios with 25 and more lines, as also shown in the results. We present a novel method for recovering the 3D-line structure of a scene from multiple widely separated views. Traditional optimization-based approaches to line-based structure from motion minimize the error between measured line segments and the projections of corresponding 3D lines. In such a case, 3D lines can be optimized using a minimum of 4 parameters. We show that this number of parameters can be further reduced by introducing additional constraints on the orientations of lines in a 3D scene. In our approach, 2D-lines are automatically detected in images with the assistance of an EM-based vanishing point estimation method which assumes the existence of edges along mutally orthogonal vanishing directions. Each detected line is automatically labeled with the orientation (e.g. vertical, horizontal) of the 3D line which generated the measurement, and it is this additional knowledge that we use to reduce the number of degrees of freedom of 3D lines during optimization. We present 3D reconstruction results for urban scenes based on manually established feature correspondences across images. We propose a minimal solution for the similarity registration (rigid pose and scale) between two sets of 3D lines, and also between a set of co-planar points and a set of 3D lines. The first problem is solved up to 8 discrete solutions with a minimum of 2 line-line correspondences, while the second is solved up to 4 discrete solutions using 4 point-line correspondences. We use these algorithms to perform the extrinsic calibration between a pose tracking sensor and a 2D 3D ultrasound (US) curvilinear probe using a tracked needle as calibration target. The needle is tracked as a 3D line, and is scanned by the ultrasound as either a 3D line (3D US) or as a 2D point (2D US). Since the scale factor that converts US scan units to metric coordinates is unknown, the calibration is formulated as a similarity registration problem. We present results with both synthetic and real data and show that the minimum solutions outperform the correspondent non-minimal linear formulations. Recovering 3D geometry from a single 2D line drawing is an important and challenging problem in computer vision. It has wide applications in interactive 3D modeling from images, computer-aided design, and 3D object retrieval. Previous methods of 3D reconstruction from line drawings are mainly based on a set of heuristic rules. They are not robust to sketch errors and often fail for objects that do not satisfy the rules. In this paper, we propose a novel approach, called example-based 3D object reconstruction from line drawings, which is based on the observation that a natural or man-made complex 3D object normally consists of a set of basic 3D objects. Given a line drawing, a graphical model is built where each node denotes a basic object whose candidates are from a 3D model (example) database. The 3D reconstruction is solved using a maximum-a-posteriori (MAP) estimation such that the reconstructed result best fits the line drawing. Our experiments show that this approach achieves much better reconstruction accuracy and are more robust to imperfect line drawings than previous methods. In this study, the authors have proposed a new solution for the problem of pose estimation from a set of matched 3D model and 2D image lines. Traditional line-based pose estimation methods utilising the finite information of the observations are based on the assumption that the noises for the two endpoints of the image line segment are statistically independent. However, in this study, the authors prove that these two noises are negatively correlative when the image line segment is fitted by the least-squares technique from the noisy edge points. Moreover, the authors derive the noise model describing the probabilistic relationship between the 3D model line and their finite image observations. Based on the proposed noise model, the maximum-likelihood approach is exploited to estimate the pose parameters. The authors have carried out synthetic experiments to compare the proposed method to other pose optimisation methods in the literature. The experimental results show that the proposed methods yield a clear higher precision than the traditional methods. The authors also use real image sequences to demonstrate the performance of the proposed method. We propose a non-iterative solution for the Perspective-n-Line (PnL) problem, which can efficiently and accurately estimate the camera pose for both small number and large number of line correspondences. By selecting a rotation axis in the camera framework, the reference lines are divided into triplets to form a sixteenth order cost function, and then the optimum is retrieved from the roots of the derivative of the cost function by evaluating the orthogonal errors and the reprojected errors of the local minima. The final pose estimation is normalized by a 3D alignment approach. The advantages of the proposed method are as follows: (1) it stably retrieves the optimum of the solution with very little computational complexity and high accuracy; (2) small line sets can be robustly handled to achieve highly accurate results and; (3) large line sets can be efficiently handled because it is O(n). Correspondences between 2D lines in an image and 3D lines in the surrounding environment can be exploited to determine the camera's position and attitude (pose). In this paper, we introduce a novel approach to estimate the camera's pose by directly solving the corresponding least-squares problem algebraically. Specifically, the optimality conditions of the least-squares problem form a system of polynomial equations, which we efficiently solve through the eigendecomposition of a so-called multiplication matrix. Contrary to existing methods, the proposed algorithm (i) is guaranteed to find the globally optimal estimate in the least-squares sense, (ii) does not require initialization, and (iii) has computational cost only linear in the number of measurements. The superior performance of the proposed algorithm compared to previous approaches is demonstrated through extensive simulations and experiments. Line segment detection in images is already a well-investigated topic, although it has received considerably less attention in 3D point clouds. Benefiting from current LiDAR devices, large-scale point clouds are becoming increasingly common. Most human-made objects have flat surfaces. Line segments that occur where pairs of planes intersect give important information regarding the geometric content of point clouds, which is especially useful for automatic building reconstruction and segmentation. This paper proposes a novel method that is capable of accurately extracting plane intersection line segments from large-scale raw scan points. The 3D line-support region, namely, a point set near a straight linear structure, is extracted simultaneously. The 3D line-support region is fitted by our Line-Segment-Half-Planes (LSHP) structure, which provides a geometric constraint for a line segment, making the line segment more reliable and accurate. We demonstrate our method on the point clouds of large-scale, complex, real-world scenes acquired by LiDAR devices. We also demonstrate the application of 3D line-support regions and their LSHP structures on urban scene abstraction. This paper identifies and fills the probably last two missing items in minimal pose estimation algorithms using points and lines. Pose estimation refers to the problem of recovering the pose of a calibrated camera given known features (points or lines) in the world and their projections on the image. There are four minimal configurations using point and line features: 3 points, 2 points and 1 line, 1 point and 2 lines, 3 lines. The first and the last scenarios that depend solely on either points or lines have been studied a few decades earlier. However the mixed scenarios, which are more common in practice, have not been solved yet. In this paper we show that it is indeed possible to develop a general technique that can solve all four scenarios using the same approach and that the solutions involve computing the roots of either a 4th degree or an 8th degree equation. The centerpiece of our method is a simple and generic method that uses collinearity and coplanarity constraints for solving the pose. In addition to validating the performance of these algorithms in simulations, we also show a compelling application for geo-localization using image sequences and coarse (plane-based) 3D models of GPS-challenged urban canyons. A method for finding analytical solutions to the problem of determining the attitude of a 3D object in space from a single perspective image is presented. Its principle is based on the interpretation of a triplet of any image lines as the perspective projection of a triplet of linear ridges of the object model, and on the search for the model attitude consistent with these projections. The geometrical transformations to be applied to the model to bring it into the corresponding location are obtained by the resolution of an eight-degree equation in the general case. Using simple logical rules, it is shown on examples related to polyhedra that this approach leads to results useful for both location and recognition of 3D objects because few admissible hypotheses are retained from the interpolation of the three line segments. Line matching by the prediction-verification procedure is thus less complex. > We address the problem of camera motion and 3D structure reconstruction from line correspondences across multiple views, from initialization to final bundle adjustment. One of the main difficulties when dealing with line features is their algebraic representation. First, we consider the triangulation problem. Based on Plucker coordinates to represent the 3D lines, we propose a maximum likelihood algorithm, relying on linearizing the Plucker constraint and on a Plucker correction procedure, computing the closest Plucker coordinates to a given 6-vector. Second, we consider the bundle adjustment problem, which is essentially a nonlinear optimization process on camera motion and 3D line parameters. Previous overparameterizations of 3D lines induce gauge freedoms and or internal consistency constraints. We propose the orthonormal representation, which allows handy nonlinear optimization of 3D lines using the minimum four parameters with an unconstrained optimization engine. We compare our algorithms to existing ones on simulated and real data. Results show that our triangulation algorithm outperforms standard linear and bias-corrected quasi-linear algorithms, and that bundle adjustment using our orthonormal representation yields results similar to the standard maximum likelihood trifocal tensor algorithm, while being usable for any number of views. This paper presents a new linear method for reconstructing simultaneously 3D features (points, lines and planes) and cameras from many perspective views by solving a single linear system. It assumes that a real or virtual reference plane is visible in all views. We call it the Direct Reference Plane (DRP) method. It is well known that the projection relationship between uncalibrated cameras and 3D features is nonlinear in the absence of a reference plane. With a known reference plane, points and cameras have a linear relationship, as shown by Rother and Carlsson (2001). The main contribution of this paper is that lines and cameras, as well as, planes and cameras also have a linear relationship. Consequently, all 3D features and all cameras can be reconstructed simultaneously from a single linear system, which handles missing image measurements naturally. A further contribution is an extensive experimental comparison, using real data, of different reference plane and nonreference plane reconstruction methods. For difficult reference plane scenarios, with point or line features, the DRP method is superior to all compared methods. Finally, an extensive list of reference plane scenarios is presented, which shows the wide applicability of the DRP method. Discovering and extracting new image features pertaining to scene geometry is important to 3D reconstruction and scene understanding. Examples include the classical vanishing points observed in a centric camera and the recent coplanar common points (CCPs) in a crossed-slit camera [21,17]. A CCP is a point in the image plane corresponding to the intersection of the projections of all lines lying on a common 3D plane. In this paper, we address the problem of determining CCP existence in general non-centric cameras. We first conduct a ray-space analysis to show that finding the CCP of a 3D plane is equivalent to solving an array of ray constraint equations. We then derive the necessary and sufficient conditions for CCP to exist in an arbitrary non-centric camera such as non-centric catadioptric mirrors. Finally, we present robust algorithms for extracting the CCPs from a single image and validate our theories and algorithms through experiments. We consider the problem of reconstructing the 3D coordinates of a moving point seen from a monocular moving camera, i.e., to reconstruct moving objects from line-of-sight measurements only. The task is feasible only when some constraints are placed on the shape of the trajectory of the moving point. We coin the family of such tasks as "trajectory triangulation." We investigate the solutions for points moving along a straight-line and along conic-section trajectories, We show that if the point is moving along a straight line, then the parameters of the line (and, hence, the 3D position of the point at each time instant) can be uniquely recovered, and by linear methods, from at least five views. For the case of conic-shaped trajectory, we show that generally nine views are sufficient for a unique reconstruction of the moving point and fewer views when the conic is of a known type (like a circle in 3D Euclidean space for which seven views are sufficient). The paradigm of trajectory triangulation, in general, pushes the envelope of processing dynamic scenes forward. Thus static scenes become a particular case of a more general task of reconstructing scenes rich with moving objects (where an object could be a single point). We propose a method to recover the shape of a 3D room from a full-view indoor panorama. Our algorithm can automatically infer a 3D shape from a collection of partially oriented superpixel facets and line segments. The core part of the algorithm is a constraint graph, which includes lines and superpixels as vertices, and encodes their geometric relations as edges. A novel approach is proposed to perform 3D reconstruction based on the constraint graph by solving all the geometric constraints as constrained linear least-squares. The selected constraints used for reconstruction are identified using an occlusion detection method with a Markov random field. Experiments show that our method can recover room shapes that can not be addressed by previous approaches. Our method is also efficient, that is, the inference time for each panorama is less than 1 minute. We present a new shape-from-distortion framework for recovering specular (reflective refractive) surfaces. While most existing approaches rely on accurate correspondences between 2D pixels and 3D points, we focus on analyzing the curved images of 3D lines which we call curved line images or CLIs. Our approach models CLIs of local reflections or refractions using the recently proposed general linear cameras (GLCs). We first characterize all possible CLIs in a GLC. We show that a 3D line will appear as a conic in any GLC. For a fixed GLC, the conic type is invariant to the position and orientation of the line and is determined by the GLC parameters. Furthermore, CLIs under single reflection refraction can only be lines or hyperbolas. Based on our new theory, we develop efficient algorithms to use multiple CLIs to recover the GLC camera parameters. We then apply the curvature-GLC theory to derive the Gaussian and mean curvatures from the GLC intrinsics. This leads to a complete distortion-based reconstruction framework. Unlike conventional correspondence-based approaches that are sensitive to image distortions, our approach benefits from the CLI distortions. Finally, we demonstrate applying our framework for recovering curvature fields on both synthetic and real specular surfaces. Large lighting variation challenges all visual odometry methods, even with RGB-D cameras. Here we propose a line segment-based RGB-D indoor odometry algorithm robust to lighting variation. We know line segments are abundant indoors and less sensitive to lighting change than point features. However, depth data are often noisy, corrupted or even missing for line segments which are often found on object boundaries where significant depth discontinuities occur. Our algorithm samples depth data along line segments, and uses a random sample consensus approach to identify correct depth and estimate 3D line segments. We analyze 3D line segment uncertainties and estimate camera motion by minimizing the Mahalanobis distance. In experiments we compare our method with two state-of-the-art methods including a keypoint-based approach and a dense visual odometry algorithm, under both constant and varying lighting. Our method demonstrates superior robustness to lighting change by outperforming the competing methods on 6 out of 8 long indoor sequences under varying lighting. Meanwhile our method also achieves improved accuracy even under constant lighting when tested using public data. A recent trend in graph drawing is directed to the visualization of graphs in 3D [1, 5, 6]. A promising research direction concerns the extension of proven 2D techniques to 3D. We present a system extending the simulated annealing algorithm of Davidson and Harel [2] for straight-line two-dimensional drawings of general undirected graphs to three dimensions. This system features an advanced 3D user interface that assists the user in choosing and modifying the cost function and the optimization components on-line.