In this paper we present a semi-automatic 2D-3D local registration pipeline capable of coloring 3D models obtained from 3D scanners by using uncalibrated images. The proposed pipeline exploits the Structure from Motion (SfM) technique in order to reconstruct a sparse representation of the 3D object and obtain the camera parameters from image feature matches. We then coarsely register the reconstructed 3D model to the scanned one through the Scale Iterative Closest Point (SICP) algorithm. SICP provides the global scale, rotation and translation parameters, using minimal manual user intervention. In the final processing stage, a local registration refinement algorithm optimizes the color projection of the aligned photos on the 3D object removing the blurring ghosting artefacts introduced due to small inaccuracies during the registration. The proposed pipeline is capable of handling real world cases with a range of characteristics from objects with low level geometric features to complex ones.
Numerous techniques exist and a number of different ways exist to try to solve this problem. The methods can be classified into (i) manual, (ii) automatic or semi-automatic depending mainly on matches or features. In the (i) manual methods the registration is performed manually selecting correspondences between each image and the 3D geometry. This technique is often used for medical applications @cite_6 . Others instead, have used features in order to automate the process, but finding consistent correspondences is a very complex problem. Due to the different appearance of photographs and geometric models, (ii) automatic methods are limited to some specific models and information. For example, line features are mostly used for urban environments @cite_33 , @cite_16 ; and silhouette information is used when the contour of the objects is visible in the images and the 3D model projected onto an image plane @cite_12 @cite_23 @cite_34 .
@cite_33 @cite_6 @cite_34 @cite_23 @cite_16 @cite_12
2334682945 1970132064 2056798562 2074208271
The paper addresses the problem of camera tracking, which denotes the continuous image-based computation of a camera’s position and orientation with respect to a reference frame. The method aims at regular cameras, which means that 3D-3D registration methods applicable to RGB-D cameras are not an option. The tracked frame contains only 2D information, thus requiring a solution to the absolute pose or 2D-3D registration problem. While traditional solutions to camera tracking [3] rely on sparse feature correspondences, the community has recently seen a number of direct photometric registration methods such as [8] and [1]. [1] is conceptually similar to [8], however gains computational efficiency by reducing the computation from dense to semi-dense regions that correspond to a thresholded edge-map of the image. Photometric methods have the more general advantage of compensating for appearance variations caused by perspective view-point changes, whereas classical sparse methods often rely on static feature descriptors only (providing at most rotation and scale invariant properties [5, 6]). However, photometric registration techniques inherently suffer from the disability to overcome large disparities, where large sometimes means even just a couple of pixels [7]. Many photometric registration techniques therefore depend on pyramidal subsampling schemes in order to alleviate this problem. The goal of the present paper is a novel 2D-3D registration paradigm for semi-dense depth maps that relies on the Iterative Closest Point (ICP) technique, and thus a reintroduction of geometric error minimization as a valid alternative for real-time monocular camera tracking in the case of semi-dense features. An example semi-dense depth map is indicated in Figure 1. In comparison to photometric registration techniques, our ICP technique has the conceptual advantage of requiring neither isotropic enlargement of the employed semi-dense regions, nor pyramidal subsampling. The work is in line with [2], Tomono [9], and Klein and Murray [4], which already attempt curve or edge registration in 2D using ICP. Based on a hypothesized relative pose, the basic idea consists of warping a reference curve into the tracked image based on a prior 3D model or depth (in our case semi-dense) inside a reference frame. From a mathematical point of view, our idea may be formulated as follows. Let To generate a 3D computer model of a free-form object, multiple range images (point clouds) covering its entire surface are acquired from different viewpoints. These views are then aligned in a common coordinate basis by minimizing the distance error between their corresponding points. Establishing correspondences automatically is an inherently challenging problem due to the lack of any type of information other than the geometrical information extracted from the point clouds. Existing "automatic" correspondence techniques achieve automatism at the expense of other important specifications namely, applicability to free-form objects, accuracy, efficiency, robustness to resolution and surface sampling, robustness to overlap, robustness to noise and finally their applicability to simultaneous multiview correspondence. There is also a lack of a review paper that describes and critically analyzes these techniques. In this paper, we present such an extensive review and carry out the analysis of each technique according to the above listed indispensable criteria. Our analysis shows that none of these techniques fully meets these criteria and that there is still a need for the development of practical automatic correspondence algorithms. In this paper a system is presented that automatically registers and stitches textures acquired from multiple photographic images onto the surface of a given corresponding 3D model. Within this process the camera position, direction, and field of view must be determined for each of the images. For this registration, which aligns a 2D image to a 3D model, we present an efficient hardware-accelerated silhouette-based algorithm working on different image resolutions that accurately registers each image without any user interaction. Besides the silhouettes, the given texture information also can be used to improve accuracy by comparing one stitched texture to already registered images resulting in a global multiview optimization. After the 3D?2D registration for each part of the 3D model's surface the view is determined which provides the best available texture. Textures are blended at the borders of regions assigned to different views. We present a framework for robustly estimating registration between a 3D volume image and a 2D projection image and evaluate its precision and robustness in spine interventions for vertebral localization in the presence of anatomical deformation. The framework employs a normalized gradient information similarity metric and multi-start covariance matrix adaptation evolution strategy optimization with local-restarts, which provided improved robustness against deformation and content mismatch. The parallelized implementation allowed orders-of-magnitude acceleration in computation time and improved the robustness of registration via multi-start global optimization. Experiments involved a cadaver specimen and two CT datasets (supine and prone) and 36 C-arm fluoroscopy images acquired with the specimen in four positions (supine, prone, supine with lordosis, prone with kyphosis), three regions (thoracic, abdominal, and lumbar), and three levels of geometric magnification (1.7, 2.0, 2.4). Registration accuracy was evaluated in terms of projection distance error (PDE) between the estimated and true target points in the projection image, including 14 400 random trials (200 trials on the 72 registration scenarios) with initialization error up to ±200 mm and ±10°. The resulting median PDE was better than 0.1 mm in all cases, depending somewhat on the resolution of input CT and fluoroscopy images. The cadaver experiments illustrated the tradeoff between robustness and computation time, yielding a success rate of 99.993 in vertebral labeling (with 'success' defined as PDE <5 mm) using 1,718 664 ± 96 582 function evaluations computed in 54.0 ± 3.5 s on a mid-range GPU (nVidia, GeForce GTX690). Parameters yielding a faster search (e.g., fewer multi-starts) reduced robustness under conditions of large deformation and poor initialization (99.535 success for the same data registered in 13.1 s), but given good initialization (e.g., ±5 mm, assuming a robust initial run) the same registration could be solved with 99.993 success in 6.3 s. The ability to register CT to fluoroscopy in a manner robust to patient deformation could be valuable in applications such as radiation therapy, interventional radiology, and an assistant to target localization (e.g., vertebral labeling) in image-guided spine surgery.