Many telepresence robots are equipped with a forward-facing camera for video communication and a downward-facing camera for navigation. In this paper, we propose to stitch videos from the FF-camera with a wide-angle lens and the DF-camera with a fisheye lens for telepresence robots. We aim at providing more compact and efficient visual feedback for the user interface of telepresence robots with user-friendly interactive experiences. To this end, we present a multi-homography-based video stitching method which stitches videos from a wide-angle camera and a fisheye camera. The method consists of video image alignment, seam cutting, and image blending. We directly align the wide-angle video image and the fisheye video image based on the multi-homography alignment without calibration, distortion correction, and unwarping procedures. Thus, we can obtain a stitched video with shape preservation in the non-overlapping regions and alignment in the overlapping area for telepresence. To alleviate ghosting effects caused by moving objects and or moving cameras during telepresence robot driving, an optimal seam is found for aligned video composition, and the optimal seam will be updated in subsequent frames, considering spatial and temporal coherence. The final stitched video is created by image blending based on the optimal seam. We conducted a user study to demonstrate the effectiveness of our method and the superiority of telepresence robots with a stitched video as visual feedback.
The alignment of wide-angle images and or fisheye images is more challenging, and suffers from heavy radial distortion @cite_37 . In order for bringing wide-angle images into alignment, Jin @cite_14 and Byr " @cite_17 estimated jointly the lens distortion and the alignment transformation, assuming that all images share the same distortion factors. For cameras in different radial distortions, Ju and Kang @cite_1 estimated the lens distortion factor for each image, and then computed a homography for the alignment of synthetic images, whereas @cite_2 estimated a homography and different distortion factors to bring images of real scene into alignment. In addition, Ho and Budagavi @cite_18 proposed to align two images captured by a dual-fisheye lens camera. They unwarped the fisheye images into spherical 2-Dimensional space, and then employed a two-step alignment to register the unwarped images. Due to the unwarping process, the regions near edges of original images are stretched, leading to shape distortions in alignment results.
@cite_37 @cite_14 @cite_18 @cite_1 @cite_2 @cite_17
2105060035 2789430240 2564804471 2592756114 2021581587 1910129379 2773774647 2075544016 2202251471 2796773185 2044439645 2144325227 2052094314 2809451359 2059971317 1957250183 2090518410 2114113040 2082344075 2171244244
The use of super-wide angle and fish-eye lenses causes strong distortions in the resulting images. A methodology for the correction of distortions in these cases using only single images and linearity of imaged objects is presented. Contrary to most former algorithms, the algorithm discussed here does not depend on information about the real world co-ordinates of matching points. Moreover reference points determination and camera calibration is not required in this case. The algorithm is based on circle fitting. It requires only the possibility of the extraction of distorted image points from straight lines in the 3D scene. Further, the actual distortion must approximately fit the chosen distortion model. For most fish-eye lenses appropriate distortion correction results can be obtained. For wide-angle images with heavy lens distortion, especially those widely used in multi-camera systems, the single global homography cannot satisfy the required warp due to nonlinear distortions, leading to misalignment and shape distortion. This paper proposes a multi-homography warping to stitch wide-angle images with unknown lens distortion, which integrates multiple local homographies with a global homography for accurate alignment and shape preservation. We suggest a solution by conditional sampling to obtain a larger proportion of inliers for more accurate estimation of local and global projective transformations. We introduce an adaptive weighting scheme to combine these transformations for smoothing our warp over the entire target image from the local homographies to the global homography. The experiments evaluate the alignment accuracy and shape preservation of the proposed method. We present a direct visual odometry algorithm for a fisheye-stereo camera. Our algorithm performs simultaneous camera motion estimation and semi-dense reconstruction. The pipeline consists of two threads: a tracking thread and a mapping thread. In the tracking thread, we estimate the camera pose via semi-dense direct image alignment. To have a wider field of view (FoV) which is important for robotic perception, we use fisheye images directly without converting them to conventional pinhole images which come with a limited FoV. To address the epipolar curve problem, plane-sweeping stereo is used for stereo matching and depth initialization. Multiple depth hypotheses are tracked for selected pixels to better capture the uncertainty characteristics of stereo matching. Temporal motion stereo is then used to refine the depth and remove false positive depth hypotheses. Our implementation runs at an average of 20 Hz on a low-end PC. We run experiments in outdoor environments to validate our algorithm, and discuss the experimental results. We experimentally show that we are able to estimate 6D poses with low drift, and at the same time, do semi-dense 3D reconstruction with high accuracy. To the best of our knowledge, there is no other existing semi-dense direct visual odometry algorithm for a fisheye-stereo camera. Radial lens distortion often exists in images taken by common cameras, which violates the assumption of pinhole camera model. Estimating the radial lens distortion of an image is an important preprocessing step for many vision applications. This paper intends to employ CNNs (Convolutional Neural Networks), to achieve radial distortion correction. However, the main issue hinder its progress is the scarcity of training data with radial distortion annotations. Inspired by the growing availability of image dataset with non-radial distortion, we propose a framework to address the issue by synthesizing images with radial distortion for CNNs. We believe that a large number of images of high variation of radial distortion is generated, which can be well exploited by deep CNN with a high learning capacity. We present quantitative results that demonstrate the ability of our technique to estimate the radial distortion with comparisons against several baseline methods, including an automatic method based on Hough transforms of distorted line images. We present a method to automatically correct the radial distortion caused by wide-angle lenses using the distorted lines generated by the projection of 3D straight lines onto the image. Lens distortion is estimated by the division model using one parameter, which allows to state the problem into the Hough transform scheme by adding a distortion parameter to better extract straight lines from the image. This paper describes an algorithm which applies this technique, providing all the details of the design of an improved Hough transform. We perform experiments using calibration patterns and real scenes showing a strong distortion to illustrate the performance of the proposed method. Source Code The source code, the code documentation, and the online demo are accessible at the IPOL web page of this article 1 . In this page, an implementation is available for download. Compilation and usage instructions are included in the README.txt file of the archive. Fisheye image rectification and estimation of intrinsic parameters for real scenes have been addressed in the literature by using line information on the distorted images. In this paper, we propose an easily implemented fisheye image rectification algorithm with line constrains in the undistorted perspective image plane. A novel Multi-Label Energy Optimization (MLEO) method is adopted to merge short circular arcs sharing the same or the approximately same circular parameters and select long circular arcs for camera rectification. Further we propose an efficient method to estimate intrinsic parameters of the fisheye camera by automatically selecting three properly arranged long circular arcs from previously obtained circular arcs in the calibration procedure. Experimental results on a number of real images and simulated data show that the proposed method can achieve good results and outperforms the existing approaches and the commercial software in most cases. We propose a novel omnidirectional stereo camera setup that is formed by two ultra-wide field-of-view (FOV) fisheye cameras. The proposed configuration is formed by two 245-degree FOV fisheye cameras, facing opposite directions, that are rigidly mounted on two sides of a rod. The overlapping view in the two fisheye images forms a ring-shaped spatial stereo setup. Our system provides stereo observations with full 360-degree FOV in horizontal directions and 65-degree FOV in the vertical direction. In addition, the two fisheye cameras altogether also provide full spherical monocular coverage of the surrounding environment. We address challenges in camera modeling, fisheye intrinsic calibration, stereo self-calibration, and depth estimation. We develop a lens-specific fisheye camera calibration method that uses manufacturer data for the optical lens to assist with the intrinsic calibration and develop an online self-calibration approach for estimating stereo extrinsic parameters. The overlapping camera views are rectified into stereo image pairs, from which a spatial stereo matching pipeline is developed for depth estimation in all horizontal directions. We show both qualitative and quantitative analysis to validate our approach. Many computer vision algorithms rely on the assumptions of the pinhole camera model, but lens distortion with off-the-shelf cameras is usually significant enough to violate this assumption. Many methods for radial distortion estimation have been proposed, but they all have limitations. Robust automatic radial distortion estimation from a single natural image would be extremely useful for many applications, particularly those in human-made environments containing abundant lines. For example, it could be used in place of an extensive calibration procedure to get a mobile robot or quadrotor experiment up and running quickly in an indoor environment. We propose a new method for automatic radial distortion estimation based on the plumb-line approach. The method works from a single image and does not require a special calibration pattern. It is based on Fitzgibbon's division model, robust estimation of circular arcs, and robust estimation of distortion parameters. We perform an extensive empirical study of the method on synthetic images. We include a comparative statistical analysis of how different circle fitting methods contribute to accurate distortion parameter estimation. We finally provide qualitative results on a wide variety of challenging real images. The experiments demonstrate the method's ability to accurately identify distortion parameters and remove distortion from images. We propose a real-time, direct monocular SLAM method for omnidirectional or wide field-of-view fisheye cameras. Both tracking (direct image alignment) and mapping (pixel-wise distance filtering) are directly formulated for the unified omnidirectional model, which can model central imaging devices with a field of view above 180 °. This is in contrast to existing direct mono-SLAM approaches like DTAM or LSD-SLAM, which operate on rectified images, in practice limiting the field of view to around 130 ° diagonally. Not only does this allows to observe - and reconstruct - a larger portion of the surrounding environment, but it also makes the system more robust to degenerate (rotation-only) movement. The two main contribution are (1) the formulation of direct image alignment for the unified omnidirectional model, and (2) a fast yet accurate approach to incremental stereo directly on distorted images. We evaluated our framework on real-world sequences taken with a 185 ° fisheye lens, and compare it to a rectified and a piecewise rectified approach. Images captured by fisheye lenses violate the pinhole camera assumption and suffer from distortions. Rectification of fisheye images is therefore a crucial preprocessing step for many computer vision applications. In this paper, we propose an end-to-end multi-context collaborative deep network for removing distortions from single fisheye images. In contrast to conventional approaches, which focus on extracting hand-crafted features from input images, our method learns high-level semantics and low-level appearance features simultaneously to estimate the distortion parameters. To facilitate training, we construct a synthesized dataset that covers various scenes and distortion parameter settings. Experiments on both synthesized and real-world datasets show that the proposed model significantly outperforms current state of the art methods. Our code and synthesized dataset will be made publicly available. In short-exposure imaging through turbulence, there is some probability that the image will be nearly diffraction limited because the instantaneous wave-front distortion over the aperture was negligible. A number of years ago in a rather brief paper, Hufnagel (1966) argued heuristically that the probability of getting a good image would decrease exponentially with aperture area. This paper undertakes a rigorous quantitative analysis of the probability. We find that the probability of obtaining a good short-exposure image is Prob ≈ 5.6 exp[−0.1557 (D r0)2] (for D r0 ≥ 3.5), where D is the aperture diameter and r0 is the coherence length of the distorted wave front, as defined by Fried (1967). A good image is taken to be one for which the squared wave-front distortion over the aperture is 1 rad2 or less. The analysis is based on the decomposition of the distorted wave front over the aperture, in an orthonormal series with randomly independent coefficients. The orthonormal functions used are the eigenfunctions of a Karhunen-Loeve integral equation. The integral equation is solved using a separation of variables into radial and azimuthal dependence. The azimuthal dependence was solved analytically and the radial, numerically. The first 569 radial eigenfunctions and eigenvalues were obtained. The probability of obtaining a good short-exposure image corresponds to a hyperspace integral in which the spatial dimensions are the independent random coefficients in the orthonormal series expansion. It is equal to the probability that a randomly chosen point in the hyperspace will lie within a hypersphere of unit radius, the points in the hyperspace being randomly chosen in accordance with the product of independent Gaussian probability distribution—one distribution for each dimension. The variance of these distrbutions is directly proportional to the eigenvalues of the Karhunen-Loeve equation. This hyperspace integral (involving up to several hundred dimensions) has been evaluated using Monte Carlo techniques. We introduce a new rational function (RF) model for radial lens distortion in wide-angle and catadioptric lenses, which allows the simultaneous linear estimation of motion and lens geometry from two uncalibrated views of a 3D scene. In contrast to existing models which admit such linear estimates, the new model is not specialized to any particular lens geometry, but is sufficiently general to model a variety of extreme distortions. The key step is to define the mapping between image (pixel) coordinates and 3D rays in camera coordinates as a linear combination of nonlinear functions of the image coordinates. Like a "kernel trick", this allows a linear algorithm to estimate nonlinear models, and in particular offers a simple solution to the estimation of nonlinear image distortion. The model also yields an explicit form for the epipolar curves, allowing correspondence search to be efficiently guided by the epipolar geometry. We show results of an implementation of the RF model in estimating the geometry of a real camera lens from uncalibrated footage, and compare the estimate to one obtained using a calibration grid. If a physical object has a smooth or piecewise smooth boundary, its images obtained by cameras in varying positions undergo smooth apparent deformations. These deformations are locally well approximated by affine transforms of the image plane. In consequence the solid object recognition problem has often been led back to the computation of affine invariant image local features. Such invariant features could be obtained by normalization methods, but no fully affine normalization method exists for the time being. Even scale invariance is dealt with rigorously only by the scale-invariant feature transform (SIFT) method. By simulating zooms out and normalizing translation and rotation, SIFT is invariant to four out of the six parameters of an affine transform. The method proposed in this paper, affine-SIFT (ASIFT), simulates all image views obtainable by varying the two camera axis orientation parameters, namely, the latitude and the longitude angles, left over by the SIFT method. Then it covers the other four parameters by using the SIFT method itself. The resulting method will be mathematically proved to be fully affine invariant. Against any prognosis, simulating all views depending on the two camera orientation parameters is feasible with no dramatic computational load. A two-resolution scheme further reduces the ASIFT complexity to about twice that of SIFT. A new notion, the transition tilt, measuring the amount of distortion from one view to another, is introduced. While an absolute tilt from a frontal to a slanted view exceeding 6 is rare, much higher transition tilts are common when two slanted views of an object are compared (see Figure hightransitiontiltsillustration). The attainable transition tilt is measured for each affine image comparison method. The new method permits one to reliably identify features that have undergone transition tilts of large magnitude, up to 36 and higher. This fact is substantiated by many experiments which show that ASIFT significantly outperforms the state-of-the-art methods SIFT, maximally stable extremal region (MSER), Harris-affine, and Hessian-affine. We propose a novel real-time direct monocular visual odometry for omnidirectional cameras. Our method extends direct sparse odometry by using the unified omnidirectional model as a projection function, which can be applied to fisheye cameras with a field-of-view (FoV) well above 180 @math . This formulation allows for using the full area of the input image even with strong distortion, while most existing visual odometry methods can only use a rectified and cropped part of it. Model parameters within an active keyframe window are jointly optimized, including the intrinsic extrinsic camera parameters, three-dimensional position of points, and affine brightness parameters. Thanks to the wide FoV, image overlap between frames becomes bigger and points are more spatially distributed. Our results demonstrate that our method provides increased accuracy and robustness over state-of-the-art visual odometry algorithms. In this paper, we propose a new method to compensate for lens distortion in image stitching. Lens distortions that arise from the nonlinearity of a lens are the main causes for mismatches in stitching images. We estimate the distortion factors for each image using the Division Model and linearize the projected relationships between matching distorted feature points. Because our method works at the RANSAC stage, the estimated distortion factors are further refined during the bundle adjustment phase, and thus accurate distortion factors are obtained. Our experimental results show that the proposed method can efficiently and accurately estimate the distortion factors, and result in a more accurate stitched image than other previous methods based on estimated lens distortion factors. The importance of precise homography estimation is often underestimated even though it plays a crucial role in various vision applications such as plane or planarity detection, scene degeneracy tests, camera motion classification, image stitching, and many more. Ignoring the radial distortion component in homography estimation—even for classical perspective cameras—may lead to significant errors or totally wrong estimates. In this paper, we fill the gap among the homography estimation methods by presenting two algorithms for estimating homography between two cameras with different radial distortions. Both algorithms can handle planar scenes as well as scenes where the relative motion between the cameras is a pure rotation. The first algorithm uses the minimal number of five image point correspondences and solves a nonlinear system of polynomial equations using Grobner basis method. The second algorithm uses a non-minimal number of six image point correspondences and leads to a simple system of two quadratic equations in two unknowns and one system of six linear equations. The proposed algorithms are fast, stable, and can be efficiently used inside a RANSAC loop. While image alignment has been studied in different areas of computer vision for decades, aligning images depicting different scenes remains a challenging problem. Analogous to optical flow, where an image is aligned to its temporally adjacent frame, we propose SIFT flow, a method to align an image to its nearest neighbors in a large image corpus containing a variety of scenes. The SIFT flow algorithm consists of matching densely sampled, pixelwise SIFT features between two images while preserving spatial discontinuities. The SIFT features allow robust matching across different scene object appearances, whereas the discontinuity-preserving spatial model allows matching of objects located at different parts of the scene. Experiments show that the proposed approach robustly aligns complex scene pairs containing significant spatial differences. Based on SIFT flow, we propose an alignment-based large database framework for image analysis and synthesis, where image information is transferred from the nearest neighbors to a query image according to the dense scene correspondence. This framework is demonstrated through concrete applications such as motion field prediction from a single image, motion synthesis via object transfer, satellite image registration, and face recognition. This paper studies the problem of simultaneously aligning a batch of linearly correlated images despite gross corruption (such as occlusion). Our method seeks an optimal set of image domain transformations such that the matrix of transformed images can be decomposed as the sum of a sparse matrix of errors and a low-rank matrix of recovered aligned images. We reduce this extremely challenging optimization problem to a sequence of convex programs that minimize the sum of l1-norm and nuclear norm of the two component matrices, which can be efficiently solved by scalable convex optimization techniques with guaranteed fast convergence. We verify the efficacy of the proposed robust alignment algorithm with extensive experiments with both controlled and uncontrolled real data, demonstrating higher accuracy and efficiency than existing methods over a wide range of realistic misalignments and corruptions. Tomographic image reconstruction, such as the reconstruction of computed tomography projection values, of tomosynthesis data, positron emission tomography or SPECT events, and of magnetic resonance imaging data is computationally very demanding. One of the most time-consuming steps is the backprojection. Recently, a novel general purpose architecture optimized for distributed computing became available: the cell broadband engine (CBE). To maximize image reconstruction speed we modified our parallel-beam backprojection algorithm [two dimensional (2D)] and our perspective backprojection algorithm [three dimensional (3D), cone beam for flat–panel detectors] and optimized the code for the CBE. The algorithms are pixel or voxel driven, run with floating point accuracy and use linear (LI) or nearest neighbor (NN) interpolation between detector elements. For the parallel-beam case, 512 projections per half rotation, 1024 detector channels, and an image of size 512 2 was used. The cone-beam backprojection performance was assessed by backprojecting a full circle scan of 512 projections of size 1024 2 into a volume of size 512 3 voxels. The field of view was chosen to completely lie within the field of measurement and the pixel or voxel size was set to correspond to the detector element size projected to the center of rotation divided by 2 . Both the PC and the CBE were clocked at 3 GHz . For the parallel backprojection of 512 projections into a 512 2 image, a throughput of 11 fps (LI) and 15 fps (NN) was measured on the PC, whereas the CBE achieved 126 fps (LI) and 165 fps (NN), respectively. The cone-beam backprojection of 512 projections into the 512 3 volume took 3.2 min on the PC and is as fast as 13.6 s on the cell. Thereby, the cell greatly outperforms today’s top-notch backprojections based on graphical processing units. Using both CBEs of our dual cell-based blade (Mercury Computer Systems) allows to 2D backproject 330 images s and one can complete the 3D cone-beam backprojection in 6.8 s . It is known that the problem of multiview reconstruction can be solved in two steps: first estimate camera rotations and then translations using them. This paper presents new robust techniques for both of these steps, (i) Given pair-wise relative rotations, global camera rotations are estimated linearly in least squares, (ii) Camera translations are estimated using a standard technique based on Second Order Cone Programming. Robustness is achieved by using only a subset of points according to a new criterion that diminishes the risk of choosing a mismatch. It is shown that only four points chosen in a special way are sufficient to represent a pairwise reconstruction almost equally as all points. This leads to a significant speedup. In image sets with repetitive or similar structures, non-existent epipolar geometries may be found. Due to them, some rotations and consequently translations may be estimated incorrectly. It is shown that iterative removal of pairwise reconstructions with the largest residual and reregistration removes most non-existent epipolar geometries. The performance of the proposed method is demonstrated on difficult wide base-line image sets.