Many telepresence robots are equipped with a forward-facing camera for video communication and a downward-facing camera for navigation. In this paper, we propose to stitch videos from the FF-camera with a wide-angle lens and the DF-camera with a fisheye lens for telepresence robots. We aim at providing more compact and efficient visual feedback for the user interface of telepresence robots with user-friendly interactive experiences. To this end, we present a multi-homography-based video stitching method which stitches videos from a wide-angle camera and a fisheye camera. The method consists of video image alignment, seam cutting, and image blending. We directly align the wide-angle video image and the fisheye video image based on the multi-homography alignment without calibration, distortion correction, and unwarping procedures. Thus, we can obtain a stitched video with shape preservation in the non-overlapping regions and alignment in the overlapping area for telepresence. To alleviate ghosting effects caused by moving objects and or moving cameras during telepresence robot driving, an optimal seam is found for aligned video composition, and the optimal seam will be updated in subsequent frames, considering spatial and temporal coherence. The final stitched video is created by image blending based on the optimal seam. We conducted a user study to demonstrate the effectiveness of our method and the superiority of telepresence robots with a stitched video as visual feedback.
The alignment of wide-angle images and or fisheye images is more challenging, and suffers from heavy radial distortion @cite_37 . In order for bringing wide-angle images into alignment, Jin @cite_14 and Byr " @cite_17 estimated jointly the lens distortion and the alignment transformation, assuming that all images share the same distortion factors. For cameras in different radial distortions, Ju and Kang @cite_1 estimated the lens distortion factor for each image, and then computed a homography for the alignment of synthetic images, whereas @cite_2 estimated a homography and different distortion factors to bring images of real scene into alignment. In addition, Ho and Budagavi @cite_18 proposed to align two images captured by a dual-fisheye lens camera. They unwarped the fisheye images into spherical 2-Dimensional space, and then employed a two-step alignment to register the unwarped images. Due to the unwarping process, the regions near edges of original images are stretched, leading to shape distortions in alignment results.
@cite_37 @cite_14 @cite_18 @cite_1 @cite_2 @cite_17
2105060035 2789430240 2564804471 2592756114
The use of super-wide angle and fish-eye lenses causes strong distortions in the resulting images. A methodology for the correction of distortions in these cases using only single images and linearity of imaged objects is presented. Contrary to most former algorithms, the algorithm discussed here does not depend on information about the real world co-ordinates of matching points. Moreover reference points determination and camera calibration is not required in this case. The algorithm is based on circle fitting. It requires only the possibility of the extraction of distorted image points from straight lines in the 3D scene. Further, the actual distortion must approximately fit the chosen distortion model. For most fish-eye lenses appropriate distortion correction results can be obtained. For wide-angle images with heavy lens distortion, especially those widely used in multi-camera systems, the single global homography cannot satisfy the required warp due to nonlinear distortions, leading to misalignment and shape distortion. This paper proposes a multi-homography warping to stitch wide-angle images with unknown lens distortion, which integrates multiple local homographies with a global homography for accurate alignment and shape preservation. We suggest a solution by conditional sampling to obtain a larger proportion of inliers for more accurate estimation of local and global projective transformations. We introduce an adaptive weighting scheme to combine these transformations for smoothing our warp over the entire target image from the local homographies to the global homography. The experiments evaluate the alignment accuracy and shape preservation of the proposed method. We present a direct visual odometry algorithm for a fisheye-stereo camera. Our algorithm performs simultaneous camera motion estimation and semi-dense reconstruction. The pipeline consists of two threads: a tracking thread and a mapping thread. In the tracking thread, we estimate the camera pose via semi-dense direct image alignment. To have a wider field of view (FoV) which is important for robotic perception, we use fisheye images directly without converting them to conventional pinhole images which come with a limited FoV. To address the epipolar curve problem, plane-sweeping stereo is used for stereo matching and depth initialization. Multiple depth hypotheses are tracked for selected pixels to better capture the uncertainty characteristics of stereo matching. Temporal motion stereo is then used to refine the depth and remove false positive depth hypotheses. Our implementation runs at an average of 20 Hz on a low-end PC. We run experiments in outdoor environments to validate our algorithm, and discuss the experimental results. We experimentally show that we are able to estimate 6D poses with low drift, and at the same time, do semi-dense 3D reconstruction with high accuracy. To the best of our knowledge, there is no other existing semi-dense direct visual odometry algorithm for a fisheye-stereo camera. Radial lens distortion often exists in images taken by common cameras, which violates the assumption of pinhole camera model. Estimating the radial lens distortion of an image is an important preprocessing step for many vision applications. This paper intends to employ CNNs (Convolutional Neural Networks), to achieve radial distortion correction. However, the main issue hinder its progress is the scarcity of training data with radial distortion annotations. Inspired by the growing availability of image dataset with non-radial distortion, we propose a framework to address the issue by synthesizing images with radial distortion for CNNs. We believe that a large number of images of high variation of radial distortion is generated, which can be well exploited by deep CNN with a high learning capacity. We present quantitative results that demonstrate the ability of our technique to estimate the radial distortion with comparisons against several baseline methods, including an automatic method based on Hough transforms of distorted line images.