Dense and accurate 3-D mapping from a monocular sequence is a key technology for several applications and still an open research area. This letter leverages recent results on single-view convolutional network (CNN)-based depth estimation and fuses them with multiview depth estimation. Both approaches present complementary strengths. Multiview depth is highly accurate but only in high-texture areas and high-parallax cases. Single-view depth captures the local structure of midlevel regions, including texture-less areas, but the estimated depth lacks global coherence. The single and multiview fusion we propose is challenging in several aspects. First, both depths are related by a deformation that depends on the image content. Second, the selection of multiview points of high accuracy might be difficult for low-parallax configurations. We present contributions for both problems. Our results in the public datasets of NYUv2 and TUM shows that our algorithm outperforms the individual single and multiview approaches. A video showing the key aspects of mapping in our single and multiview depth proposal is available at https: youtu.be ipc5HukTb4k.
TV regularization has low accuracy for large textureless areas, as shown recently in @cite_15 @cite_3 @cite_25 among others. In order to overcome this @cite_15 proposes a piecewise-planar regularization; the plane parameters coming from multi-view superpixel triangulation @cite_18 or layout estimation @cite_0 . @cite_3 proposes higher-order regularization terms that enforce piecewise affine constraints even in separated pixels. @cite_25 selects the best regularization function among a set using sparse laser data. Building on @cite_15 , @cite_19 adds the sparse data-driven 3D primitives of @cite_9 as a regularization prior. Compared to these works, our fusion is the first one where the information added to the multi-view depth is fully dense, data-driven and single-view; and hence it does not rely on additional sensors, parallax or Manhattan and piecewise-planar assumptions. It only relies on the network capabilities for the current domain, assuming that the test data follows the same distribution that the data used for training.
@cite_18 @cite_9 @cite_3 @cite_0 @cite_19 @cite_15 @cite_25
2462462929 2394774286 1997409415 2800554784
In this paper, we propose a non-local structured prior for volumetric multi-view 3D reconstruction. Towards this goal, we present a novel Markov random field model based on ray potentials in which assumptions about large 3D surface patches such as planarity or Manhattan world constraints can be efficiently encoded as probabilistic priors. We further derive an inference algorithm that reasons jointly about voxels, pixels and image segments, and estimates marginal distributions of appearance, occupancy, depth, normals and planarity. Key to tractable inference is a novel hybrid representation that spans both voxel and pixel space and that integrates non-local information from 2D image segmentations in a principled way. We compare our non-local prior to commonly employed local smoothness assumptions and a variety of state-of-the-art volumetric reconstruction baselines on challenging outdoor scenes with textureless and reflective surfaces. Our experiments indicate that regularizing over larger distances has the potential to resolve ambiguities where local regularizers fail. The development of multisensor systems in recent years has led to great increase in the amount of available remote sensing data. Image fusion techniques aim at inferring high quality images of a given area from degraded versions of the same area obtained by multiple sensors. This paper focuses on pansharpening, which is the inference of a high spatial resolution multispectral image from two degraded versions with complementary spectral and spatial resolution characteristics: 1) a low spatial resolution multispectral image and 2) a high spatial resolution panchromatic image. We introduce a new variational model based on spatial and spectral sparsity priors for the fusion. In the spectral domain, we encourage low-rank structure, whereas in the spatial domain, we promote sparsity on the local differences. Given the fact that both panchromatic and multispectral images are integrations of the underlying continuous spectra using different channel responses, we propose to exploit appropriate regularizations based on both spatial and spectral links between panchromatic and fused multispectral images. A weighted version of the vector total variation norm of the data matrix is employed to align the spatial information of the fused image with that of the panchromatic image. With regard to spectral information, two different types of regularization are proposed to promote a soft constraint on the linear dependence between the panchromatic and fused multispectral images. The first one estimates directly the linear coefficients from the observed panchromatic and low-resolution multispectral images by linear regression while the second one employs the principal component pursuit to obtain a robust recovery of the underlying low-rank structure. We also show that the two regularizers are strongly related. The basic idea of both regularizers is that the fused image should have low-rank and preserve edge locations. We use a variation of the recently proposed split augmented Lagrangian shrinkage algorithm to effectively solve the proposed variational formulations. Experimental results on simulated and real remote sensing images show the effectiveness of the proposed pansharpening method compared with the state-of-the-art. TV regularization is an L1 penalization of the flow gradient magnitudes, and due to the tendency of the L1 norm to favor sparse solutions (i.e. lots of ‘zeros’), the fill-in effect caused by the regularizer leads to piecewise constant solutions in weakly textured areas. This effect, known as ‘staircasing’ in a 1D setting, can be reduced significantly by using a quadratic penalization for small gradient magnitudes while sticking to linear penalization for larger magnitudes to maintain the discontinuity preserving properties known from TV. A comparison of isotropic TV and isotropic Huber regularity is shown in Fig. 1 by means of rendering the disparities u1 of the Dimetrodon dataset. The color coded flow (cf. Fig. 1(a)) is superimposed as texture. Based on the two observations that motion discontinuities often occur along object boundaries and that in turn object boundaries often coincide The low spatial resolution of light-field image poses significant difficulties in exploiting its advantage. To mitigate the dependency of accurate depth or disparity information as priors for light-field image super-resolution, we propose an implicitly multi-scale fusion scheme to accumulate contextual information from multiple scales for super-resolution reconstruction. The implicitly multi-scale fusion scheme is then incorporated into bidirectional recurrent convolutional neural network, which aims to iteratively model spatial relations between horizontally or vertically adjacent sub-aperture images of light-field data. Within the network, the recurrent convolutions are modified to be more effective and flexible in modeling the spatial correlations between neighboring views. A horizontal sub-network and a vertical sub-network of the same network structure are ensembled for final outputs via stacked generalization. Experimental results on synthetic and real-world data sets demonstrate that the proposed method outperforms other state-of-the-art methods by a large margin in peak signal-to-noise ratio and gray-scale structural similarity indexes, which also achieves superior quality for human visual systems. Furthermore, the proposed method can enhance the performance of light field applications such as depth estimation.