For numerous domains, including for instance earth observation, medical imaging, astrophysics,..., available image and signal datasets often irregular space-time sampling patterns and large missing data rates. These sampling properties is a critical issue to apply state-of-the-art learning-based (e.g., auto-encoders, CNNs,...) to fully benefit from the available large-scale observations and reach breakthroughs in the reconstruction and identification of processes of interest. In this paper, we address the end-to-end learning of representations of signals, images and image sequences from irregularly-sampled data, {\em i.e.} when the training data involved missing data. From an analogy to Bayesian formulation, we consider energy-based representations. Two energy forms are investigated: one derived from auto-encoders and one relating to Gibbs energies. The learning stage of these energy-based representations (or priors) involve a joint interpolation issue, which resorts to solving an energy minimization problem under observation constraints. Using a neural-network-based implementation of the considered energy forms, we can state an end-to-end learning scheme from irregularly-sampled data. We demonstrate the relevance of the proposed representations for different case-studies: namely, multivariate time series, 2{\sc } images and image sequences. In numerous application domains, the available observation datasets do not involve gap-free and regularly-gridded signals or images. The irregular-sampling may result both from the characteristics of the sensors and sampling strategy, e.g. considered orbits and swaths in spacebone earth observation and astrophysics, sampling schemes in medical imaging, as well as environmental conditions which may affect the sensor, e.g. atmospheric conditions and clouds for earth observation. A rich literature exists on interpolation for irregularly-sampled signals and images (also referred to as inpainting in image processing (4)). A classic framework states the interpolation issue as the miminisation of an energy, which may be interpreted in a Bayesian framework. A variety of energy forms, including Markovian priors (12) , patch-based priors (20) , gradient norms in variational and/or PDE-based formulations (4), Gaussian priors () as well as dynamical priors in fluid dynamics (3) . The later relates to optimal interpolation and kriging (8) , which is among the state-of-the-art and operational schemes in geoscience (10) . Optimal schemes classically involve the inference of the considered covariance-based priors from irregularly-sampled data. This may however be at the expense of Gaussianity and linearity assumptions, which do not often apply for real signals and images. For the other types of energy forms, their parameterization are generally set a priori and not learnt from the data. Regarding more particularly data-driven and learning-based approaches, most previous works (2; 11; 20) have addressed the learning of interpolation schemes under the assumption that a representative gap-free dataset is available. This gap-free dataset may be the image itself (9; 20; 18) . For numerous application domains, as mentionned above, this assumption cannot be fulfilled. Regarding recent advances in learning-based schemes, a variety of deep learning models, e.g. (7; 16; 24; 23) , have been proposed. Most of these works focus on learning an interpolator. One may however expect to learn not only an interpolator but also some representation of considered data, which may be of interest for other applications. In this respect, RBM models (Restricted Boltzmann In this paper, we have addressed the learning of energy-based representations of signals and images from observation datasets involving missing data (with possibly very large missing data rates). Using the proposed architectures, we can jointly learn relevant representations of signals and images while jointly providing the associated interpolation schemes. Our experiments stress that learning representations from gap-free data may lead to representations poorly adapted to the analysis of data with large missing data areas. We have also introduced a Gibbs priors embedded in a neural network architecture. Relying on local characteristics rather than global ones as in AE schemes, these priors involve a much lower complexity. Our experiments support their relevance for addressing inverse problems in signal and image analysis. Future work may further explore multi-scale extensions of the proposed schemes along with couplings between global and local energy representations and hybrid minimization schemes combining both gradient-based and fixed-point strategies in the considered end-to-end formulation.
We address the end-to-end learning of energy-based representations for signal and image observation dataset with irregular sampling patterns.
{"src": "scitldr-AIC_train", "paper_id": "rJe8pxSFwr"}