In this paper, both non-mixing and mixing local minima of the entropy are analyzed from the viewpoint of blind source separation (BSS); they correspond respectively to acceptable and spurious solutions of the BSS problem. The contribution of this work is twofold. First, a Taylor development is used to show that the exact output entropy cost function has a non-mixing minimum when this output is proportional to any of the non-Gaussian sources, and not only when the output is proportional to the lowest entropic source. Second, in order to prove that mixing entropy minima exist when the source densities are strongly multimodal, an entropy approximator is proposed. The latter has the major advantage that an error bound can be provided. Even if this approximator (and the associated bound) is used here in the BSS context, it can be applied for estimating the entropy of any random variable with multimodal density
As for the mutual information, simulations results in @cite_11 suggest that mixing local entropy minima exist in specific cases (i.e. when the source pdfs are strongly multimodal, which sometimes occur in practice, for sinusoid waveforms among other). These results, based on density estimation using the Parzen kernel method, are confirmed by other simulations using directly entropy estimation, such as Vasicek's one in @cite_8 or based on the approximator analyzed in this paper in @cite_18 . Rigorously speaking, the above results do not constitute an absolute proof since error bounds are not available for the approximation procedure. By contrast, a theoretical proof is given in @cite_14 , but for a specific example only (two bimodal sources sharing the same symmetric pdf). The existence of mixing local entropy minima has also been shown in @cite_9 (without detailed proof) in the case of two non symmetric sources with strongly multimodal pdfs.
@cite_18 @cite_14 @cite_8 @cite_9 @cite_11
2073632369 2146830462 2332339252 2135806772
The marginal entropy h(Z) of a weighted sum of two variables Z = αX + βY, expressed as a function of its weights, is a usual cost function for blind source separation (BSS), and more precisely for independent component analysis (ICA). Even if some theoretical investigations were done about the relevance from the BSS point of view of the global minimum of h(Z), very little is known about possible local spurious minima.In order to analyze the global shape of this entropy as a function of the weights, its analytical expression is derived in the ideal case of independent variables. Because of the ICA assumption that distributions are unknown, simulation results are used to show how and when local spurious minima may appear. Firstly, the entropy of a whitened mixture, as a function of the weights and under the constraint of independence between the source variables, is shown to have only relevant minima for ICA if at most one of the source distributions is multimodal. Secondly, it is shown that if independent multimodal sources are involved in the mixture, spurious local minima may appear. Arguments are given to explain the existence of spurious minima of h(Z) in the case of multimodal sources. The presented justification can also explain the location of these minima knowing the source distributions. Finally, it results from numerical examples that the maximum-entropy mixture is not necessarily reached for the 'most mixed' one (i.e. equal mixture weights), but depends of the entropy of the mixed variables. Recent simulation results have indicated that spurious minima in information-theoretic criteria with an orthogonality constraint for blind source separation may exist. Nevertheless, those results involve approximations (e.g., density estimation), so that they do not constitute an absolute proof. In this letter, the problem is tackled from a theoretical point of view. An example is provided for which it is rigorously proved that spurious minima can exist in both mutual information and negentropy optima. The proof is based on a Taylor expansion of the entropy. The aim of this paper is to show that spatial coupling can be viewed not only as a means to build better graphical models, but also as a tool to better understand uncoupled models. The starting point is the observation that some asymptotic properties of graphical models are easier to prove in the case of spatial coupling. In such cases, one can then use the so-called interpolation method to transfer known results for the spatially coupled case to the uncoupled one. Our main use of this framework is for Low-density parity check (LDPC) codes, where we use interpolation to show that the average entropy of the codeword conditioned on the observation is asymptotically the same for spatially coupled as for uncoupled ensembles. We give three applications of this result for a large class of LDPC ensembles. The first one is a proof of the so-called Maxwell construction stating that the MAP threshold is equal to the area threshold of the BP GEXIT curve. The second is a proof of the equality between the BP and MAP GEXIT curves above the MAP threshold. The third application is the intimately related fact that the replica symmetric formula for the conditional entropy in the infinite block length limit is exact. Marginal entropy can be used as cost function for blind source separation (BSS). Recently, some authors have experimentally shown that such information-theoretic cost function may have spurious minima in specific situations. Hence, one could face spurious solutions of the BSS problem even if the mixture model is known, exactly as when using the maximum-likelihood criterion. Intuitive justifications of the spurious minima have been proposed, when the sources have multimodal densities. This paper aims to give mathematical arguments, complementary to existing simulation results, to explain the existence of such minima. This is done by first deriving a specific entropy estimator. Then, this estimator, although reliable only for multimodal sources with small-overlapping Gaussian modes, allows one to show that spurious minima may exist when dealing with such sources.