Bayesian inference is a statistical inference technique in which Bayes’ theorem is used to update the probability distribution of a random variable using observations. Except for few simple cases, expression of such probability distributions using compact analytical expressions is infeasible. Approximation methods are required to express the a priori knowledge about a random variable in form of prior distributions. Further approximations are needed to compute posterior distributions of the random variables using the observations. When the computational complexity of representation of such posteriors increases over time as in mixture models, approximations are required to reduce the complexity of such representations.This thesis further extends existing approximation methods for Bayesian inference, and generalizes the existing approximation methods in three aspects namely; prior selection, posterior evaluation given the observations and maintenance of computation complexity.Particularly, the maximum entropy properties of the first-order stable spline kernel for identification of linear time-invariant stable and causal systems are shown. Analytical approximations are used to express the prior knowledge about the properties of the impulse response of a linear time-invariant stable and causal system.Variational Bayes (VB) method is used to compute an approximate posterior in two inference problems. In the first problem, an approximate posterior for the state smoothing problem for linear statespace models with unknown and time-varying noise covariances is proposed. In the second problem, the VB method is used for approximate inference in state-space models with skewed measurement noise.Moreover, a novel approximation method for Bayesian inference is proposed. The proposed Bayesian inference technique is based on Taylor series approximation of the logarithm of the likelihood function. The proposed approximation is devised for the case where the prior distribution belongs to the exponential family of distributions.Finally, two contributions are dedicated to the mixture reduction (MR) problem. The first contribution, generalize the existing MR algorithms for Gaussian mixtures to the exponential family of distributions and compares them in an extended target tracking scenario. The second contribution, proposes a new Gaussian mixture reduction algorithm which minimizes the reverse Kullback-Leibler divergence and has specific peak preserving properties.
Williams' method, being a global greedy approach to MR, is computationally quite expensive for mixture densities with many components. The computational burden results from the following facts. Reducing a mixture with @math components to a mixture with @math components involves @math hypotheses. Since computational load of calculating the ISE between mixtures of @math and @math components is @math , reducing a mixture with @math components to a mixture with @math components has the computation complexity @math with Williams' method. On the other hand, using the upper bound , Runnalls' method avoids the computations associated with the components which are not directly involved in the merging operation resulting in just @math computations for the same reduction. Another disadvantage of Williams' method is that the ISE does not scale up with the dimension nicely, as pointed out in an example in @cite_7 .
@cite_7
2140136927 1974193495 1907927509 16236365
This article concerns the greedy learning of gaussian mixtures. In the greedy approach, mixture components are inserted into the mixture one after the other. We propose a heuristic for searching for the optimal component to insert. In a randomized manner, a set of candidate new components is generated. For each of these candidates, we find the locally optimal new component and insert it into the existing mixture. The resulting algorithm resolves the sensitivity to initialization of state-of-the-art methods, like expectation maximization, and has running time linear in the number of data points and quadratic in the (final) number of mixture components. Due to its greedy nature, the algorithm can be particularly useful when the optimal number of mixture components is unknown. Experimental results comparing the proposed algorithm to other methods on density estimation and texture segmentation are provided. The problem of tracking targets in clutter naturally leads to a Gaussian mixture representation of the probability density function of the target state vector. Modern tracking methods maintain the mean, covariance and probability weight corresponding to each hypothesis, yet they rely on simple merging and pruning rules to control the growth of hypotheses. This paper proposes a structured, cost-function-based approach to the hypothesis control problem, utilizing the Integral Square Error (ISE) cost measure. A comparison of track life performance versus computational cost is made between the ISE-based filter and previously proposed approximations including simple pruning, Singer's n-scan memory filter, Salmond's joining filter, and Chen and Liu's Mixture Kalman Filter (MKF). The results demonstrate that the ISE-based mixture reduction algorithm provides mean track life which is significantly greater than that of the compared techniques using similar numbers of mixture components, and mean track life competitive with that of the compared algorithms for similar mean computation times. In recent years analysis of complexity of learning Gaussian mixture models from sampled data has received significant attention in computational machine learning and theory communities. In this paper we present the first result showing that polynomial time learning of multidimensional Gaussian Mixture distributions is possible when the separation between the component means is arbitrarily small. Specifically, we present an algorithm for learning the parameters of a mixture of k identical spherical Gaussians in n-dimensional space with an arbitrarily small separation between the components, which is polynomial in dimension, inverse component separation and other input parameters for a fixed number of components k. The algorithm uses a projection to k dimensions and then a reduction to the 1-dimensional case. It relies on a theoretical analysis showing that two 1-dimensional mixtures whose densities are close in the L norm must have similar means and mixing coefficients. To produce the necessary lower bound for the L norm in terms of the distances between the corresponding means, we analyze the behavior of the Fourier transform of a mixture of Gaussians in one dimension around the origin, which turns out to be closely related to the properties of the Vandermonde matrix obtained from the component means. Analysis of minors of the Vandermonde matrix together with basic function approximation results allows us to provide a lower bound for the norm of the mixture in the Fourier domain and hence a bound in the original space. Additionally, we present a separate argument for reconstructing variance. We present a real-time version of the DUET algorithm for the blind separation of any number of sources using only two mixtures. The method applies when sources are Wdisjoint orthogonal, that is, when the supports of the windowed Fourier transform of any two signals in the mixture are disjoint sets, an assumption which is justified in the Appendix. The online algorithm is a Maximum Likelihood (ML) based gradient search method that is used to track the mixing parameters. The estimates of the mixing parameters are then used to partition the time-frequency representation of the mixtures to recover the original sources. The technique is valid even in the case when the number of sources is larger than the number of mixtures. The method was tested on mixtures generated from different voices and noises recorded from varying angles in both anechoic and echoic rooms. In total, over 1500 mixtures were tested. The average SNR gain of the demixing was 15 dB for anechoic room mixtures and 5 dB for echoic office mixtures. The algorithm runs 5 times faster than real time on a 750MHz laptop computer. Sample sound files can be found here: