Future prediction is a fundamental principle of intelligence that helps plan actions and avoid possible dangers. As the future is uncertain to a large extent, modeling the uncertainty and multimodality of the future states is of great relevance. Existing approaches are rather limited in this regard and mostly yield a single hypothesis of the future or, at the best, strongly constrained mixture components that suffer from instabilities in training and mode collapse. In this work, we present an approach that involves the prediction of several samples of the future with a winner-takes-all loss and iterative grouping of samples to multiple modes. Moreover, we discuss how to evaluate predicted multimodal distributions, including the common real scenario, where only a single sample from the ground-truth distribution is available for evaluation. We show on synthetic and real data that the proposed approach triggers good estimates of multimodal distributions and avoids mode collapse.
Another option is to estimate a complete mixture distribution from a network, like the Mixture Density Networks (MDNs) by Bishop @cite_48 . @cite_3 used MDNs with von Mises distributions for pose estimation. @cite_25 utilized MDNs for uncertainties in autonomous driving by using mixture components as samples alternative to dropout @cite_41 . However, optimizing for a general mixture distribution comes with problems, such as numerical instability, requirement for good initializations, and collapsing to a single mode @cite_30 @cite_33 @cite_11 @cite_5 @cite_44 @cite_28 . The Evolving WTA loss and two stage approach proposed in this work addresses these problems.
@cite_30 @cite_33 @cite_41 @cite_48 @cite_28 @cite_3 @cite_44 @cite_5 @cite_25 @cite_11
2089912168 2805701896 2150260301 1579853615
Mixture modelling involves explaining some observed evidence using a combination of probability distributions. The crux of the problem is the inference of an optimal number of mixture components and their corresponding parameters. This paper discusses unsupervised learning of mixture models using the Bayesian Minimum Message Length (MML) criterion. To demonstrate the effectiveness of search and inference of mixture parameters using the proposed approach, we select two key probability distributions, each handling fundamentally different types of data: the multivariate Gaussian distribution to address mixture modelling of data distributed in Euclidean space, and the multivariate von Mises-Fisher (vMF) distribution to address mixture modelling of directional data distributed on a unit hypersphere. The key contributions of this paper, in addition to the general search and inference methodology, include the derivation of MML expressions for encoding the data using multivariate Gaussian and von Mises-Fisher distributions, and the analytical derivation of the MML estimates of the parameters of the two distributions. Our approach is tested on simulated and real world data sets. For instance, we infer vMF mixtures that concisely explain experimentally determined three-dimensional protein conformations, providing an effective null model description of protein structures that is central to many inference problems in structural bioinformatics. The experimental results demonstrate that the performance of our proposed search and inference method along with the encoding schemes improve on the state of the art mixture modelling techniques. Artificial Neural Networks (ANNs) are increasingly used in modeling and prediction due to advances in deep learning architecture and training. As ANNs become a viable option for making state or measurement predictions in navigation algorithms, special care must be taken in order to determine the confidence (covariance) of the predictions. A special kind of ANN called a Mixture Density Network (MDN) approximates the prediction distribution with a mixture of distributions such as Gaussian distributions. This allows the MDN to output a full Probability Distribution Function (PDF) rather than just a mean estimate. The prediction mixtures can also be converted into other forms for processing by a filter. In this paper, MDN are trained to model a measurement equation which converts measurements to state estimates represented as a Gaussian Mixture Model (GMM). The GMM distributions are simplified into a single Gaussian distribution for each measurement for inclusion into an EKF. An experiment was conducted using magnetic field measurements and Very Low Frequency (VLF) signals to estimate position using an EKF with measurements from a MDN. This method shows a performance improvement using the MDN covariance estimate to determine measurement noise when compared to using static estimates for measurement noise. Thus, the MDN output allows the modeling and prediction power of ANN to be used in standard navigation filters with minimal changes to the filter. This paper proposes a generative model-based speaker clustering algorithm in the maximum a posteriori adapted Gaussian mixture model (GMM) mean supervector space. The algorithm can be viewed as an extension of the standard expectation maximization algorithm for fitting a mixture model to the data, which iterates between two steps - a sample re-assignment step (E-step) and a model re-estimation step (M-step) - until it converges. The directional scattering patterns of GMM mean supervectors suggest that we employ a mixture of von Mises-Fisher distributions in the model re-estimation step. In the sample re-assignment step, four sample-to-mixture assignment strategies, namely soft, hard, stochastic, and deterministic annealing assignments, are used. Our experiments on the GALE Mandarin dataset show that the use of a mixture of von Mises-Fisher distributions as the underlying model yields significantly higher speaker clustering accuracies than the use of a mixture of Gaussian distributions. It is further shown that deterministic annealing assignment outperforms soft assignment, that soft assignment is comparable to stochastic assignment, and that both soft and stochastic assignments outperform hard assignment. Minimization of a sum-of-squares or cross-entropy error function leads to network outputs which approximate the conditional averages of the target data, conditioned on the input vector. For classifications problems, with a suitably chosen target coding scheme, these averages represent the posterior probabilities of class membership, and so can be regarded as optimal. For problems involving the prediction of continuous variables, however, the conditional averages provide only a very limited description of the properties of the target variables. This is particularly true for problems in which the mapping to be learned is multi-valued, as often arises in the solution of inverse problems, since the average of several correct target values is not necessarily itself a correct value. In order to obtain a complete description of the data, for the purposes of predicting the outputs corresponding to new input vectors, we must model the conditional probability distribution of the target data, again conditioned on the input vector. In this paper we introduce a new class of network models obtained by combining a conventional neural network with a mixture density model. The complete system is called a Mixture Density Network, and can in principle represent arbitrary conditional probability distributions in the same way that a conventional neural network can represent arbitrary functions. We demonstrate the effectiveness of Mixture Density Networks using both a toy problem and a problem involving robot inverse kinematics.