When optimizing against the mean loss over a distribution of predictions in the context of a regression task, then even if there is a distribution of targets the optimal prediction distribution is always a delta function at a single value. Methods of constructing generative models need to overcome this tendency. We consider a simple method of summarizing the prediction error, such that the optimal strategy corresponds to outputting a distribution of predictions with a support that matches the support of the distribution of targets --- optimizing against the minimal value of the loss given a set of samples from the prediction distribution, rather than the mean. We show that models trained against this loss learn to capture the support of the target distribution and, when combined with an auxiliary classifier-like prediction task, can be projected via rejection sampling to reproduce the full distribution of targets. The resulting method works well compared to other generative modeling approaches particularly in low dimensional spaces with highly non-trivial distributions, due to mode collapse solutions being globally suboptimal with respect to the extreme value loss. However, the method is less suited to high-dimensional spaces such as images due to the scaling of the number of samples needed in order to accurately estimate the extreme value loss when the dimension of the data manifold becomes large.
: Mixture density networks @cite_7 directly parameterize the output distribution with (generally) a sum of Gaussians, and then train the network to maximize the likelihood of the empirical distribution under the model. When the dimension of the target space is low, this is an effective way to capture complex multi-modal structure in the joint distribution. However, the covariance matrices to specify each Gaussian grow quadratically in the dimension of the target space, and the number of Gaussians needed to fit a curved manifold structure in high dimension may likewise grow quickly. MDNs were used in SketchRNN @cite_11 to model the continuation of lines in drawing figures and kanji.
@cite_7 @cite_11
2168227362 1579853615 2026302946 2144100131
Gaussian mixtures (or so-called radial basis function networks) for density estimation provide a natural counterpart to sigmoidal neural networks for function fitting and approximation. In both cases, it is possible to give simple expressions for the iterative improvement of performance as components of the network are introduced one at a time. In particular, for mixture density estimation we show that a k-component mixture estimated by maximum likelihood (or by an iterative likelihood improvement that we introduce) achieves log-likelihood within order 1 k of the log-likelihood achievable by any convex combination. Consequences for approximation and estimation using Kullback-Leibler risk are also given. A Minimum Description Length principle selects the optimal number of components k that minimizes the risk bound. Minimization of a sum-of-squares or cross-entropy error function leads to network outputs which approximate the conditional averages of the target data, conditioned on the input vector. For classifications problems, with a suitably chosen target coding scheme, these averages represent the posterior probabilities of class membership, and so can be regarded as optimal. For problems involving the prediction of continuous variables, however, the conditional averages provide only a very limited description of the properties of the target variables. This is particularly true for problems in which the mapping to be learned is multi-valued, as often arises in the solution of inverse problems, since the average of several correct target values is not necessarily itself a correct value. In order to obtain a complete description of the data, for the purposes of predicting the outputs corresponding to new input vectors, we must model the conditional probability distribution of the target data, again conditioned on the input vector. In this paper we introduce a new class of network models obtained by combining a conventional neural network with a mixture density model. The complete system is called a Mixture Density Network, and can in principle represent arbitrary conditional probability distributions in the same way that a conventional neural network can represent arbitrary functions. We demonstrate the effectiveness of Mixture Density Networks using both a toy problem and a problem involving robot inverse kinematics. Given data drawn from a mixture of multivariate Gaussians, a basic problem is to accurately estimate the mixture parameters. We give an algorithm for this problem that has running time and data requirements polynomial in the dimension and the inverse of the desired accuracy, with provably minimal assumptions on the Gaussians. As a simple consequence of our learning algorithm, we we give the first polynomial time algorithm for proper density estimation for mixtures of k Gaussians that needs no assumptions on the mixture. It was open whether proper density estimation was even statistically possible (with no assumptions) given only polynomially many samples, let alone whether it could be computationally efficient. The building blocks of our algorithm are based on the work (Kalai , STOC 2010) that gives an efficient algorithm for learning mixtures of two Gaussians by considering a series of projections down to one dimension, and applying the method of moments to each univariate projection. A major technical hurdle in the previous work is showing that one can efficiently learn univariate mixtures of two Gaussians. In contrast, because pathological scenarios can arise when considering projections of mixtures of more than two Gaussians, the bulk of the work in this paper concerns how to leverage a weaker algorithm for learning univariate mixtures (of many Gaussians) to learn in high dimensions. Our algorithm employs hierarchical clustering and rescaling, together with methods for backtracking and recovering from the failures that can occur in our univariate algorithm. Finally, while the running time and data requirements of our algorithm depend exponentially on the number of Gaussians in the mixture, we prove that such a dependence is necessary. This paper characterizes the eigenvalue distributions of full-rank Hermitian matrices generated from a set of independent (non)zero-mean proper complex Gaussian random vectors with a scaled-identity covariance matrix. More specifically, the joint and marginal cumulative distribution function (CDF) of any subset of unordered eigenvalues of the so-called complex (non)central Wishart matrices, as well as new simple and tractable expressions for their joint probability density function (PDF), are derived in terms of a finite sum of determinants. As corollaries to these new results, explicit expressions for the statistics of the smallest and largest eigenvalues, of (non)central Wishart matrices, can be easily obtained. Moreover, capitalizing on the foregoing distributions, it becomes possible to evaluate exactly the mean, variance, and other higher order statistics such as the skewness and kurtosis of the random channel capacity, in the case of uncorrelated multiple-input multiple-output (MIMO) Ricean and Rayleigh fading channels. Doing so bridges the gap between Telatar's initial approach for evaluating the average MIMO channel capacity (Telatar, 1999), and the subsequently widely adopted moment generating function (MGF) approach, thereby setting the basis for a PDF-based framework for characterizing the capacity statistics of MIMO Ricean and Rayleigh fading channels.