Inference in latent variable models, like the one we propose here, can be challenging.
This is especially difficult when there is lack of conjugacy between the distributions of the latent variables and the observed variables. One approach is to perform point estimation for both the latent variables and the parameters in an alternating fashion [1]}, but this is prone to over-fitting [2]} and convergence issues. Another approach is to use Markov Chain Monte Carlo (MCMC) methods, which can be computationally expensive [3]}, especially in high dimension. As an alternative, variational Bayes inference has shown much promise [4]}. Note that
Variational Bayes
is not a general purpose method and must be tailored to the specific statistical model [5]}.
When there is a lack of conjugacy, as is the case for the multinomial-Gaussian model in this paper, local variational bound approximations are often adopted [6]}. Additionally, when there is a problematic expression in the joint density, such as LogSumExp or LogGamma function, which may prevent the inference of the latent variables, surrogate optimization transfer based on Taylor series expansion can be applied to approximate the non-linear function either with linear [7]} or quadratic [8]}, [9]}, [10]}, [11]}, [12]} functions. We adopt such a local variational bound approach for deriving an inference algorithm for our proposed model.
