Proximity penalty priors for Bayesian mixture models

When using mixture models it may be the case that the modeller has a-priori beliefs or desires about what the components of the mixture should represent. For example, if a mixture of normal densities is to be fitted to some data, it may be desirable for components to focus on capturing differences in location rather than scale. We introduce a framework called proximity penalty priors (PPPs) that allows this preference to be made explicit in the prior information. The approach is scale-free and imposes minimal restrictions on the posterior; in particular no arbitrary thresholds need to be set. We show the theoretical validity of the approach, and demonstrate the effects of using PPPs on posterior distributions with simulated and real data.


Introduction
Mixture models are widely recognized as a useful tool for inference in a variety of settings. Having been first used over 100 years ago (for example, in Pearson, 1894), more recently mixture models are enjoying a revival, thanks to advances in computational methods for inference. In particular, the EM algorithm (Dempster et al., 1977) and MCMC (see, for example, Diebolt and Robert, 1994) have driven considerable advances in the field. See McLachlan and Peel (2000) for a general overview of mixture models; Fruhwirth-Schnatter (2006) provides an overview of Bayesian mixture models, which are the focus of this paper.
A Bayesian approach to estimating the parameters of the mixture distribution of Equation (1) involves the specification of priors for the parameters γ. The issue of prior specification in this context has a number of difficulties.
First, fully improper priors cannot be used for component-specific parameters in mixture models, since doing so causes the posterior to be improper also (see, for example, McLachlan and Peel, 2000). However, proper priors, even with large variance, can have considerable influence on the posterior distribution, and the extent of this influence can be difficult to assess (Marin et al., 2005). Re-parametrisation in a hierarchical manner and allowing only the global parameters to be improper is one solution: this is considered by Mengersen and Robert (1996), and Roeder and Wasserman (1997). Another possibility is to use data-dependent priors, as considered by Richardson andGreen (1997), andWasserman (2000).
Second, where no component specific information is available, identical priors may be proposed for the components of each parameter. This leads to a non-identifiable posterior, which is known as the label switching problem. This has been well studied (see, for example Stephens, 2000;Jasra et al., 2005;Sperrin et al., 2010, and references therein).
Third, constructing independent priors for component parameters may not be sensible, as the components only have meaning relative to one another (Lee et al., 2008). This third issue is the focus of this paper. We consider in detail the idea that priors should be specified relative to each other. We introduce a strategy for doing so that we call 'proximity penalty priors' (PPPs). The basic idea is that priors are specified in two parts: first, each prior is specified independently, corresponding to standard existing approaches; second, a proximity penalty is applied, which penalises the joint prior distribution of certain configurations of parameters. We show that the construction makes theoretical sense.
Section 2 introduces the idea of PPPs. Section 3 illustrates the consequences of the PPP approach on real and simulated data; the paper concludes with a discussion in Section 4.

Proximity Penalty Priors
We begin with a simple result that establishes the validity of the PPP approach.
Denote the likelihood by L(γ) and the posterior by q(γ), so that q(γ) ∝ L(γ)p(γ). Suppose that a new parameter vector γ * can be simulated from a proposal distribution r(γ * ) = L(γ * )p 1 (γ * ), and the existing value of γ is γ m . Then if we set the result is equivalent to a Metropolis-Hastings update.
Proof. The acceptance probability for the Metropolis-Hastings procedure with proposal density r(·) and posterior q(·) is Substituting in these densities gives the result.
In the context of this work the portion of the prior p 1 (·) corresponds to the independent specification of the parameters, for which standard distributions could be used; the portion p 2 (·) corresponds to the novel part of the prior that jointly assesses the values of the parameters and penalises undesirable combinations.
Suppose that the priors p 1 (·) are conjugate. Then an MCMC approach would proceed, on each iteration, by generating proposed new parameters according to a Gibbs sampling scheme with the full conditionals based on the prior component p 1 (·), then accepting the proposed parameters according to a Metropolis Hastings ratio on the prior component p 2 (·).
We illustrate the idea with an example. Consider a mixture of two normal distributions with π 1 + π 2 = 1, and all the parameters γ = (π 1 , π 2 , µ 1 , µ 2 , σ 2 1 , σ 2 2 ) unknown. Standard conjugate prior choices would then be a Dirichlet distribution for the pair (π 1 , π 2 ), normal distributions for µ 1 and µ 2 , and inverse-gamma distributions for σ 2 1 and σ 2 2 . Throughout this paper we will use the empirical Bayes prior distributions suggested by Richardson and Green (1997) unless otherwise stated. We may believe a-priori that the key difference between the two components is the location. If the components are not well separated or the amount of data is small it is important that such prior information is captured. By Proposition 1, we can reflect these beliefs in a separate part of the prior p 2 (·). A sensible such choice is Such a function assigns more prior weight to larger differences between µ 1 and µ 2 . In isolation, the above p 2 (·) is improper but provided p 1 (·) is proper the overall prior is proper. Such a prior enjoys scale invariance in the sense that p 2 (ax 1 )/p 2 (ax 2 ) = p 2 (x 1 )/p 2 (x 2 ) for all non-zero a. This may or may not be desirable. An alternative would be to specify a distance δ as a minimum distance between µ 1 and µ 2 , i.e.
This generates the question of how δ should be specified, but may be appropriate in some situations.
More generally, for a mixture distribution with K parameters, suppose there exists a component-specific parameter φ k for each component k = 1, . . . , K, and the difference between the components is a-priori believed (or, from the point of view of model interpretation, desired) to be in terms of this parameter. Then we propose setting On the other hand, for a mixture distribution with K parameters, if there exists a component-specific parameter ψ k for each component k = 1, . . . , K, and each component is a-priori expected or desired to have similar values of this parameter, we could set Here, the scale free nature of p 2 (·) is an advantage in that we do not have to quantify 'similar'. More generally, p 2 (γ) could be constructed as any multiplicative combination of Equations (5) and (6). The procedure can also be applied when the number of components K is allowed to vary, in which case it makes sense only within fixed values of K in the same way that the label switching problem only has meaning within fixed values of K (Nobile and Fearnside, 2007).
In both cases we fix the number of components K = 2. In (b), we are therefore adding an explicit prior opinion that the difference between the two components is in the locations µ 1 and µ 2 . Figure 1 compares a bivariate projection of the posterior onto the absolute difference |µ 1 − µ 2 | and max(σ 2 1 , σ 2 2 ) without and with the PPP. Without the PPP, posterior mass is assigned to the situation where |µ 1 − µ 2 | is small and max(σ 2 1 , σ 2 2 ) is large. This corresponds to a case where a mixture distribution with similar means but different variances is fitted. In Figure 2 we see that such a mixture is well supported by the data (dashed line in the figure). Once the PPP is applied, far less posterior mass is assigned to this scenario, since our prior distribution specifically tells us to exclude such cases. Figure 3 gives the marginal bivariate posterior of (µ 1 , µ 2 ), with and without the PPP. Without the PPP, the posterior appears to have a single mode at approximately µ 1 = µ 2 = 1; with the PPP, the posterior is bimodal with modes at approximately (µ 1 = 0, µ 2 = 2) and (µ 1 = 2, µ 2 = 0). The bimodality in the PPP case is a consequence of label switching; if component-specific inference is required, post-hoc relabelling should be carried out (see, for example, Sperrin et al., 2010). The unimodality in the non PPP case is caused by the two means being very close together and the variances to differ, corresponding to a different interpretation of the mixture components.
We also ran the same comparison without assuming a fixed number of components K (using the birth-death method of Stephens, 2000), putting a Poisson(1) prior distribution on the number of components K (see Nobile and Fearnside, 2007, for a justification of the use of this prior). Similar results to the above were observed when we looked at the output conditional on K = 2.

Galaxy Data
The galaxy dataset is commonly used to illustrate mixture modelling techniques (see Jasra et al., 2005, for a recent investigation of this dataset in the mixture modelling context). Briefly, it consists of the velocities of 82 galaxies, but the velocities appear to cluster, suggesting different groups of galaxies that we may wish to identify (see Figure 4). If we model these data using a mixture, it is likely that we wish our mixture components to represent the clusters with different mean velocities, hence the PPP of Equation (5) could be considered in this scenario. We run a variable dimension sampler with the details as above, with normally distributed components assumed and a Poisson(1) prior distribution on the number of components K. We compare the results of standard priors (i.e. those given in Richardson and Green, 1997) with the standard priors plus the PPP. Both with and without the PPP, the values of K with the majority of posterior support are K = 3 and K = 4 (but see Aitkin, 2001, for discussion on the posterior of the number of components in a mixture model). For the K = 3 case the posterior means are already well separated, and the PPP has little or no effect on the posterior means. We look in more detail at the K = 4 case. In order to avoid the label switching issue, we first consider the posterior of a generic µ k without relabelling, estimating this by combining into a single vector all samples from the posterior µ k , for k = 1, 2, 3, 4, conditional on K = 4. We can do this since invariance of the posterior under re-parametrisation means we can ignore the labels. The resulting density plot is given in Figure 5. The interesting difference to note here is that with the PPP four distinct peaks can be observed in the density, whereas without the PPP the middle two peaks cannot be distinguished. This does, however, depend on the smoothing parameter used in the non-parametric density estimate.
To consider this further we mitigate the label switching issue by applying the identifiability constraint µ 1 < µ 2 < µ 3 < µ 4 , then look at the posterior density of (µ 3 − µ 2 ). This is given in Figure 6. We see that applying the PPP causes more separation between the two component means (less mass at small differences).

Discussion
In this paper we have introduced the idea of incorporating weak joint information about parameters in a mixture model into the prior specification. In particular we have introduced proximity penalty priors (PPPs) as a method of explicitly declaring an a-priori opinion (or interest) in components that differ on a certain parameter. The formulation is designed to allow this opinion to be as vague as possible: we avoid making any statement about the magnitude of the difference that should be observed between the components, i.e. the method is scale-free.
With the focus of this paper being introduction of the idea, the examples were kept fairly simple. The idea, however, is very general and could be applied in more complex models. For example, in an application such as genetics we may wish to construct a mixture of regressions with many co- Figure 6: Smoothed density of (µ 3 − µ 2 ) for the galaxy data with K = 4 after an IC is applied. Without PPP: dashed line; with PPP: solid line.
variates. Suppose there are p covariates and K mixtures, with the coefficient of the j th covariate in the k th mixture given by β jk . Then we could consider the PPP p 2 (γ) = max j min k =l |β jk − β jl |, to reflect a belief that each component should have at least one coefficient that differs from the value in every other component. Another potential extension is to replace the L 1 -norm assumed in the PPP with an L s -norm, i.e. considering a generalisiation of, for example, Equation (4), to p 2 (γ) = |µ 1 − µ 2 | s .
In this generalised setting, we note that s = 0 clearly corresponds to an unpenalised prior and s = 1 reduces to the original Equation (4). Also, setting s = −1 encodes a PPP like Equation (6). This generalisation then raises the question of how should s be chosen? We suggest s = 1 is a very natural choice, since this means the penalty is being applied on the original scale of the data. We have, however, looked at the sensitivity to the choice of s. For the example considered in Section 3, once s becomes large the posteriors for µ become very flat.