
where the first R.H.S. term is the Kullback-Leibler divergence between two probability distributions \(Q\)  and \(P\) , and \(\phi \)  denotes variational parameters. The generative model and inference model are trained jointly by maximizing the ELBO w.r.t their parameters. The KL-divergence \(D_{KL}(q_{\phi }(\mathbf {z}|\mathbf {x})\Vert p_{\theta _{}}(\mathbf {z}))\)  of Eq. REF  can often be integrated analytically. However, the gradient of the second term is problematic because latent variable \(\mathbf {z}\sim q_{\phi }(\mathbf {z}|\mathbf {x})\)  is stochastic which blocks backpropagation. The Stochastic Gradient Variational Bayes (SGVB) [1]} expresses the random variable \(\mathbf {z}\)  as a deterministic variable \(\mathbf {z}=g_{\phi }(\mathbf {\epsilon },\mathbf {x})\) , where \(\mathbf {\epsilon }\)  is an auxiliary variable with independent marginal distribution \(p(\mathbf {\epsilon })\) . Then we have the SGVB estimator \(\tilde{\mathcal {L}}(\theta _{},\phi ;\mathbf {x})\simeq \mathcal {L}(\theta _{},\phi ;\mathbf {x})\)  of the lower bound:
\(\begin{split}\tilde{\mathcal {L}}(\theta _{},\phi ;\mathbf {x})= -D_{KL}\left(q_{\phi }(\mathbf {z}|\mathbf {x})\Vert p_{\theta _{}})\right)+ \log p_{\theta _{}}(\mathbf {x}|\mathbf {z})\\\text{where \quad } \mathbf {z}=g_{\phi }(\mathbf {\epsilon },\mathbf {x})\text{\quad and \quad } \mathbf {\epsilon }\sim p(\mathbf {\epsilon })\end{split}\) 
