<jupyter_start><jupyter_text>Gaussian Mixture ExampleAssume we have mixture of two Gaussians and we want to distinguish them. Here we do it using Black Box SVI II algorithm, described in the article by [Ranganath et al.](https://arxiv.org/abs/1401.0118)Our model contains one local variable $z_i$ for every observed data point $x_i$, which reflects the component of our mixture $x_i$ came from. Both Gaussians have variance equals to one and means $\mu_1$ and $\mu_2$ we wants to find.Our variational distribution consists from $N$ independent Categorical Distributions $q_i(z_i | x_i)$, which parameters we want to tune. At the end we assign every data point to one of the mixture components according to this distribution.While the inference we learn both $q(Z | X)$ and $\mu_1$ and $\mu_2$.<jupyter_code>import torch
import numpy as np
import matplotlib.pyplot as plt

import torch.nn as nn
import torch.optim as optim
import torch.nn.functional as F

from torch.nn import Parameter
from torch.autograd import Variable

from math import pi, log
from BBSVI import SVI
# generating data
std = 1
mu = np.array([-5, 5])
num_components = 2

np.random.seed(42)
num_samples = 100

components = np.random.choice(num_components, size=num_samples, p=np.array([1/2, 1/2]))
data = torch.Tensor(np.random.normal(mu[components], std))<jupyter_output><empty_output><jupyter_text>Below we define two classes for prior and variational distributions:<jupyter_code>class GaussianMixture:
    def __init__(self, num_components, std=1):
        
        self.std = torch.Tensor([std])
        self.means = nn.Parameter(1 * torch.randn(num_components, requires_grad=True))
        self.weights = torch.ones(num_components, requires_grad=True) / num_components
        self.weights_distr = torch.distributions.Categorical(probs=self.weights)
        self.components_distrs = [torch.distributions.Normal(self.means[i], 
                                                             self.std) for i in range(num_components)]
        self.parameters = [self.means]

    def log_likelihood_global(self, beta):
        return torch.zeros(1, requires_grad=True)
    
    def log_likelihood_local(self, z, beta):
        return self.weights_distr.log_prob(z)
    
    def log_likelihood_joint(self, x, z, beta):
        mix_term = self.weights_distr.log_prob(z)
        normal_term = self.components_distrs[z].log_prob(x)
        return mix_term + normal_term
class VariationalDistribution:
    def __init__(self, num_components, data_size):
        self.num_components = num_components
        self.probs = [torch.nn.Parameter(torch.ones(num_components) / num_components) for _ in range(data_size)]
        self.distrs = [torch.distributions.Categorical(probs=self.probs[i]) for i in range(data_size)]
        self.global_parameters = []
        self.local_parameters = self.probs
        
    def sample_global(self, num_samples = 1):
        return None
    
    def sample_local(self, beta, idx):
        return self.distrs[idx].sample()
    
    def entropy(self, batch_indices):
        ent = torch.zeros(1, requires_grad=True)
        for idx in batch_indices:
            ent = ent + self.distrs[idx].entropy()
        return ent
    
    def log_likelihood_global(self, beta):
        return torch.zeros(1, requires_grad=True)
    
    def log_likelihood_local(self, z, idx):
        return self.distrs[idx].log_prob(z)<jupyter_output><empty_output><jupyter_text>Define prior, variational distribution and optimizer<jupyter_code>prior = GaussianMixture(num_components)
var = VariationalDistribution(num_components, num_samples)
opt = torch.optim.Adam([{'params': var.local_parameters},
                        {'params': var.global_parameters},
                        {'params': prior.parameters}], lr=1e-3)<jupyter_output><empty_output><jupyter_text>Do the inference<jupyter_code>svi = SVI(data, prior, var, opt)<jupyter_output><empty_output><jupyter_text>Results:<jupyter_code>predicted_mu = prior.means.data.numpy()
print('Initial means:   \t %.2f \t %.2f' % (predicted_mu[0], predicted_mu[1]))

num_steps = 300
discounter_scheduler = torch.Tensor(np.linspace(0, 1, num_steps))
svi.make_inference(num_steps=num_steps, shuffle=False, loss='bb2', 
                   print_progress=True, retain_graph=True, discounter_schedule=discounter_scheduler)

predicted_mu = prior.means.data.numpy()
print('Predicted means: \t %.2f \t %.2f' % (predicted_mu[0], predicted_mu[1]))
print('Real means:      \t %.2f \t %.2f' % (mu[0], mu[1]))

predicted_components = np.array([torch.max(var.probs[i], dim=-1)[1].data.numpy() for i in range(num_samples)])
accuracy = np.sum(predicted_components == components) / len(predicted_components)
accuracy = max(accuracy, 1 - accuracy)
print('Mixture components detecting accuracy: %.2f %%' % (accuracy * 100))<jupyter_output>Initial means:   	 1.07 	 -1.26
...................................................................................................
Predicted means: 	 4.33 	 -4.44
Real means:      	 -5.00 	 5.00
Mixture components detecting accuracy: 100.00 %
{'fraction_non_alphanumeric': 0.1268465629308647, 'fraction_numerical': 0.015954303722670867, 'mean_word_length': 3.505767524401065, 'pattern_counts': {'":': 0, '<': 20, '<?xml version=': 0, '>': 20, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '268265', 'license': 'MIT', 'n_tokens_mistral': 1652, 'n_tokens_neox': 1572, 'n_words': 395, 'path': 'examples/Gaussian_Mixture_Example.ipynb', 'repo_name': 'zhevnerchuk/Variational-Inference', 'size': '5077'}