<jupyter_start><jupyter_text>HIAuxiliary Deep Generative ModelThe Auxiliary Deep Generative Model [[Maaløe, 2016]](https://arxiv.org/abs/1602.05473) posits a model that with an auxiliary latent variable $a$ that infers the variables $z$ and $y$. This helps in terms of semi-supervised learning by delegating causality to their respective variables. This model was state-of-the-art in semi-supervised until 2017, and is still very powerful with an MNIST accuracy of *99.4%* using just 10 labelled examples per class.  TrainingThe lower bound we derived in the notebook for the **deep generative model** is similar to the one for the ADGM. Here, we also need to integrate over a continuous auxiliary variable $a$.For labelled data, the lower bound is given by.\begin{align}\log p(x,y) &= \log \int \int p(x, y, a, z) \ dz \ da\\&\geq \mathbb{E}_{q(a,z|x,y)} \bigg [\log \frac{p(x,y,a,z)}{q(a,z|x,y)} \bigg ] = - \mathcal{L}(x,y)\end{align}Again when no label information is available we sum out all of the labels.\begin{align}\log p(x) &= \log \int \sum_{y} \int p(x, y, a, z) \ dz \ da\\&\geq \mathbb{E}_{q(a,y,z|x)} \bigg [\log \frac{p(x,y,a,z)}{q(a,y,z |x)} \bigg ] = - \mathcal{U}(x)\end{align}Where we decompose the q-distribution into its constituent parts. $q(a, y, z|x) = q(z|a,y,x)q(y|a,x)q(a|x)$, which is also what can be seen in the figure.The distribution over $a$ is similar to $z$ in the sense that it is also a diagonal Gaussian distribution. However by introducing the auxiliary variable we allow for $z$ to become arbitrarily complex - something we can also see when using normalizing flows.<jupyter_code># from datautils import get_mnist

# # Only use 10 labelled examples per class
# # The rest of the data is unlabelled.
# labelled, unlabelled, validation = get_mnist(location="./", batch_size=64, labels_per_class=10)
# alpha = 0.1 * (len(unlabelled) + len(labelled)) / len(labelled)
import read_functions as rf
dataset = 'defaultCredit'
data_file = dataset+'/data.csv'
types_file = dataset+'/data_types.csv'
miss_file = dataset+'/Missing10-50_1.csv'
true_miss_file = None
batch_size = 128
train_data, types_dict, miss_mask, true_miss_mask, n_samples = rf.read_data(data_file, types_file,
                                                                                        miss_file,
                                                                                        true_miss_file)
# Randomize the data in the mini-batches
random_perm = np.random.permutation(range(np.shape(train_data)[0]))
train_data = train_data[random_perm, :]
miss_mask = miss_mask[random_perm, :]
true_miss_mask = true_miss_mask[random_perm, :]

# Check batch size
if batch_size > n_samples:
    batch_size = n_samples
    
for t in types_dict:
    t['dim'] = int(t['dim'])

# Compute the real miss_mask
miss_mask = np.multiply(miss_mask, true_miss_mask)

train_data = torch.Tensor(train_data)
miss_mask = torch.Tensor(miss_mask)

labelled = train_data[miss_mask[:,-1] == 1, :-2], train_data[miss_mask[:,-1] == 1, -2:]
unlabelled = train_data[miss_mask[:,-1] == 0, :-2], train_data[miss_mask[:,-1] == 0, -2:]
miss_mask = miss_mask[:,:-1]
type_label = types_dict[-1]
types_dict = types_dict[:-1]
alpha = 0.1 * (len(unlabelled) + len(labelled)) / len(labelled)
x, y = labelled
u, v = unlabelled
# Get an integer number of batches
n_batches = int(np.floor(max(x.size()[0], u.size()[0]) / batch_size))
n_batches
if x.size()[0] < u.size()[0]:
    xx = torch.zeros_like(u)
    yy = torch.zeros_like(v)
    xx[:x.size()[0], :] = x
    xx[x.size()[0]:, :] = x[:u.size()[0] - x.size()[0], :]
    x = xx
    yy[:y.size()[0], :] = y
    yy[y.size()[0]:, :] = y[:v.size()[0] - y.size()[0], :]
    y = yy
u.size(), x.size(), y.size(), v.size(), len(types_dict)
from models import HIAuxiliaryDeepGenerativeModel

y_dim = y.size()[1]
z_dim = 5
a_dim = 5 
gamma_dim = 5
h_dim = [5]
input_dim = x.size()[1]

model = HIAuxiliaryDeepGenerativeModel([input_dim, y_dim, z_dim, a_dim, gamma_dim, h_dim], types_dict)
model
x.shape
types_dict
def binary_cross_entropy(r, x):
    return -torch.sum(x * torch.log(r + 1e-8) + (1 - x) * torch.log(1 - r + 1e-8), dim=-1)

optimizer = torch.optim.Adam(model.parameters(), lr=3e-4, betas=(0.9, 0.999))
from itertools import cycle
from inference import HISVI, DeterministicWarmup

# We will need to use warm-up in order to achieve good performance.
# Over 200 calls to SVI we change the autoencoder from
# deterministic to stochastic.
beta = DeterministicWarmup(n=200)


if cuda: model = model.cuda()
elbo = HISVI(model, beta=beta)
alpha *= 100
losses = []
for epoch in tqdm(range(100)):
    model.train()
    total_loss, L_loss, U_loss, class_loss, accuracy = (0, 0, 0, 0, 0)
    for i in range(n_batches):
        # Create inputs for the feed_dict
        data_tensor, labels, batch_miss_mask = rf.next_batch(x, y, types_dict, miss_mask,
                                                         batch_size, index_batch=i)
        unl_data_tensor, hid_labels, unl_batch_miss_mask = rf.next_batch(u, v, types_dict, miss_mask,
                                                         batch_size, index_batch=i)

        # Delete not known data (input zeros)
        data_observed = data_tensor * batch_miss_mask
        unl_data_observed = unl_data_tensor * unl_batch_miss_mask

        # easier names
        x_batch = data_observed
        u_batch = unl_data_observed
        y_batch = labels
        v_batch = hid_labels

        ######
        if cuda:
            # They need to be on the same device and be synchronized.
            x_batch, y_batch = x_batch.cuda(device=0), y_batch.cuda(device=0)
            u_batch, v_batch = u_batch.cuda(device=0), v_batch.cuda(device=0)

        L = -elbo(x_batch, y_batch, batch_miss_mask)
        
        # Add auxiliary classification loss q(y|x,a)
        logits = model.classify(x_batch)
        
        U = -elbo(u_batch, y=None, miss_list=unl_batch_miss_mask)

        # Regular cross entropy
        classication_loss = torch.sum(y_batch * torch.log(logits + 1e-8), dim=1).mean()
        J_alpha = L - alpha * classication_loss + U
#         J_alpha = L

        J_alpha.backward()
        optimizer.step()
        optimizer.zero_grad()

        total_loss += J_alpha.item()
        U_loss += U.item()
        L_loss += L.item()
        class_loss += classication_loss.item()
        accuracy += torch.mean((torch.max(logits, 1)[1].data == torch.max(y_batch, 1)[1].data).float()).item() / n_batches
        ######
    samples_z = model.samples_z
    samples_qa = model.samples_qa
    samples_pa = model.samples_pa
    model.samples_z = []
    model.samples_qa = []
    model.samples_pa = []
    print('Epoch:', epoch+1)
    print('Loss:', total_loss)
    print('---- U:', U_loss)
    print('---- L:', L_loss)
    print('---- cl/tion loss:', class_loss)
    print('Accuracy:', accuracy)
    print('-------------------------------------------\n')
    losses.append((total_loss, U_loss, L_loss, class_loss, accuracy))
        
         # TODO
#     if epoch % 1 == 0:
#         model.eval()
#         m = len(unlabelled)
#         print("Epoch: {}".format(epoch))
#         print("[Train]\t\t J_a: {:.2f}, accuracy: {:.2f}".format(total_loss / m, accuracy / m))

#         total_loss, accuracy = (0, 0)
#         for x, y in validation:
#             x, y = Variable(x), Variable(y)

#             if cuda:
#                 x, y = x.cuda(device=0), y.cuda(device=0)

#             L = -elbo(x, y)
#             U = -elbo(x)

#             logits = model.classify(x)
#             classication_loss = -torch.sum(y * torch.log(logits + 1e-8), dim=1).mean()

#             J_alpha = L + alpha * classication_loss + U

#             total_loss += J_alpha.data[0]

#             _, pred_idx = torch.max(logits, 1)
#             _, lab_idx = torch.max(y, 1)
#             accuracy += torch.mean((torch.max(logits, 1)[1].data == torch.max(y, 1)[1].data).float())

#         m = len(validation)
#         print("[Validation]\t J_a: {:.2f}, accuracy: {:.2f}".format(total_loss / m, accuracy / m))
alpha
losses = np.array(losses)
for l, title in zip(losses.T, ['Total loss', "Unlabeled loss", 'Labeled loss', 'Classification loss', 'accuracy']):
    plt.plot(l)
    plt.title(title)
    plt.xlabel('epoch')
    plt.show()<jupyter_output><empty_output><jupyter_text>plot a try without classloss  try breast try more dimensions try different α because classloss is very small<jupyter_code>ZZ = torch.cat([z for z, _ in samples_z], dim=0)
ZY = torch.cat([y for _, y in samples_z], dim=0)
pca = PCA(n_components=2)
ZZ_pca = pca.fit_transform(ZZ.detach().numpy())
plt.scatter(ZZ_pca[:,0], ZZ_pca[:,1], c=np.argmax(ZY, axis=1), alpha=0.5, s=4);
tsne = TSNE(n_components=2, verbose=1, perplexity=20, n_iter=250)
ZZ_tsne = tsne.fit_transform(ZZ.detach().numpy())
plt.scatter(ZZ_tsne[:,0], ZZ_tsne[:,1], c=np.argmax(ZY, axis=1), alpha=0.5, s=4);
for zz, yy in samples_z:
    zz = zz.detach().numpy()
    plt.scatter(zz[:,0], zz[:,1], c=np.argmax(yy, axis=1), alpha=0.5, s=4)
plt.title('Latent Space for z')
plt.show()
for qa, yy in samples_qa:
    qa = qa.detach().numpy()
    plt.scatter(qa[:,0], qa[:,1], c=np.argmax(yy, axis=1), alpha=0.5, s=4)
plt.title('Latent Space for a')
plt.show()
for pa, yy in samples_pa:
    pa = pa.detach().numpy()
    plt.scatter(pa[:,0], pa[:,1], c=np.argmax(yy, axis=1), alpha=0.5, s=4)
plt.title('Latent Space for a')
plt.show()<jupyter_output><empty_output><jupyter_text>The library is conventially packed with the `SVI` method that does all of the work of calculating the lower bound for both labelled and unlabelled data depending on whether the label is given. It also manages to perform the enumeration of all the labels.Remember that the labels have to be in a *one-hot encoded* format in order to work with SVI.  Conditional generationWhen the model is done training you can generate samples conditionally given some normal distributed noise $z$ and a label $y$.*The model below has only trained for 10 iterations, so the perfomance is not representative*.<jupyter_code>from utils import onehot
model.eval()

z = Variable(torch.randn(16, 32))

# Generate a batch of 5s
y = Variable(onehot(10)(5).repeat(16, 1))

x_mu = model.sample(z, y)
f, axarr = plt.subplots(1, 16, figsize=(18, 12))

samples = x_mu.data.view(-1, 28, 28).numpy()

for i, ax in enumerate(axarr.flat):
    ax.imshow(samples[i])
    ax.axis("off")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14875478927203065, 'fraction_numerical': 0.019444444444444445, 'mean_word_length': 3.1138691883372736, 'pattern_counts': {'":': 0, '<': 14, '<?xml version=': 0, '>': 14, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 12, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '205438', 'license': 'MIT', 'n_tokens_mistral': 3812, 'n_tokens_neox': 3649, 'n_words': 969, 'path': 'examples/notebooks/HIAuxiliary Deep Generative Model.ipynb', 'repo_name': 'NightmareNyx/semi-supervised-pytorch', 'size': '10440'}