<commit_before>---
layout: article
title: The unreasonable elegance of deep generative models
comments: true
categories: data_science
image:
  teaser: jupyter-main-logo.svg
---

## Background
- Data surrounds us, however, most of it is unlabeled data. Anotating data is a very energy-gobbling and time-consuming effort. Therefore, labeled data is few and expensive.

- Supervised learning works very well, however, it requires a lot of labeled data which is scarce. Semi-supervised learning, transfer learning, and few shot learning are promising approaches for overcoming this problem. 

- We focus on the former. Supervised learning addresses the problem of classification when only a small number of labeled examples are available. The idea is to employ both labeled and unlabeled data in our modelling process. Although unlabeled data might not be able to help in discriminative tasks, theoretically, it can be useful in finding the right representation for the data. 
    + The simplest approach is self-training, where the model is fed the data it has confidently classified data. This can reinforce poor predictions. most existing methods not really scalable.
        * Transductive SVM -> use distance to margin as confidence?
    + Graph-based -> propagate the label information from labeled nodes to unlabeled nodes (MAP inference?)
    + neural net based -> train a classifier using an autoencoder (or other unsupervised embeddings) as a regularizer. 
    + Manifold tangent classifier -> trains contrastive autoencoders to learn the manifold on which the data lies. Follows this with TangentProp to train a classifier that is approximately invariant to local perturbations along the manifold. (Tangent prop has an additional derivative of weights penalty term for known invariances in the data)
    + Manifold learning using graph-based methods + SVM kernel (rbf) method.


## Representation learning with VAE
M1 is a simple VAE model for unsupervised feature learning. The learned features are used for training a separate classifier. Approximate samples from the posterior distribution over the latent variables p(z|x) are used as features to train a classifier that predicts class labels y from data in a lower dimensional space. 

<img src="/images/VAE_intuitions/vae_semi_M1.png" alt="semi-supervised model inference" width="350" height="350">

Let observations be $$X = {x_1, x_2, .., x_n}$$. We assume a simple generative model with a set of latent variables $$Z = {z_1, z_2,...,z_n}$$. 

$$ Z ~ p(Z) = N(0,I) ;  X ~ p(X|Z) = N(\mu, \sigma^2)$$

![alt text](/images/VAE_intuitions/vae_semi_M1.png "a simple vae latent variable model")

- Variational Inference turns the inference into optimization. It posits a variational family of distributions over the latent variables,  and fit the variational parameters to be close (in KL sense or another divergence like BP, EP, etc) to the exact posterior. KL is intractable (only possible exactly if q is simple enough and compatible with the prior), so VI optimizes the evidence lower bound (ELBO) instead which is a lower bound on log p(x). Maximizing the ELBO is equivalent to minimizing the KL, note that the ELBO is not convex. The ELBO trades off two terms, the first term prefers q(.) to place its mass on the MAP estimate. The second term encourages q(.) to be diffuse.

- Note that in the above derivation of the ELBO, the first term is the entropy of the variational posterior and second term is log of joint distribution. However we usually write joint distribution as $$p(x,z)=p(x|z)p(z)$$ to rewrite the ELBO as $$ E_q[\log\ p(x|z)+KL(q(z|x)\ | \ p(z))]$$. This derivation is much closer to the typical machine learning literature in deep networks. The first term is log likelihood (i.e. reconstruction cost) while the second term is KL divergence between the prior and the posterior (i.e a regularization term that won't allow posterior to deviate much from the prior). Also note that if we only use the first term as our cost function, the learning with correspond to maximum likelihood learning that does not include regularization and might overfit.

- What we need for optimization is actually the gradient of the ELBO not the ELBO itself. Using pathwise gradient (reparameterization trick) we can calculate the gradient without evaluating the ELBO

## Conditional VAE
<commit_msg>working on semi-supervised post<commit_after>---
layout: article
title: The unreasonable elegance of deep generative models
comments: true
categories: data_science
image:
  teaser: jupyter-main-logo.svg
---

## Background

So we live in the age of data they say. Singularity is near they say. It's soon that artificially intelligent robots will take over and we will have to bow down to our robot overlords. Between the [Internet](), [smart cities](), [Internet of things](), [big data](), [Block chain](), it's safe to say data is [eating the world]() (I tried to pack in as many buzz words as I can!). However, what they don't tell you is that almost all of it is unlabeled data. Data that is not annotated as belonging to a certain class. 

- Supervised learning works very well, however, it requires a lot of labeled data which is scarce. 

With the exception of a few cool developments, almost all recent success stories of deep learning come from the supervised learning front. Cases where vast amount of labeled data is available. However, annotating data is a very energy-gobbling and time-consuming effort. Therefore, labeled data is scarce and expensive while unlabeled data is vast and cheap. That's why we might want to develop methodologies that try to learn from unlabeled data along with our precious labeled samples. Semi-supervised learning, transfer learning, few shot learning, active learning, and reinforcement learning are promising approaches for overcoming this problem. 

In this post, I want to focus on a the former. In particular, I will focus on a promising and cool approach called semi-supervised learning with deep generative models. The idea is to employ both labeled and unlabeled data in our modeling process. Although unlabeled data might not be able to help in discriminative tasks, theoretically, it can be useful in finding the right representation for the data. 
    + The simplest approach is self-training, where the model is fed the data it has confidently classified data. This can reinforce poor predictions. most existing methods not really scalable.
        * Transductive SVM -> use distance to margin as confidence?
    + Graph-based -> propagate the label information from labeled nodes to unlabeled nodes (MAP inference?)
    + neural net based -> train a classifier using an autoencoder (or other unsupervised embeddings) as a regularizer. 
    + Manifold tangent classifier -> trains contrastive autoencoders to learn the manifold on which the data lies. Follows this with TangentProp to train a classifier that is approximately invariant to local perturbations along the manifold. (Tangent prop has an additional derivative of weights penalty term for known invariances in the data)
    + Manifold learning using graph-based methods + SVM kernel (rbf) method.


## Representation learning with VAE
M1 is a simple VAE model for unsupervised feature learning. The learned features are used for training a separate classifier. Approximate samples from the posterior distribution over the latent variables p(z|x) are used as features to train a classifier that predicts class labels y from data in a lower dimensional space. 

<img src="/images/VAE_intuitions/vae_semi_M1.png" alt="semi-supervised model inference" width="350" height="350">

Let observations be $$X = {x_1, x_2, .., x_n}$$. We assume a simple generative model with a set of latent variables $$Z = {z_1, z_2,...,z_n}$$. 

$$ Z ~ p(Z) = N(0,I) ;  X ~ p(X|Z) = N(\mu, \sigma^2)$$

![alt text](/images/VAE_intuitions/vae_semi_M1.png "a simple vae latent variable model")

- Variational Inference turns the inference into optimization. It posits a variational family of distributions over the latent variables,  and fit the variational parameters to be close (in KL sense or another divergence like BP, EP, etc) to the exact posterior. KL is intractable (only possible exactly if q is simple enough and compatible with the prior), so VI optimizes the evidence lower bound (ELBO) instead which is a lower bound on log p(x). Maximizing the ELBO is equivalent to minimizing the KL, note that the ELBO is not convex. The ELBO trades off two terms, the first term prefers q(.) to place its mass on the MAP estimate. The second term encourages q(.) to be diffuse.

- Note that in the above derivation of the ELBO, the first term is the entropy of the variational posterior and second term is log of joint distribution. However we usually write joint distribution as $$p(x,z)=p(x|z)p(z)$$ to rewrite the ELBO as $$ E_q[\log\ p(x|z)+KL(q(z|x)\ | \ p(z))]$$. This derivation is much closer to the typical machine learning literature in deep networks. The first term is log likelihood (i.e. reconstruction cost) while the second term is KL divergence between the prior and the posterior (i.e a regularization term that won't allow posterior to deviate much from the prior). Also note that if we only use the first term as our cost function, the learning with correspond to maximum likelihood learning that does not include regularization and might overfit.

- What we need for optimization is actually the gradient of the ELBO not the ELBO itself. Using pathwise gradient (reparameterization trick) we can calculate the gradient without evaluating the ELBO

## Conditional VAE

{'fraction_non_alphanumeric': 0.057951482479784364, 'fraction_numerical': 0.003110097449720091, 'mean_word_length': 4.478137421919364, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 13, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3142267', 'n_tokens_mistral': 2479, 'n_tokens_neox': 2286, 'n_words': 1418}