<commit_before>Code for NIPS 2015 paper on Max-margin Deep Generative Models(MMDGM)

<NAME>, <NAME>, <NAME> and <NAME>,
Max-margin Deep Generative Models 
Advances in Neural Information Processing Systems (NIPS15), Montreal  
Please cite this paper when using this code for your research.

For questions and bug reports, please send me an e-mail at _chongxuanli1991[at]gmail.com_.


## Prerequisites

1. Some libs we used in our experiments:
    - Python (version 2.7)
	- Numpy
    - Scipy
    - Theano (version 0.7.0)
    - Cuda (7.0)
    - Cudnn (optional, see details below, version 6.5)
    - Pylearn2 (for pre-processing of SVHN)

2. GPU: TITAN X Black
    - memory of GPU is at least 9G for SVHN

## Some matters

1. We found that theano is computationally unstable with Cudnn. For MNIST, we do NOT use Cudnn and the error rate should be exact 0.45% with same version of libs and machine. For SVHN, we DO use Cudnn for faster training and there exists additional randomness even given fixed random seed. However, we run this experiments for 5 times and choose the lowest accuracy to report. Typically, the generative results won't be changed. 

2. In MLP case, we use code from Kingma[https://github.com/dpkingma/nips14-ssl]. In CNN case, we use code from Goodfellow to do local contrast normalization(LCN) for SVHN data(pylearn2). We also use code from the tutorial on deeplearning.org.

3. You may need to resave .sh file in unix format to avoid bugs for strings at the end of a line. Further, you should run the command chmod +x *.sh.

## MLP results, code is in folder mlp-mmdgm

0. We did our experiments based on Kingma's code[https://github.com/dpkingma/nips14-ssl]. 
You should run this command at first: export ML_DATA_PATH="[dir]/mmdgm/mlp_mmdgm/data"

1. VA on MNIST with pre-training, lower bound (with visualization): run_va.sh
You can also train the model without pre-training by setting the [pretrain] flag to 0 in the .sh file.

2. VA on MNIST, error rate: python pegasos.py dir/full_latent.mat mnist

3. MMVA on MNIST with pre-training (with visualization): run_mmva.sh
You can also train the model without pre-training by setting the [pretrain] flag to 0 in the .sh file.<commit_msg>Links for full version with trained models<commit_after>Code for NIPS 2015 paper on Max-margin Deep Generative Models(MMDGM)

<NAME>, <NAME>, <NAME> and <NAME>,
Max-margin Deep Generative Models 
Advances in Neural Information Processing Systems (NIPS15), Montreal  
Please cite this paper when using this code for your research.

For questions and bug reports, please send me an e-mail at _chongxuanli1991[at]gmail.com_.


## Prerequisites

1. Some libs we used in our experiments:
    - Python (version 2.7)
	- Numpy
    - Scipy
    - Theano (version 0.7.0)
    - Cuda (7.0)
    - Cudnn (optional, see details below, version 6.5)
    - Pylearn2 (for pre-processing of SVHN)

2. GPU: TITAN X Black
    - memory of GPU is at least 9G for SVHN

## Some matters

1. We found that theano is computationally unstable with Cudnn. For MNIST, we do NOT use Cudnn and the error rate should be exact 0.45% with same version of libs and machine. For SVHN, we DO use Cudnn for faster training and there exists additional randomness even given fixed random seed. However, we run this experiments for 5 times and choose the lowest accuracy to report. Typically, the generative results won't be changed. 

2. In MLP case, we use code from Kingma[https://github.com/dpkingma/nips14-ssl]. In CNN case, we use code from Goodfellow to do local contrast normalization(LCN) for SVHN data(pylearn2). We also use code from the tutorial on deeplearning.org.

3. You may need to resave .sh file in unix format to avoid bugs for strings at the end of a line. Further, you should run the command chmod +x *.sh.

4. I didn't upload our trained model to github and you may need to train models following the command below and put models in the right place. For a full version of code with trained models, you can access http://ml.cs.tsinghua.edu.cn/~chongxuan/static/mmdgm_release.rar.

## MLP results, code is in folder mlp-mmdgm

0. We did our experiments based on Kingma's code[https://github.com/dpkingma/nips14-ssl]. 
You should run this command at first: export ML_DATA_PATH="[dir]/mmdgm/mlp_mmdgm/data"

1. VA on MNIST with pre-training, lower bound (with visualization): run_va.sh
You can also train the model without pre-training by setting the [pretrain] flag to 0 in the .sh file.

2. VA on MNIST, error rate: python pegasos.py dir/full_latent.mat mnist

3. MMVA on MNIST with pre-training (with visualization): run_mmva.sh
You can also train the model without pre-training by setting the [pretrain] flag to 0 in the .sh file.
{'fraction_non_alphanumeric': 0.07018659881255301, 'fraction_numerical': 0.017599660729431723, 'mean_word_length': 3.8083588175331293, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 11, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2156526', 'n_tokens_mistral': 1513, 'n_tokens_neox': 1409, 'n_words': 701}