<commit_before>#Deep Learning for Visual Question Answering

This project uses Keras to train a variety of **Feedforward** and **Recurrent Neural Networks** for the task of Visual Question Answering. It is designed to work with the [VQA](http://visualqa.org) dataset. 

Models Implemented:

1. A Feedforward Model
2. An LSTM-based model

##Requirements
1. Keras 0.20
2. spaCy 0.94
3. scikit-learn 0.16
4. progressbar
5. Nvidia CUDA 7.5 (optional, for GPU acceleration)

Tested on Python 2.7 on Ubuntu 14.04 and Centos 7.1.

###Notes:
1. Keras needs the latest Theano, which in turn needs Numpy/Scipy. 
2. spaCy is currently used only for converting questions to a vector (or a sequence of vectors), this dependency can be easily be removed if you want to.

##Get Started
Have a look at the `get_started.sh` script in the `scripts` folder. Also, have a look at the readme present in each of the folders.<commit_msg>Update main readme<commit_after>#Deep Learning for Visual Question Answering
![Results](https://raw.githubusercontent.com/avisingh599/homepage/master/images/vqa/sample_results.jpg)

Accompanying blog post: 

This project uses Keras to train a variety of **Feedforward** and **Recurrent Neural Networks** for the task of Visual Question Answering. It is designed to work with the [VQA](http://visualqa.org) dataset. 

Models Implemented:

1. A Feedforward Model
![MLP Model](https://raw.githubusercontent.com/avisingh599/homepage/master/images/vqa/model_1.jpg)

2. An LSTM-based model
![LSTM Encoder](https://raw.githubusercontent.com/avisingh599/homepage/master/images/vqa/lstm_encoder.jpg)


##Requirements
1. [Keras 0.20](http://keras.io/)
2. [spaCy 0.94](http://spacy.io/)
3. [scikit-learn 0.16](http://scikit-learn.org/)
4. [progressbar](https://pypi.python.org/pypi/progressbar)
5. Nvidia CUDA 7.5 (optional, for GPU acceleration)

Tested with Python 2.7 on Ubuntu 14.04 and Centos 7.1.

###Notes:
1. Keras needs the latest Theano, which in turn needs Numpy/Scipy. 
2. spaCy is currently used only for converting questions to a vector (or a sequence of vectors), this dependency can be easily be removed if you want to.
3. spaCy uses Goldberg and Levy's word vectors by default, but I found the performance to be much superior with Stanford's [Glove word vectors]. 
Performance on the validation set of the [VQA Challenge](http://visualqa.org/challenge.html):

| Model     		   | Accuracy      |
| ---------------------|:-------------:|
| BOW+CNN              | 44.30%		   |
| LSTM-Language only   | 42.51%        |
| LSTM+CNN             | 47.80%        |

![Validation Accuracy with number of epochs](https://raw.githubusercontent.com/avisingh599/homepage/master/images/vqa/learning_curve.jpg)

Notes:

##Get Started
Have a look at the `get_started.sh` script in the `scripts` folder. Also, have a look at the readme present in each of the folders.
{'fraction_non_alphanumeric': 0.11258278145695365, 'fraction_numerical': 0.02858138724294179, 'mean_word_length': 3.7126436781609193, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3326537', 'n_tokens_mistral': 991, 'n_tokens_neox': 884, 'n_words': 342}