<commit_before>This is the code used for the experiments in the paper ["A Theoretically Grounded Application of Dropout in Recurrent Neural Networks"](http://mlg.eng.cam.ac.uk/yarin/publications.html#Gal2015Theoretically). The [sentiment analysis experiment](Sentiment_analysis_code/) relies on a [fork of keras](https://github.com/yaringal/keras/tree/BayesianRNN) which implements Bayesian LSTM, Bayesian GRU, embedding dropout, and MC dropout. The [language model experiment](LM_code/) extends [wojzaremba's lua code](https://github.com/wojzaremba/lstm).

## Update 1 (Feb 22): 
[keras](https://github.com/fchollet/keras) now supports dropout in RNNs following the implementation above. A simplified example of the [sentiment analysis experiment](Sentiment_analysis_code/) using the latest keras implementation is given in [here](Example/).

## Update 2 (March 28): 
The script [main_new_dropout_SOTA](LM_code/main_new_dropout_SOTA.lua) implements Bayesian LSTM (Gal, 2015) for the large model of Zaremba et al. (2014). In the setting of Zaremba et al. the states are not reset and the testing is done with a single pass through the test set. The only changes I've made to the setting of Zaremba et al. are:
1. dropout technique (using a Bayesian LSTM)
2. weight decay (which was chosen to be zero in Zaremba et al.)
3. a slightly smaller network was used to fit on my GPU (1250 units per layer instead of 1500)
All other hypers being identical to Zaremba et al.: learning rate decay was not tuned for my setting and is used following Zaremba et al., and the sequences are initialised with the previous state following Zaremba et al. (unlike in main_dropout.lua). Dropout parameters were optimised with grid search (tying dropout_x & dropout_h and dropout_i & dropout_o) over validation perplexity (optimal values are 0.3 and 0.5 compared Zaremba et al.'s 0.6).

Single model test perplexity is improved from Zaremba et al.'s 78.4 to 76.5. Validation perplexity is reduced from 82.2 to 79.1, see [log](LM_code/main_new_dropout_SOTA.log).

References:
<NAME>, "A Theoretically Grounded Application of Dropout in Recurrent Neural Networks", 2015.
Zaremba, <NAME>, "Recurrent neural network regularization", 2014.<commit_msg>formatting<commit_after>This is the code used for the experiments in the paper ["A Theoretically Grounded Application of Dropout in Recurrent Neural Networks"](http://mlg.eng.cam.ac.uk/yarin/publications.html#Gal2015Theoretically). The [sentiment analysis experiment](Sentiment_analysis_code/) relies on a [fork of keras](https://github.com/yaringal/keras/tree/BayesianRNN) which implements Bayesian LSTM, Bayesian GRU, embedding dropout, and MC dropout. The [language model experiment](LM_code/) extends [wojzaremba's lua code](https://github.com/wojzaremba/lstm).

## Update 1 (Feb 22): 
[keras](https://github.com/fchollet/keras) now supports dropout in RNNs following the implementation above. A simplified example of the [sentiment analysis experiment](Sentiment_analysis_code/) using the latest keras implementation is given in [here](Example/).

## Update 2 (March 28): 
The script [main_new_dropout_SOTA](LM_code/main_new_dropout_SOTA.lua) implements Bayesian LSTM (Gal, 2015) for the large model of Zaremba et al. (2014). In the setting of Zaremba et al. the states are not reset and the testing is done with a single pass through the test set. The only changes I've made to the setting of Zaremba et al. are:

1. dropout technique (using a Bayesian LSTM)
2. weight decay (which was chosen to be zero in Zaremba et al.)
3. a slightly smaller network was used to fit on my GPU (1250 units per layer instead of 1500)

All other hypers being identical to Zaremba et al.: learning rate decay was not tuned for my setting and is used following Zaremba et al., and the sequences are initialised with the previous state following Zaremba et al. (unlike in main_dropout.lua). Dropout parameters were optimised with grid search (tying dropout_x & dropout_h and dropout_i & dropout_o) over validation perplexity (optimal values are 0.3 and 0.5 compared Zaremba et al.'s 0.6).

Single model test perplexity is improved from Zaremba et al.'s 78.4 to 76.5. Validation perplexity is reduced from 82.2 to 79.1, see [log](LM_code/main_new_dropout_SOTA.log).

References:
<NAME>, "A Theoretically Grounded Application of Dropout in Recurrent Neural Networks", 2015.
<NAME>, <NAME>, "Recurrent neural network regularization", 2014.
{'fraction_non_alphanumeric': 0.07737291947818263, 'fraction_numerical': 0.024741340530814216, 'mean_word_length': 5.050340136054421, 'pattern_counts': {'":': 0, '<': 8, '<?xml version=': 0, '>': 8, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3970627', 'n_tokens_mistral': 1423, 'n_tokens_neox': 1272, 'n_words': 579}