<commit_before>## Sequence-to-Sequence Learning with Attentional Neural Networks

[Torch](http://torch.ch) implementation of a standard sequence-to-sequence model with attention
where the encoder-decoder are LSTMs. Also has the option to use characters
(instead of input word embeddings) by running a convolutional neural network followed by a
highway network over character embeddings to use as inputs.

The attention model is from
[Effective Approaches to Attention-based
Neural Machine Translation](http://stanford.edu/~lmthang/data/papers/emnlp15_attn.pdf),
Luong et al. EMNLP 2015. We use the *global-attention* model with the *input-feeding* approach
from the paper.

The character model is from [Character-Aware Neural
Language Models](http://arxiv.org/abs/1508.06615), Kim et al. AAAI 2016.

### Dependencies

#### Python
* h5py
* numpy

#### Lua 
You will need the following packages:
* hdf5
* nngraph

GPU usage will additionally require:
* cutorch
* cunn

If running the character model, you should also install:
* cudnn
* luautf8

**Note on CUDA: For some reason the gradients are incorrectly calculated with CUDA 7.5 and hence the
model fails to train. See [here](https://github.com/harvardnlp/seq2seq-attn/issues/10)
for more details. For now, models should be trained with CUDA 7.0.**

### Quickstart

We are going to be working with some example data in `data/` folder.
First run the data-processing code

```
python preprocess.py --srcfile data/src-train.txt --targetfile data/targ-train.txt
--srcvalfile data/src-val.txt --targetvalfile data/targ-val.txt --outputfile data/demo
```

This will take the source/target train/valid files (`src-train.txt, targ-train.txt,
src-val.txt, targ-val.txt`) and make some hdf5 files to be consumed by Lua.

`demo.src.dict`: Dictionary of source vocab to index mappings.  
`demo.targ.dict`: Dictionary of target vocab to index mappings.  
`demo-train.hdf5`: hdf5 containing the train data.  
`demo-val.hdf5`: hdf5 file containing the validation data.  

The `*.dict` files will be needed when predicting on new data.

Now run the model

```
th train.lua -data_file data/demo-train.hdf5 -val_data_file data/demo-val.hdf5 -savefile demo-model
```
This will run the default model, which consists of a 2-layer LSTM with 500 hidden units
on both the encoder/decoder.
You can also add `-gpuid 1` to use (say) GPU 1 in the cluster.

Now you have a model which you can use to predict on new data. To do this we are
going to be running beam search

```
th beam.lua -model demo-model_final.t7 -src_file data/src-val.txt -out_file pred.txt 
-src_dict data/demo.src.dict -targ_dict data/demo.targ.dict
```
This will output predictions into `pred.txt`. The predictions are going to be quite terrible,
as the demo dataset is small. Try running on some larger datasets! For example you can download
millions of parallel sentences for various language pairs from the [Workshop
on Machine Translation 2015](http://www.statmt.org/wmt15/translation-task.html).

### Details
#### Preprocessing options (`preprocess.py`)

* `srcvocabsize, targetvocabsize`: Size of source/target vocabularies. This is constructed
by taking the top X most frequent words. Rest are replaced with special UNK tokens.  
* `srcfile, targetfile`: Path to source/target training data, where each line represents a single
source/target sequence.  
* `srcvalfile, targetvalfile`: Path to source/target validation data.  
* `batchsize`: Size of each mini-batch.  
* `seqlength`: Maximum sequence length (sequences longer than this are dropped).  <commit_msg>readme typo<commit_after>## Sequence-to-Sequence Learning with Attentional Neural Networks

[Torch](http://torch.ch) implementation of a standard sequence-to-sequence model with attention
where the encoder-decoder are LSTMs. Also has the option to use characters
(instead of input word embeddings) by running a convolutional neural network followed by a
highway network over character embeddings to use as inputs.

The attention model is from
[Effective Approaches to Attention-based
Neural Machine Translation](http://stanford.edu/~lmthang/data/papers/emnlp15_attn.pdf),
Luong et al. EMNLP 2015. We use the *global-attention* model with the *input-feeding* approach
from the paper.

The character model is from [Character-Aware Neural
Language Models](http://arxiv.org/abs/1508.06615), Kim et al. AAAI 2016.

### Dependencies

#### Python
* h5py
* numpy

#### Lua 
You will need the following packages:
* hdf5
* nngraph

GPU usage will additionally require:
* cutorch
* cunn

If running the character model, you should also install:
* cudnn
* luautf8

**Note on CUDA: For some reason the gradients are incorrectly calculated with CUDA 7.5 and hence the
model fails to train. See [here](https://github.com/harvardnlp/seq2seq-attn/issues/10)
for more details. For now, models should be trained with CUDA 7.0.**

### Quickstart

We are going to be working with some example data in `data/` folder.
First run the data-processing code

```
python preprocess.py --srcfile data/src-train.txt --targetfile data/targ-train.txt
--srcvalfile data/src-val.txt --targetvalfile data/targ-val.txt --outputfile data/demo
```

This will take the source/target train/valid files (`src-train.txt, targ-train.txt,
src-val.txt, targ-val.txt`) and make some hdf5 files to be consumed by Lua.

`demo.src.dict`: Dictionary of source vocab to index mappings.  
`demo.targ.dict`: Dictionary of target vocab to index mappings.  
`demo-train.hdf5`: hdf5 containing the train data.  
`demo-val.hdf5`: hdf5 file containing the validation data.  

The `*.dict` files will be needed when predicting on new data.

Now run the model

```
th train.lua -data_file data/demo-train.hdf5 -val_data_file data/demo-val.hdf5 -savefile demo-model
```
This will run the default model, which consists of a 2-layer LSTM with 500 hidden units
on both the encoder/decoder.
You can also add `-gpuid 1` to use (say) GPU 1 in the cluster.

Now you have a model which you can use to predict on new data. To do this we are
going to be running beam search

```
th beam.lua -model demo-model_final.t7 -src_file data/src-val.txt -output_file pred.txt 
-src_dict data/demo.src.dict -targ_dict data/demo.targ.dict
```
This will output predictions into `pred.txt`. The predictions are going to be quite terrible,
as the demo dataset is small. Try running on some larger datasets! For example you can download
millions of parallel sentences for various language pairs from the [Workshop
on Machine Translation 2015](http://www.statmt.org/wmt15/translation-task.html).

### Details
#### Preprocessing options (`preprocess.py`)

* `srcvocabsize, targetvocabsize`: Size of source/target vocabularies. This is constructed
by taking the top X most frequent words. Rest are replaced with special UNK tokens.  
* `srcfile, targetfile`: Path to source/target training data, where each line represents a single
source/target sequence.  
* `srcvalfile, targetvalfile`: Path to source/target validation data.  
* `batchsize`: Size of each mini-batch.  
* `seqlength`: Maximum sequence length (sequences longer than this are dropped).  
{'fraction_non_alphanumeric': 0.08893252910646654, 'fraction_numerical': 0.013746668536961706, 'mean_word_length': 4.32089552238806, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6824952', 'n_tokens_mistral': 2230, 'n_tokens_neox': 2073, 'n_words': 893}