<commit_before># MMT 0.11 - ALPHA RELEASE 

## About MMT
MMT is a context-aware, incremental and distributed general purpose Machine Translation technology.

MMT goal is to make MT easy to adopt and scale.

With MMT you don't need anymore to train multiple custom engines for each of your domains/projects/customers, you can push all your data to a single engine that will automatically and in real-time adapt to the context you provide.

MMT aims to deliver the quality of a custom engine and the low sparsity of your all data combined.

You can find more information on: http://www.modermmt.eu


## About this Release

This application is the binary version of MMT (open source distribution expected by the end 2016). 

This MMT release will allow you to create an MT engine, available via a REST API, given your training data (folder with line aligned text files)
Ex. domain1.en domain1.it domain2.en domain2.it 
In general:
<domain-id>.<2 letters iso lang code|5 letters RFC3066>

Note: domain-id must be [a-zA-Z0-9] only without spaces.

## Your first translation with MMT

Assuming you read INSTALL.

### Create an Engine

We included a very small dataset, just to verify that training works.

```bash
./mmt create engine en it examples/data/train
```

### To Start/Stop an existing engine
```bash
./mmt start|stop
```

### Translate via API

http://localhost:8000/translate?q=party&context=President

### MMT Tuning (Expert)

MMT quality can be increased by tuning the parameters providing unseen translation examples. 

```
./mmt tune examples/data/dev
```

### MMT distributed (Expert)

Let's distribute MMT to a second machine. Login into the new second machine and run

```bash 
./mmt start --master user:pass@3.14.15.16
or for private key auth 
./mmt start --master user@192.168.3.11 --master-pem master-credentials.pem
```

3.14.15.16 being the IP address of the machine where is.
master-credentials.pem being your ssh key to the master machine for rsync.

The engine will be synced from the master and translation requests will be load balanced across the 2 istances.
Only the master will respond to the Translation API and distribute load.

Every time you start a slave the model data will be rsynced to the slave.

Note: rsyncing  of the models has been temporanealy disabled in 0.11 and models files have to be copied manually. To test the release please contact <EMAIL> 

## How to prepare your data

MMT uses standard sentence aligned corpora, optionally divided into files by domain. 

Example:
```
data/microsoft.en
data/microsoft.fr
data/europarl.en
data/europarl.fr
data/wmt10.en
data/wmt10.fr
```

In general:
```
domain-id.(2 letters iso lang code|5 letters RFC3066)
```

Note: domain-id must be [a-zA-Z0-9] only without spaces.<commit_msg>Added Large Model Instructions<commit_after># MMT 0.11 - ALPHA RELEASE 

## About MMT
MMT is a context-aware, incremental and distributed general purpose Machine Translation technology.

MMT goal is to make MT easy to adopt and scale.

With MMT you don't need anymore to train multiple custom engines for each of your domains/projects/customers, you can push all your data to a single engine that will automatically and in real-time adapt to the context you provide.

MMT aims to deliver the quality of a custom engine and the low sparsity of your all data combined.

You can find more information on: http://www.modermmt.eu


## About this Release

This application is the binary version of MMT (open source distribution expected by the end 2016). 

This MMT release will allow you to create an MT engine, available via a REST API, given your training data (folder with line aligned text files)
Ex. domain1.en domain1.it domain2.en domain2.it 
In general:
<domain-id>.<2 letters iso lang code|5 letters RFC3066>

Note: domain-id must be [a-zA-Z0-9] only without spaces.

## Your first translation with MMT

Assuming you read INSTALL.

### Create an Engine

We included a very small dataset, just to verify that training works.

```bash
./mmt create engine en it examples/data/train
```

### To Start/Stop an existing engine
```bash
./mmt start|stop
```

### Translate via API

http://localhost:8000/translate?q=party&context=President

### MMT Tuning (Expert)

MMT quality can be increased by tuning the parameters providing unseen translation examples. 

```
./mmt tune examples/data/dev
```

### Creating a large translation model

You can create a 1B words engine in around 8 hours of training using 16 Cores and 50GB of RAM.

If you want to try, you can download the WMT10 corpus from here:

(WMT 10 Corpus)[http://www.statmt.org/wmt10/training-giga-fren.tar]

Untar the archive and place the ungipped giga-fren.release2.?? corpus in a training directory.


### MMT distributed (Expert)

Let's distribute MMT to a second machine. Login into the new second machine and run

```bash 
./mmt start --master user:pass@3.14.15.16

or for private key auth (eg. AWS)
./mmt start --master user@3.14.15.16 --master-pem master-credentials.pem
```

192.168.3.11 being the IP address of the machine where is.
master-credentials.pem being your ssh key to the master machine for rsync.

The engine files will be synced from the master and translation requests will be load balanced across the 2 instances.
Only the master will respond to the Translation API and distribute load.

If you updated the model on the master, just stop and start the slave and the model data will be rsynced again.

Note: rsyncing  of the models has been temporarily disabled in 0.11 and models files have to be copied manually. To test the release please contact <EMAIL> 

## How to prepare your data

MMT uses standard sentence aligned corpora, optionally divided into files by domain. 

Example:
```
data/microsoft.en
data/microsoft.fr
data/europarl.en
data/europarl.fr
data/wmt10.en
data/wmt10.fr
```

In general:
```
domain-id.(2 letters iso lang code|5 letters RFC3066)
```

Note: domain-id must be [a-zA-Z0-9] only without spaces.
{'fraction_non_alphanumeric': 0.07438431898140392, 'fraction_numerical': 0.022951918244262022, 'mean_word_length': 4.04222972972973, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 6, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6024608', 'n_tokens_mistral': 1849, 'n_tokens_neox': 1689, 'n_words': 848}