<commit_before>0) Preliminaries

Before building you need to decide what language model toolkit (SRI's,
IRST's, or Ken's) you want to use.

If you want to use SRI's, you will need to download its source 
and build it.  The SRILM can be downloaded from 
http://www.speech.sri.com/projects/srilm/download.html .


If you want to use IRST's, you will need to download its source and
build it.  The IRSTLM can be downloaded from either the SourceForge
website
http://sourceforge.net/projects/irstlm
or the official IRSTLM website
http://hlt.fbk.eu/en/irstlm


Ken's LM is included with the Moses distribution.  

IMPORTANT: These instructions are for building the moses decoder ONLY,
the training and tuning SCRIPTS contained in scripts/ must be built
and installed separately.  Also, they may require modification to
work in certain environments.

Versions 1.9 (or higher) of aclocal and automake are required.

For Mac OSX users:
Standard distribution usually includes versions 1.6.
Get correct versions, and set the environment variables ACLOCAL and 
AUTOMAKE to the correct paths in ./regenerate-makefiles.sh.

--------------------------------------------------------------------------

1) Instructions for building with SRILM

Build SRILM according to their release instructions.  Make sure that
you DO NOT override the MACHINE_TYPE variable on the command line when
you do so, as this can lead to problems locating the library.

 ./regenerate-makefiles.sh
 ./configure [--with-srilm=/path/to/srilm] [--with-irstlm=/path/to/irstlm]

The resulting decoder binary will be moses-cmd/src/moses .

--------------------------------------------------------------------------

2) Instructions for building with IRSTLM

Build IRSTLM according to its release instructions.  Make sure that
you DO NOT override the MACHINE_TYPE variable on the command line when
you do so, as this can lead to problems locating the library.

 ./regenerate-makefiles.sh
 ./configure [--with-srilm=/path/to/srilm] [--with-irstlm=/path/to/irstlm]

The resulting decoder binary will be moses-cmd/src/moses .

--------------------------------------------------------------------------

3) Instructions for building with Ken's LM

 ./regenerate-makefiles.sh
 ./configure

The resulting decoder binary will be moses-cmd/src/moses .
Use language model 8 in your moses.ini:
[lmodel-file]
8 0 2 /path/to/lm.arpa

If you don't want Ken's LM, ./configure --without-kenlm

--------------------------------------------------------------------------



ALTERNATIVE WAYS TO BUILD ON UNIX AND OTHER PLATFORMS

Microsoft Windows
-----------------
Tested on 32-bit Windows XP and Vista using Visual Studio 2005.
Again, refer to the old manual
	http://homepages.inf.ed.ac.uk/s0565741/papers/developers-manual.pdf
The Windows build doesn't use the SRI or IRST language model libraries as they can't be compiled 
under Windows using Visual Studio. Instead, an internal language model, which behave like SRILM is used,
however, it can only handle up to trigrams.

Binaries for all external libraries needed can be downloaded from 
	http://www.statmt.org/moses/?n=Moses.LibrariesUsed

Only the decoder is developed and tested under Windows. There are difficulties using the training scripts under Windows, even with Cygwin.

<commit_msg>testing<commit_after>0) Preliminaries

Before building you need to decide what language model toolkit (SRI's,
IRST's, or Ken's) you want to use.

If you want to use SRI's, you will need to download its source 
and build it.  The SRILM can be downloaded from 
http://www.speech.sri.com/projects/srilm/download.html .


If you want to use IRST's, you will need to download its source and
build it.  The IRSTLM can be downloaded from either the SourceForge
website
http://sourceforge.net/projects/irstlm
or the official IRSTLM website
http://hlt.fbk.eu/en/irstlm


Ken's LM is included with the Moses distribution.  

IMPORTANT: These instructions are for building the moses decoder ONLY,
the training and tuning SCRIPTS contained in scripts/ must be built
and installed separately.  Also, they may require modification to
work in certain environments.

Versions 1.9 (or higher) of aclocal and automake are required.

For Mac OSX users:
Standard distribution usually includes versions 1.6.
Get correct versions, and set the environment variables ACLOCAL and 
AUTOMAKE to the correct paths in ./regenerate-makefiles.sh.

--------------------------------------------------------------------------

1) Instructions for building with SRILM

Build SRILM according to their release instructions.  Make sure that
you DO NOT override the MACHINE_TYPE variable on the command line when
you do so, as this can lead to problems locating the library.

 ./regenerate-makefiles.sh
 ./configure [--with-srilm=/path/to/srilm] [--with-irstlm=/path/to/irstlm]

The resulting decoder binary will be moses-cmd/src/moses .

--------------------------------------------------------------------------

2) Instructions for building with IRSTLM

Build IRSTLM according to its release instructions.  Make sure that
you DO NOT override the MACHINE_TYPE variable on the command line when
you do so, as this can lead to problems locating the library.

 ./regenerate-makefiles.sh
 ./configure [--with-srilm=/path/to/srilm] [--with-irstlm=/path/to/irstlm]

The resulting decoder binary will be moses-cmd/src/moses .

--------------------------------------------------------------------------

3) Instructions for building with Ken's LM

 ./regenerate-makefiles.sh
 ./configure

The resulting decoder binary will be moses-cmd/src/moses .
Use language model 8 in your moses.ini:
[lmodel-file]
8 0 2 /path/to/lm.arpa

If you don't want Ken's LM, ./configure --without-kenlm

--------------------------------------------------------------------------



ALTERNATIVE WAYS TO BUILD ON UNIX AND OTHER PLATFORMS

Microsoft Windows
-----------------
Tested on 32-bit Windows XP and Vista using Visual Studio 2005.
Again, refer to the old manual
	http://homepages.inf.ed.ac.uk/s0565741/papers/developers-manual.pdf
The Windows build doesn't use the SRI or IRST language model libraries as they can't be compiled 
under Windows using Visual Studio. Instead, an internal language model, which behave like SRILM is used,
however, it can only handle up to trigrams.

Binaries for all external libraries needed can be downloaded from 
	http://www.statmt.org/moses/?n=Moses.LibrariesUsed

Only the decoder is developed and tested under Windows. There are difficulties using the training scripts under Windows, even with Cygwin.



{'fraction_non_alphanumeric': 0.15384615384615385, 'fraction_numerical': 0.007616146230007616, 'mean_word_length': 2.647777777777778, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 12, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6306098', 'n_tokens_mistral': 1856, 'n_tokens_neox': 1713, 'n_words': 790}