<commit_before><commit_msg>First draft of README for exp/wsj<commit_after>To reproduce our Wall Street Journal (WSJ) experiments, please follow the instruction below:

1. Compile a Fuel-compatible dataset file in HDF5 format. The resulting file `wsj.h5` 
should be put to $FUEL_DATA_PATH folder.

**TODO**

2. Compile language model FST's from ARPA-format language models provided with WSJ.

**TODO**

3. Train the model:

`$LVSR/bin/run.py train $LVSR/lvsr/configs/wsj_paper6.yaml`

4. Decode the model on the validation and training datasets. 

`$LVSR/exp/wsj/decode.sh wsj_paper6 <beam-size>`

We typically use beam size 200 to get the best performance. However, with beam size 
the scores are typically only 10\% worse and decoding is **much** faster.

5. Score the recognized transcripts:

**TODO**
{'fraction_non_alphanumeric': 0.08841843088418432, 'fraction_numerical': 0.017434620174346202, 'mean_word_length': 4.506849315068493, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 4, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2309736', 'n_tokens_mistral': 273, 'n_tokens_neox': 252, 'n_words': 104}