<jupyter_start><jupyter_text>Speech Recognition (Recipe)Author: [<NAME>](https://github.com/ShigekiKarita)July 29 2019ESPnet Hackathon 2019 @Tokyo  AbstractThis example shows you a practical ASR example using ESPnet as a command line interface, and also as a library.See also- documetation https://espnet.github.io/espnet/- github https://github.com/espnet  InstallationESPnet depends on Kaldi ASR toolkit and Warp-CTC. This will take a few minutes.<jupyter_code># OS setup
!sudo apt-get install bc tree
!cat /etc/os-release

# espnet setup
!git clone https://github.com/espnet/espnet
!cd espnet; pip install -e .
!mkdir -p espnet/tools/venv/bin; touch espnet/tools/venv/bin/activate

# warp ctc setup
!git clone https://github.com/espnet/warp-ctc -b pytorch-1.1
!cd warp-ctc && mkdir build && cd build && cmake .. && make -j4
!cd warp-ctc/pytorch_binding && python setup.py install 

# kaldi setup
!cd ./espnet/tools; git clone https://github.com/kaldi-asr/kaldi
!echo "" > ./espnet/tools/kaldi/tools/extras/check_dependencies.sh # ignore check
!chmod +x ./espnet/tools/kaldi/tools/extras/check_dependencies.sh
!cd ./espnet/tools/kaldi/tools; make sph2pipe sclite
!rm -rf espnet/tools/kaldi/tools/python
![ ! -e ubuntu16-featbin.tar.gz ] && wget https://18-198329952-gh.circle-artifacts.com/0/home/circleci/repo/ubuntu16-featbin.tar.gz
!tar -xf ./ubuntu16-featbin.tar.gz
!cp featbin/* espnet/tools/kaldi/src/featbin/<jupyter_output><empty_output><jupyter_text>ESPnet command line usage (espnet/egs/xxx)You can use the end-to-end script `run.sh` for reproducing systems reported in `espnet/egs/*/asr1/RESULTS.md`. Typically, we organize `run.sh` with several stages:0. Data download (if available)1. Kaldi-style data preparation 2. Save python-friendly data (e.g., JSON, HDF5, etc)3. Lanuage model training4. ASR model training5. Decoding and evaluation<jupyter_code>!ls espnet/egs<jupyter_output>aishell  cmu_wilderness		  jnas	       ljspeech     timit
ami	 csj			  jsalt18e2e   m_ailabs     voxforge
an4	 fisher_callhome_spanish  jsut	       reverb	    wsj
aurora4  fisher_swbd		  li10	       ru_open_stt  wsj_mix
babel	 hkust			  librispeech  swbd	    yesno
chime4	 hub4_spanish		  libri_trans  tedlium2
chime5	 iwslt18		  libritts     tedlium3<jupyter_text>Stage 0 - 2 Data preparationFor example, if you add `--stop-stage 2`, you can stop the script before neural network training.<jupyter_code>!cd espnet/egs/an4/asr1; ./run.sh  --ngpu 1 --stop-stage 2<jupyter_output>stage -1: Data Download
local/download_and_untar.sh: an4 directory already exists in ./downloads
stage 0: Data preparation
stage 1: Feature Generation
steps/make_fbank_pitch.sh --cmd run.pl --nj 8 --write_utt2num_frames true data/test exp/make_fbank/test fbank
steps/make_fbank_pitch.sh: moving data/test/feats.scp to data/test/.backup
utils/validate_data_dir.sh: Successfully validated data-directory data/test
steps/make_fbank_pitch.sh: [info]: no segments file exists: assuming wav.scp indexed by utterance.
steps/make_fbank_pitch.sh: Succeeded creating filterbank and pitch features for test
fix_data_dir.sh: kept all 130 utterances.
fix_data_dir.sh: old files are kept in data/test/.backup
steps/make_fbank_pitch.sh --cmd run.pl --nj 8 --write_utt2num_frames true data/train exp/make_fbank/train fbank
steps/make_fbank_pitch.sh: moving data/train/feats.scp to data/train/.backup
utils/validate_data_dir.sh: Successfully validated data-directory data/train
steps/make_fbank_pitch.sh: [info]: no s[...]<jupyter_text>Kaldi-style directory structureAlways we organize each recipe placed in `egs/xxx/asr1` in Kaldi way:- `conf/`: kaldi configurations, e.g., speech feature- `data/`: almost raw [data prepared by Kaldi](https://kaldi-asr.org/doc/data_prep.html)- `exp/`: intermidiate files through experiments, e.g., log files, model parameters- `fbank/`: speech feature binary files, e.g., [ark, scp](https://kaldi-asr.org/doc/io.html)- `dump/`: ESPnet meta data for tranining, e.g., json, hdf5- `local/`: corpus specific data preparation scripts- [steps/](https://github.com/kaldi-asr/kaldi/tree/master/egs/wsj/s5/steps), [utils/](https://github.com/kaldi-asr/kaldi/tree/master/egs/wsj/s5/utils): Kaldi's helper scripts<jupyter_code>!tree -L 1 espnet/egs/an4/asr1<jupyter_output>espnet/egs/an4/asr1
├── cmd.sh
├── conf
├── data
├── downloads
├── dump
├── exp
├── fbank
├── local
├── path.sh
├── RESULTS
├── run.sh
├── steps -> ../../../tools/kaldi/egs/wsj/s5/steps
└── utils -> ../../../tools/kaldi/egs/wsj/s5/utils

9 directories, 4 files<jupyter_text>TIPS: essential files in data preparationTo create a new recipe, all you need is stage 1 that creates key-value pair files: - speech`data/xxx/wav.scp`- text`data/xxx/text`  raw speech file list<jupyter_code>!head /mnt/c/Users/User/Dropbox/rtmp/src/python/notebooks/espnet/egs/an4/asr1/data/train/wav.scp<jupyter_output>fash-an251-b /mnt/c/Users/User/Dropbox/rtmp/src/python/notebooks/espnet/egs/an4/asr1/../../../tools/kaldi/tools/sph2pipe_v2.5/sph2pipe -f wav -p -c 1 ./downloads/an4/wav/an4_clstk/fash/an251-fash-b.sph |
fash-an253-b /mnt/c/Users/User/Dropbox/rtmp/src/python/notebooks/espnet/egs/an4/asr1/../../../tools/kaldi/tools/sph2pipe_v2.5/sph2pipe -f wav -p -c 1 ./downloads/an4/wav/an4_clstk/fash/an253-fash-b.sph |
fash-an254-b /mnt/c/Users/User/Dropbox/rtmp/src/python/notebooks/espnet/egs/an4/asr1/../../../tools/kaldi/tools/sph2pipe_v2.5/sph2pipe -f wav -p -c 1 ./downloads/an4/wav/an4_clstk/fash/an254-fash-b.sph |
fash-an255-b /mnt/c/Users/User/Dropbox/rtmp/src/python/notebooks/espnet/egs/an4/asr1/../../../tools/kaldi/tools/sph2pipe_v2.5/sph2pipe -f wav -p -c 1 ./downloads/an4/wav/an4_clstk/fash/an255-fash-b.sph |
fash-cen1-b /mnt/c/Users/User/Dropbox/rtmp/src/python/notebooks/espnet/egs/an4/asr1/../../../tools/kaldi/tools/sph2pipe_v2.5/sph2pipe -f wav -p -c 1 ./downloads/an4/wav/an4_clstk/fash/[...]<jupyter_text>raw text list<jupyter_code>!head /mnt/c/Users/User/Dropbox/rtmp/src/python/notebooks/espnet/egs/an4/asr1/data/train/text<jupyter_output>fash-an251-b YES
fash-an253-b GO
fash-an254-b YES
fash-an255-b U M N Y H SIX
fash-cen1-b H I N I C H
fash-cen2-b A M Y
fash-cen4-b M O R E W O O D
fash-cen5-b P I T T S B U R G H
fash-cen7-b TWO SIX EIGHT FOUR FOUR ONE EIGHT
fbbh-an86-b C Z D Z W EIGHT<jupyter_text>TIPS: explore datasets with data.jsonTo explore datasets easily, ESPnet stores metadata `dump/xxx/data.json` in the stage 2.<jupyter_code>import json
import matplotlib.pyplot as plt
import kaldiio

# load 10-th speech/text in data.json
root = "/mnt/c/Users/User/Dropbox/rtmp/src/python/notebooks/espnet/egs/an4/asr1"
with open(root + "/dump/test/deltafalse/data.json", "r") as f:
  test_json = json.load(f)["utts"]
  
key, info = list(test_json.items())[10]

# plot the speech feature
fbank = kaldiio.load_mat(info["input"][0]["feat"])
plt.matshow(fbank.T[::-1])
plt.title(key + ": " + info["output"][0]["text"])

# print the key-value pair
key, info<jupyter_output><empty_output><jupyter_text>Stage 3 - 4 NN TrainingLet's go to the most interesting part...<jupyter_code>!tail /mnt/c/Users/User/Dropbox/rtmp/src/python/notebooks/espnet/egs/an4/asr1/conf/train_mtlalpha1.0.yaml
!pwd
!./run.sh  --ngpu 0 --stage 3 --stop-stage 4 --train-config ./conf/train_mtlalpha1.0.yaml<jupyter_output>dictionary: data/lang_1char/train_nodev_units.txt
stage 3: LM Preparation
WARNING:root:OOV rate = 0.00 %
stage 4: Network Training<jupyter_text>TIPS: change_yaml.pyYou can tweak YAML config by **$(change_yaml.py xxx.yaml -a yyy=zzz)**<jupyter_code>!source path.sh; \
  ./run.sh  --ngpu 0 --stage 4 --stop-stage 4 --use-frontend true \
  --train-config $(change_yaml.py ./conf/train_mtlalpha1.0.yaml -a eunits=100)<jupyter_output>dictionary: data/lang_1char/train_nodev_units.txt
stage 4: Network Training<jupyter_text>TIPS: tensorboardYou can easily monitor effects of the config by tensorboard  Decoding and evaluation decode config (`change_yaml.py` also works)<jupyter_code>!cat /mnt/c/Users/User/Dropbox/rtmp/src/python/notebooks/espnet/egs/an4/asr1/conf/decode_ctcweight1.0.yaml<jupyter_output># decoding parameter
beam-size: 20
penalty: 0.0
maxlenratio: 0.0
minlenratio: 0.0
ctc-weight: 1.0
lm-weight: 1.0<jupyter_text>Command line usage<jupyter_code>!./run.sh --ngpu 0 --stage 5<jupyter_output>dictionary: data/lang_1char/train_nodev_units.txt
stage 5: Decoding
2019-07-28 13:26:38,528 (splitjson:40) INFO: /usr/bin/python3 /content/espnet/egs/an4/asr1/../../../utils/splitjson.py --parts 8 dump/train_dev/deltafalse/data.json
2019-07-28 13:26:38,530 (splitjson:52) INFO: number of utterances = 100
2019-07-28 13:26:38,588 (splitjson:40) INFO: /usr/bin/python3 /content/espnet/egs/an4/asr1/../../../utils/splitjson.py --parts 8 dump/test/deltafalse/data.json
2019-07-28 13:26:38,590 (splitjson:52) INFO: number of utterances = 130
2019-07-28 13:37:48,300 (concatjson:36) INFO: /usr/bin/python3 /content/espnet/egs/an4/asr1/../../../utils/concatjson.py exp/train_nodev_pytorch_train_mtlalpha1.0/decode_train_dev_decode_ctcweight1.0_lm_word100/data.1.json exp/train_nodev_pytorch_train_mtlalpha1.0/decode_train_dev_decode_ctcweight1.0_lm_word100/data.2.json exp/train_nodev_pytorch_train_mtlalpha1.0/decode_train_dev_decode_ctcweight1.0_lm_word100/data.3.json exp/train_nodev_pytorch_train_mtlalp[...]<jupyter_text>ASR result as `data.json`<jupyter_code>!head -n20 /mnt/c/Users/User/Dropbox/rtmp/src/python/notebooks/espnet/egs/an4/asr1/exp/train_nodev_pytorch_train_mtlalpha1.0/decode_test_decode_ctcweight1.0_lm_word100/data.json<jupyter_output>{
    "utts": {
        "fcaw-an406-b": {
            "output": [
                {
                    "name": "target1[1]",
                    "rec_text": "<blank><blank><blank>RUBOU<blank>T<blank><blank><blank> T N E F THREE NINE<eos>",
                    "rec_token": "<blank> <blank> <blank> <NAME> B <NAME> <blank> T <blank> <blank> <blank> <space> T <space> N <space> E <space> F <space> T H R E E <space> N I N E <eos>",
                    "rec_tokenid": "0 0 0 20 23 4 17 23 0 22 0 0 0 2 22 2 16 2 7 2 8 2 22 10 20 7 7 2 16 11 16 7 29",
                    "score": -1.0287089347839355,
                    "shape": [
                        25,
                        30
                    ],
                    "text": "RUBOUT G M E F THREE NINE",
                    "token": "R U B O U T <space> G <space> M <space> E <space> F <space> T H R E E <space> N I N E",
                    "tokenid": "20 23 4 17 23 22 2 9 2 15 2 7 2 8 2 22 10 20 7 7 2 16 11 16 7"
                }
          [...]<jupyter_text>Recognize speech from pythonLet's use  ESPnet as a library and the trained model:<jupyter_code>!ls /mnt/c/Users/User/Dropbox/rtmp/src/python/notebooks/espnet/egs/an4/asr1/exp/train_nodev_pytorch_train_mtlalpha1.0/results<jupyter_output>cer.png		 snapshot.ep.1	 snapshot.ep.14  snapshot.ep.19  snapshot.ep.5
log		 snapshot.ep.10  snapshot.ep.15  snapshot.ep.2	 snapshot.ep.6
loss.png	 snapshot.ep.11  snapshot.ep.16  snapshot.ep.20  snapshot.ep.7
model.json	 snapshot.ep.12  snapshot.ep.17  snapshot.ep.3	 snapshot.ep.8
model.loss.best  snapshot.ep.13  snapshot.ep.18  snapshot.ep.4	 snapshot.ep.9<jupyter_text>recap: load speech from data.json<jupyter_code>import json
import matplotlib.pyplot as plt
import kaldiio

# load 10-th speech/text in data.json
root = "/mnt/c/Users/User/Dropbox/rtmp/src/python/notebooks/espnet/egs/an4/asr1"
with open(root + "/dump/test/deltafalse/data.json", "r") as f:
  test_json = json.load(f)["utts"]
  
key, info = list(test_json.items())[10]

# plot the speech feature
fbank = kaldiio.load_mat(info["input"][0]["feat"])
plt.matshow(fbank.T[::-1])
plt.title(key + ": " + info["output"][0]["text"])<jupyter_output><empty_output><jupyter_text>load model<jupyter_code>import json
import torch
import argparse
from espnet.bin.asr_recog import get_parser
from espnet.nets.pytorch_backend.e2e_asr import E2E

root = "/mnt/c/Users/User/Dropbox/rtmp/src/python/notebooks/espnet/egs/an4/asr1"
model_dir = root + "/exp/train_nodev_pytorch_train_mtlalpha1.0/results"

# load model
with open(model_dir + "/model.json", "r") as f:
  idim, odim, conf = json.load(f)
model = E2E(idim, odim, argparse.Namespace(**conf))
model.load_state_dict(torch.load(model_dir + "/model.loss.best"))
model.cpu().eval()

# load token dict
with open(root + "/data/lang_1char/train_nodev_units.txt", "r") as f:
  token_list = [entry.split()[0] for entry in f]
token_list.insert(0, '<blank>')
token_list.append('<eos>')

# recognize speech
parser = get_parser()
args = parser.parse_args(["--beam-size", "2", "--ctc-weight", "1.0", "--result-label", "out.json", "--model", ""])
result = model.recognize(fbank, args, token_list)
s = "".join(conf["char_list"][y] for y in result[0]["yseq"]).replace("<eos>", "").replace("<space>", " ").replace("<blank>", "")

print("groundtruth:", info["output"][0]["text"])
print("prediction: ", s)
import os
import kaldiio
from IPython.display import Audio


try:
  d = os.getcwd()
  os.chdir(root)
  sr, wav = kaldiio.load_scp("data/test/wav.scp")[key]
finally:
  os.chdir(d)
Audio(wav, rate=sr)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14726550565230676, 'fraction_numerical': 0.04216315307057745, 'mean_word_length': 3.8854477611940297, 'pattern_counts': {'":': 14, '<': 86, '<?xml version=': 0, '>': 89, 'https://': 11, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 11, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '132004', 'license': 'Apache-2.0', 'n_tokens_mistral': 5569, 'n_tokens_neox': 5175, 'n_words': 1083, 'path': 'egs/an4/asr1s/.ipynb_checkpoints/asr_fecli-checkpoint.ipynb', 'repo_name': 'u1273400/espnet', 'size': '13094'}