where \(ERR\)  is the sum of errors (Insertions, Deletions or Substitutions) between the transcription and the ground truth. \(N\)  is number of words in ground truth. As evident from this equation, the presence of ground truth is necessary for the calculation of errors, and hence, for WER.
However, manual transcription of speech at word level is a expensive and dilatory process. Hence, the need for an automatic ASR evaluation is important but few attempts have addressed this. Furthermore, for effectively training, evaluating and judging the performance of an ASR system, WER calculation needs to be done on adequate hours of data. As this test set increases, a more accurate estimation of WER is possible. Since automatic WER evaluation does not have the bottleneck of manual transcription, it can be calculated over large test sets leading to more accurate estimates.
Because of the immense popularity of attention based architectures for text classification [1]}, [2]}, [3]}, [4]}, [5]}, we propose the transformer [6]} encoder architecture — Bidirectional Encoder Representations from Transformers (BERT) [7]} for e-WER. BERT is pretrained on huge amounts of open domain language and is extensively used for its effectiveness in natural language understanding tasks. By pretraining on such data, the model gains knowledge of general domain language structure which aids in predicting speech errors which are typically observed as a deviation from the general syntax and semantics of a sentence. While previous approaches address e-WER [8]}, [9]}, [10]} in classification settings, their models suffer with gross imbalance in the WER classes. To address this issue, we present a training framework which will always consist of training on equal sized classes no matter the true WER distribution.
Additionally, these previous e-WER classification tasks assume that there is no inherent relative ordering to the classes. However, this is not the case with WER classification since a misclassification closer to the ground truth will lead to a lower mean absolute error (MAE) in the WER prediction. Such classification tasks are called Ordinal classification[11]}.
