task_categories:automatic-speech-recognition language:en license:cc-by-4.0 region:us
---
license: cc-by-4.0
task_categories:
- automatic-speech-recognition
language:
- en
---


# **RED-ACE**

## Dataset Summary

This dataset can be used to train and evaluate **ASR Error Detection or Correction** models. It was introduced in the [RED-ACE paper (Gekhman et al, 2022)](https://aclanthology.org/2022.emnlp-main.180.pdf).

The dataset contains ASR outputs on the LibriSpeech corpus [(Panayotov et al., 2015)](https://ieeexplore.ieee.org/document/7178964) with annotated transcription errors.

## Dataset Details

The LibriSpeech corpus was decoded using [Google Cloud Speech-to-Text API](https://cloud.google.com/speech-to-text), with the **default** and **video** [models](https://cloud.google.com/speech-to-text/docs/speech-to-text-requests#select-model).
The [word-level confidence](https://cloud.google.com/speech-to-text/docs/word-confidence#word-level_confidence) was enabled and is provided as part of the transcription hypothesis.
To annotate word-level errors (for the error detection task), the hypothesis words were aligned with the reference (correct) transcription to find an edit path
(insertions, deletions and substitutions) with the minimum edit distance (from the hypothesis to the reference).
The hypothesis words with deletions and substitutions were then labeled as ERROR (1), the rest were labeled as NOTERROR (0).

## Data format

The dataset has train, developement and test splits which correspond to the splits in Librispeech.

The data contains json lines with the following keys (note that asr_hypothesis[i], confidence_scores[i] and error_labels[i] correpond to the same word):

- `"id"` - The librispeech id.
- `"truth"` - The reference (correct) transcript from Librispeech.
- `"asr_model"` - The ASR [model](https://cloud.google.com/speech-to-text/docs/speech-to-text-requests#select-model) used for transcription.
- `"librispeech_pool"`: Corresponds to the original pool (split) in the librispeech data.
- `"asr_hypothesis"` - The transcription hypothesis.
- `"confidence_scores"` - The [word-level confidence scores](https://cloud.google.com/speech-to-text/docs/word-confidence#word-level_confidence) provided as part of the transcription hypothesis.
- `"error_labels"` - The error labels (1 error, 0 not error) that were obtained by alighning the hypothesis and the reference.



Here is an example of a single data item:


```json
{
  "id": "test-other/6070/86744/6070-86744-0024",
  "truth": "my dear franz replied albert when upon receipt of my letter you found the necessity of asking the count's assistance you promptly went to him saying my friend albert de morcerf is in danger help me to deliver him",
  "asr_model": "default",
  "librispeech_pool": "other",
  "asr_hypothesis": ["my", "dear", "friends", "replied", "Albert", "received", "my", "letter", "you", "found", "the", "necessity", "of", "asking", "the", "county", "assistance", "you", "promptly", "went", "to", "him", "saying", "my", "friend", "all", "but", "the", "most", "stuff", "is", "in", "danger", "help", "me", "to", "deliver", "it"],
  "confidence_scores": ["0.9876290559768677", "0.9875272512435913", "0.6921446323394775", "0.9613730311393738", "0.9413103461265564", "0.6563355922698975", "0.9876290559768677", "0.9876290559768677", "0.9876290559768677", "0.9876290559768677", "0.9876290559768677", "0.9876290559768677", "0.9876290559768677", "0.9876290559768677", "0.9876290559768677", "0.9876290559768677", "0.9876290559768677", "0.9876290559768677", "0.9876290559768677", "0.9876290559768677", "0.9876290559768677", "0.9876290559768677", "0.9876290559768677", "0.9876290559768677", "0.9876290559768677", "1.0", "1.0", "1.0", "1.0", "1.0", "0.9876290559768677", "0.9876290559768677", "0.9876290559768677", "0.9876290559768677", "0.9876290559768677", "0.9876290559768677", "0.5291957855224609", "0.5291957855224609"],
  "error_labels": ["0", "0", "1", "0", "0", "1", "0", "0", "0", "0", "0", "0", "0", "0", "0", "1", "0", "0", "0", "0", "0", "0", "0", "0", "0", "1", "1", "1", "1", "1", "0", "0", "0", "0", "0", "0", "0", "1"]
}
``` 

## Loading the dataset

The following code loads the dataset and locates the example data item from above:

```python
from datasets import load_dataset

red_ace_data = load_dataset("google/red_ace_asr_error_detection_and_correction", split='test')

for example in red_ace_data:
  if example['id'] == 'test-other/6070/86744/6070-86744-0024':
    break
print(example)  
```

## Citation

If you use this dataset for a research publication, please cite the **RED-ACE paper** (using the bibtex entry below), as well as the **Librispeech paper** mentioned above.


```
@inproceedings{gekhman-etal-2022-red,
    title = "{RED}-{ACE}: Robust Error Detection for {ASR} using Confidence Embeddings",
    author = "Gekhman, Zorik  and
      Zverinski, Dina  and
      Mallinson, Jonathan  and
      Beryozkin, Genady",
    booktitle = "Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing",
    month = dec,
    year = "2022",
    address = "Abu Dhabi, United Arab Emirates",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2022.emnlp-main.180",
    doi = "10.18653/v1/2022.emnlp-main.180",
    pages = "2800--2808",
    abstract = "ASR Error Detection (AED) models aim to post-process the output of Automatic Speech Recognition (ASR) systems, in order to detect transcription errors. Modern approaches usually use text-based input, comprised solely of the ASR transcription hypothesis, disregarding additional signals from the ASR model. Instead, we utilize the ASR system{'}s word-level confidence scores for improving AED performance. Specifically, we add an ASR Confidence Embedding (ACE) layer to the AED model{'}s encoder, allowing us to jointly encode the confidence scores and the transcribed text into a contextualized representation. Our experiments show the benefits of ASR confidence scores for AED, their complementary effect over the textual signal, as well as the effectiveness and robustness of ACE for combining these signals. To foster further research, we publish a novel AED dataset consisting of ASR outputs on the LibriSpeech corpus with annotated transcription errors.",
}
```