task_categories:automatic-speech-recognition language:en license:other region:us
GigaSpeech is an evolving, multi-domain English speech recognition corpus with 10,000 hours of high quality
labeled audio suitable for supervised training, and 40,000 hours of total audio suitable for semi-supervised
and unsupervised training. Around 40,000 hours of transcribed audio is first collected from audiobooks, podcasts
and YouTube, covering both read and spontaneous speaking styles, and a variety of topics, such as arts, science,
sports, etc. A new forced alignment and segmentation pipeline is proposed to create sentence segments suitable
for speech recognition training, and to filter out segments with low-quality transcription. For system training,
GigaSpeech provides five subsets of different sizes, 10h, 250h, 1000h, 2500h, and 10000h.
For our 10,000-hour XL training subset, we cap the word error rate at 4% during the filtering/validation stage,
and for all our other smaller training subsets, we cap it at 0%. The DEV and TEST evaluation sets, on the other hand,
are re-processed by professional human transcribers to ensure high transcription quality.
@article{DBLP:journals/corr/abs-2106-06909,
  author    = {Guoguo Chen and
               Shuzhou Chai and
               Guanbo Wang and
               Jiayu Du and
               Wei{-}Qiang Zhang and
               Chao Weng and
               Dan Su and
               Daniel Povey and
               Jan Trmal and
               Junbo Zhang and
               Mingjie Jin and
               Sanjeev Khudanpur and
               Shinji Watanabe and
               Shuaijiang Zhao and
               Wei Zou and
               Xiangang Li and
               Xuchen Yao and
               Yongqing Wang and
               Yujun Wang and
               Zhao You and
               Zhiyong Yan},
  title     = {GigaSpeech: An Evolving, Multi-domain {ASR} Corpus with 10, 000 Hours
               of Transcribed Audio},
  journal   = {CoRR},
  volume    = {abs/2106.06909},
  year      = {2021},
  url       = {https://arxiv.org/abs/2106.06909},
  eprinttype = {arXiv},
  eprint    = {2106.06909},
  timestamp = {Wed, 29 Dec 2021 14:29:26 +0100},
  biburl    = {https://dblp.org/rec/journals/corr/abs-2106-06909.bib},
  bibsource = {dblp computer science bibliography, https://dblp.org}
}
---
license: other
task_categories:
- automatic-speech-recognition
language:
- en
extra_gated_prompt: |-
    SpeechColab does not own the copyright of the audio files. For researchers and educators who wish to use the audio files for non-commercial research and/or educational purposes, we can provide access through the Hub under certain conditions and terms. 
    Terms of Access:
    The "Researcher" has requested permission to use the GigaSpeech database (the "Database") at Tsinghua University. In exchange for such permission, Researcher hereby agrees to the following terms and conditions:
    1. Researcher shall use the Database only for non-commercial research and educational purposes.
    2. The SpeechColab team and Tsinghua University make no representations or warranties regarding the Database, including but not limited to warranties of non-infringement or fitness for a particular purpose.
    3. Researcher accepts full responsibility for his or her use of the Database and shall defend and indemnify the SpeechColab team and Tsinghua University, including their employees, Trustees, officers and agents, against any and all claims arising from Researcher's use of the Database, including but not limited to Researcher's use of any copies of copyrighted audio files that he or she may create from the Database.
    4. Researcher may provide research associates and colleagues with access to the Database provided that they first agree to be bound by these terms and conditions.
    5. The SpeechColab team and Tsinghua University reserve the right to terminate Researcher's access to the Database at any time.
    6. If Researcher is employed by a for-profit, commercial entity, Researcher's employer shall also be bound by these terms and conditions, and Researcher hereby represents that he or she is fully authorized to enter into this agreement on behalf of such employer.
    
    Please also fill out the Google Form https://forms.gle/UuGQAPyscGRrUMLq6 to request access to the GigaSpeech dataset.
extra_gated_fields:
 Name: text
 Email: text
 Organization: text
 Address: text 
 I hereby confirm that I have requested access via the Google Form provided above: checkbox
 I accept the terms of access: checkbox
---

# Distil Whisper: GigaSpeech With Timestamps

This is a variant of the [GigaSpeech](https://huggingface.co/datasets/speechcolab/gigaspeech) dataset, augmented to return the pseudo-labelled Whisper 
Transcriptions alongside the original dataset elements. The pseudo-labelled transcriptions were generated by 
labelling the input audio data with the Whisper [large-v2](https://huggingface.co/openai/whisper-large-v2)
model with *greedy* sampling and timestamp prediction. For information on how the original dataset was curated, refer to the original 
[dataset card](https://huggingface.co/datasets/speechcolab/gigaspeech).

## Standalone Usage

First, install the latest version of the 🤗 Datasets package:

```bash
pip install --upgrade pip
pip install --upgrade datasets[audio]
```

The dataset can be downloaded and pre-processed on disk using the [`load_dataset`](https://huggingface.co/docs/datasets/v2.14.5/en/package_reference/loading_methods#datasets.load_dataset) 
function:

```python
from datasets import load_dataset

dataset = load_dataset("distil-whisper/gigaspeech-l", "l")
# take the first sample of the validation set
sample = dataset["validation"][0]
```

It can also be streamed directly from the Hub using Datasets' [streaming mode](https://huggingface.co/blog/audio-datasets#streaming-mode-the-silver-bullet).
Loading a dataset in streaming mode loads individual samples of the dataset at a time, rather than downloading the entire 
dataset to disk:

```python
from datasets import load_dataset

dataset = load_dataset("distil-whisper/gigaspeech-l", "l", streaming=True)
# take the first sample of the validation set
sample = next(iter(dataset["validation"]))
```

## Distil Whisper Usage

To use this dataset to reproduce a Distil Whisper training run, refer to the instructions on the 
[Distil Whisper repository](https://github.com/huggingface/distil-whisper#training).

## License

This dataset is licensed under custom terms. To view the custom license for this dataset, refer to the original [dataset card](https://huggingface.co/datasets/speechcolab/gigaspeech).

