task_categories:audio-classification task_ids:keyword-spotting annotations_creators:other language_creators:crowdsourced multilinguality:monolingual size_categories:100K<n<1M size_categories:10K<n<100K source_datasets:extended|speech_commands language:en license:cc-by-4.0 spotlight enriched renumics enhanced audio classification extended arxiv:1804.03209 region:us
---
annotations_creators:
- other
language_creators:
- crowdsourced
language:
- en
license:
- cc-by-4.0
multilinguality:
- monolingual
size_categories:
- 100K<n<1M
- 10K<n<100K
source_datasets:
- extended|speech_commands
task_categories:
- audio-classification
task_ids:
- keyword-spotting
pretty_name: SpeechCommands
config_names:
- v0.01
- v0.02
tags:
- spotlight
- enriched
- renumics
- enhanced
- audio
- classification
- extended
dataset_info:
- config_name: enrichment_only
  features:
  - name: label_string
    dtype: string
  - name: probability
    dtype: float64
  - name: probability_vector
    sequence: float32
  - name: prediction
    dtype: int64
  - name: prediction_string
    dtype: string
  - name: embedding_reduced
    sequence: float32
  splits:
  - name: train
    num_bytes: 8763867
    num_examples: 51093
  - name: validation
    num_bytes: 1165942
    num_examples: 6799
  - name: test
    num_bytes: 528408
    num_examples: 3081
  download_size: 0
  dataset_size: 10458217
- config_name: raw_and_enrichment_combined
  features:
  - name: file
    dtype: string
  - name: audio
    dtype:
      audio:
        sampling_rate: 16000
  - name: label
    dtype:
      class_label:
        names:
          '0': 'yes'
          '1': 'no'
          '2': up
          '3': down
          '4': left
          '5': right
          '6': 'on'
          '7': 'off'
          '8': stop
          '9': go
          '10': zero
          '11': one
          '12': two
          '13': three
          '14': four
          '15': five
          '16': six
          '17': seven
          '18': eight
          '19': nine
          '20': bed
          '21': bird
          '22': cat
          '23': dog
          '24': happy
          '25': house
          '26': marvin
          '27': sheila
          '28': tree
          '29': wow
          '30': _silence_
  - name: is_unknown
    dtype: bool
  - name: speaker_id
    dtype: string
  - name: utterance_id
    dtype: int8
  - name: logits
    sequence: float64
  - name: embedding
    sequence: float32
  - name: label_string
    dtype: string
  - name: probability
    dtype: float64
  - name: probability_vector
    sequence: float32
  - name: prediction
    dtype: int64
  - name: prediction_string
    dtype: string
  - name: embedding_reduced
    sequence: float32
  splits:
  - name: train
    num_bytes: 1803565876.375
    num_examples: 51093
  - name: validation
    num_bytes: 240795605.125
    num_examples: 6799
  - name: test
    num_bytes: 109673146.875
    num_examples: 3081
  download_size: 0
  dataset_size: 2154034628.375
configs:
- config_name: enrichment_only
  data_files:
  - split: train
    path: enrichment_only/train-*
  - split: validation
    path: enrichment_only/validation-*
  - split: test
    path: enrichment_only/test-*
- config_name: raw_and_enrichment_combined
  data_files:
  - split: train
    path: raw_and_enrichment_combined/train-*
  - split: validation
    path: raw_and_enrichment_combined/validation-*
  - split: test
    path: raw_and_enrichment_combined/test-*
---

# Dataset Card for SpeechCommands

## Dataset Description

- **Homepage:** [Renumics Homepage](https://renumics.com/?hf-dataset-card=speech-commands-enrichment_only)
- **GitHub** [Spotlight](https://github.com/Renumics/spotlight)
- **Dataset Homepage** [tensorflow.org/datasets](https://www.tensorflow.org/datasets/catalog/speech_commands)
- **Paper:** [Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition](https://arxiv.org/pdf/1804.03209.pdf)
- **Leaderboard:** [More Information Needed]

### Dataset Summary

📊 [Data-centric AI](https://datacentricai.org) principles have become increasingly important for real-world use cases.  
At [Renumics](https://renumics.com/?hf-dataset-card=speech-commands-enriched) we believe that classical benchmark datasets and competitions should be extended to reflect this development. 

🔍 This is why we are publishing benchmark datasets with application-specific enrichments (e.g. embeddings, baseline results, uncertainties, label error scores). We hope this helps the ML community in the following ways:
1. Enable new researchers to quickly develop a profound understanding of the dataset.
2. Popularize data-centric AI principles and tooling in the ML community.
3. Encourage the sharing of meaningful qualitative insights in addition to traditional quantitative metrics.

📚 This dataset is an enriched version of the [SpeechCommands Dataset](https://huggingface.co/datasets/speech_commands).


### Explore the Dataset

There are two configurations of the dataset: **Enrichment only** provides the enrichments calculated by Renumics using the MIT AST transformer, while **raw_and_enrichment_combined** provides a concatenated dataset of the original speech commands and the enrichment.  

The enrichments allow you to quickly gain insights into the dataset. The open source data curation tool [Renumics Spotlight](https://github.com/Renumics/spotlight) enables that with just a few lines of code:

Install datasets and Spotlight via [pip](https://packaging.python.org/en/latest/key_projects/#pip):

```python
!pip install renumics-spotlight datasets[audio]
```
> **_Notice:_**  On Linux, non-Python dependency on libsndfile package must be installed manually. See [Datasets - Installation](https://huggingface.co/docs/datasets/installation#audio) for more information.

Load the dataset from huggingface in your notebook and start exploring with a simple view:

```python
import datasets
from renumics import spotlight

from renumics.spotlight.layouts import debug_classification

dataset = datasets.load_dataset("renumics/speech_commands_enrichment_only", "raw_and_enrichment_combined")
joined_dataset = datasets.concatenate_datasets([dataset["train"], dataset["validation"], dataset["test"]])
layout = debug_classification(label='label_string', prediction='prediction', embedding='embedding_reduced', 
                                    features=["label", "prediction", "probability"], inspect={'audio': spotlight.Audio})
dtypes = {
    "audio": spotlight.Audio,
    "embedding_reduced": spotlight.Embedding
}
spotlight.show(
    joined_dataset,
    dtype=dtypes,
    layout= layout
)
```
You can use the UI to interactively configure the view on the data. Depending on the concrete tasks (e.g. model comparison, debugging, outlier detection) you might want to leverage different enrichments and metadata.

As a plug and play option, you can check out the Huggingface space: [Huggingface Space for speech enrichment](https://huggingface.co/spaces/renumics/speech_commands_enrichment_space)

Alternatively, you can run the notebook exploration.ipynb locally.



### SpeechCommands Dataset

This is a set of one-second .wav audio files, each containing a single spoken
English word or background noise. These words are from a small set of commands, and are spoken by a
variety of different speakers. This data set is designed to help train simple
machine learning models. It is covered in more detail at [https://arxiv.org/abs/1804.03209](https://arxiv.org/abs/1804.03209).

Version 0.01 of the data set (configuration `"v0.01"`) was released on August 3rd 2017 and contains
64,727 audio files. 

Version 0.02 of the data set (configuration `"v0.02"`) was released on April 11th 2018 and 
contains 105,829 audio files.


### Supported Tasks and Leaderboards

* `keyword-spotting`: the dataset can be used to train and evaluate keyword
spotting systems. The task is to detect preregistered keywords by classifying utterances 
into a predefined set of words. The task is usually performed on-device for the 
fast response time. Thus, accuracy, model size, and inference time are all crucial.

### Languages

The language data in SpeechCommands is in English (BCP-47 `en`).

## Dataset Structure

### Data Instances

Example of a core word (`"label"` is a word, `"is_unknown"` is `False`):
```python
{
  "file": "no/7846fd85_nohash_0.wav", 
  "audio": {
    "path": "no/7846fd85_nohash_0.wav", 
    "array": array([ -0.00021362, -0.00027466, -0.00036621, ...,  0.00079346,
          0.00091553,  0.00079346]), 
    "sampling_rate": 16000
    },
  "label": 1,  # "no"
  "is_unknown": False,
  "speaker_id": "7846fd85",
  "utterance_id": 0
}
```

Example of an auxiliary word (`"label"` is a word, `"is_unknown"` is `True`)
```python
{
  "file": "tree/8b775397_nohash_0.wav", 
  "audio": {
    "path": "tree/8b775397_nohash_0.wav", 
    "array": array([ -0.00854492, -0.01339722, -0.02026367, ...,  0.00274658,
          0.00335693,  0.0005188]), 
    "sampling_rate": 16000
    },
  "label": 28,  # "tree"
  "is_unknown": True,
  "speaker_id": "1b88bf70",
  "utterance_id": 0
}
```

Example of background noise (`_silence_`) class:

```python
{
  "file": "_silence_/doing_the_dishes.wav", 
  "audio": {
    "path": "_silence_/doing_the_dishes.wav", 
    "array": array([ 0.        ,  0.        ,  0.        , ..., -0.00592041,
         -0.00405884, -0.00253296]), 
    "sampling_rate": 16000
    }, 
  "label": 30,  # "_silence_"
  "is_unknown": False,
  "speaker_id": "None",
  "utterance_id": 0  # doesn't make sense here
}
```

### Data Fields

* `file`: relative audio filename inside the original archive. 
* `audio`: dictionary containing a relative audio filename, 
a decoded audio array, and the sampling rate. Note that when accessing 
the audio column: `dataset[0]["audio"]` the audio is automatically decoded 
and resampled to `dataset.features["audio"].sampling_rate`. 
Decoding and resampling of a large number of audios might take a significant 
amount of time. Thus, it is important to first query the sample index before 
the `"audio"` column, i.e. `dataset[0]["audio"]` should always be preferred 
over `dataset["audio"][0]`.
* `label`: either word pronounced in an audio sample or background noise (`_silence_`) class.
Note that it's an integer value corresponding to the class name.
* `is_unknown`: if a word is auxiliary. Equals to `False` if a word is a core word or `_silence_`, 
`True` if a word is an auxiliary word. 
* `speaker_id`: unique id of a speaker. Equals to `None` if label is `_silence_`.
* `utterance_id`: incremental id of a word utterance within the same speaker. 

### Data Splits

The dataset has two versions (= configurations): `"v0.01"` and `"v0.02"`. `"v0.02"` 
contains more words (see section [Source Data](#source-data) for more details).

|       | train | validation | test |
|-----  |------:|-----------:|-----:|
| v0.01 | 51093 |       6799 | 3081 |
| v0.02 | 84848 |       9982 | 4890 |

Note that in train and validation sets examples of `_silence_` class are longer than 1 second.
You can use the following code to sample 1-second examples from the longer ones:

```python
def sample_noise(example):
    # Use this function to extract random 1 sec slices of each _silence_ utterance,
    # e.g. inside `torch.utils.data.Dataset.__getitem__()`
    from random import randint

    if example["label"] == "_silence_":
        random_offset = randint(0, len(example["speech"]) - example["sample_rate"] - 1)
        example["speech"] = example["speech"][random_offset : random_offset + example["sample_rate"]]

    return example
```

## Dataset Creation

### Curation Rationale

The primary goal of the dataset is to provide a way to build and test small 
models that can detect a single word from a set of target words and differentiate it 
from background noise or unrelated speech with as few false positives as possible. 

### Source Data

#### Initial Data Collection and Normalization

The audio files were collected using crowdsourcing, see
[aiyprojects.withgoogle.com/open_speech_recording](https://github.com/petewarden/extract_loudest_section)
for some of the open source audio collection code that was used. The goal was to gather examples of
people speaking single-word commands, rather than conversational sentences, so
they were prompted for individual words over the course of a five minute
session. 

In version 0.01 thirty different words were recoded: "Yes", "No", "Up", "Down", "Left",
"Right", "On", "Off", "Stop", "Go", "Zero", "One", "Two", "Three", "Four", "Five", "Six", "Seven", "Eight", "Nine", 
"Bed", "Bird", "Cat", "Dog", "Happy", "House", "Marvin", "Sheila", "Tree", "Wow". 


In version 0.02 more words were added: "Backward", "Forward", "Follow", "Learn", "Visual".

In both versions, ten of them are used as commands by convention: "Yes", "No", "Up", "Down", "Left",
"Right", "On", "Off", "Stop", "Go". Other words are considered to be auxiliary (in current implementation 
it is marked by `True` value of `"is_unknown"` feature). Their function is to teach a model to distinguish core words 
from unrecognized ones.  

The `_silence_` label contains a set of longer audio clips that are either recordings or 
a mathematical simulation of noise.

#### Who are the source language producers?

The audio files were collected using crowdsourcing.

### Annotations

#### Annotation process

Labels are the list of words prepared in advances.
Speakers were prompted for individual words over the course of a five minute
session. 

#### Who are the annotators?

[More Information Needed]

### Personal and Sensitive Information

The dataset consists of people who have donated their voice online.  You agree to not attempt to determine the identity of speakers in this dataset.

## Considerations for Using the Data

### Social Impact of Dataset

[More Information Needed]

### Discussion of Biases

[More Information Needed]

### Other Known Limitations

[More Information Needed]

## Additional Information

### Dataset Curators

[More Information Needed]

### Licensing Information

Creative Commons BY 4.0 License ((CC-BY-4.0)[https://creativecommons.org/licenses/by/4.0/legalcode]).

### Citation Information

```
@article{speechcommandsv2,
   author = { {Warden}, P.},
    title = "{Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition}",
  journal = {ArXiv e-prints},
  archivePrefix = "arXiv",
  eprint = {1804.03209},
  primaryClass = "cs.CL",
  keywords = {Computer Science - Computation and Language, Computer Science - Human-Computer Interaction},
    year = 2018,
    month = apr,
    url = {https://arxiv.org/abs/1804.03209},
}
```
### Contributions

[More Information Needed]