Add SWRA model

README.md

@@ -0,0 +1,166 @@
+---
+language: en
+datasets:
+- librispeech_asr
+tags:
+- speech
+- audio
+- automatic-speech-recognition
+- hf-asr-leaderboard
+license: mit
+pipeline_tag: automatic-speech-recognition
+widget:
+- example_title: Librispeech sample 1
+  src: https://cdn-media.huggingface.co/speech_samples/sample1.flac
+- example_title: Librispeech sample 2
+  src: https://cdn-media.huggingface.co/speech_samples/sample2.flac
+model-index:
+- name: SWRA (SWARA)
+  results:
+  - task:
+      name: Automatic Speech Recognition
+      type: automatic-speech-recognition
+    dataset:
+      name: LibriSpeech (clean)
+      type: librispeech_asr
+      config: clean
+      split: test
+      args: 
+        language: en
+    metrics:
+    - name: Test WER
+      type: wer
+      value: 4.3
+  - task:
+      name: Automatic Speech Recognition
+      type: automatic-speech-recognition
+    dataset:
+      name: LibriSpeech (other)
+      type: librispeech_asr
+      config: other
+      split: test
+      args: 
+        language: en
+    metrics:
+    - name: Test WER
+      type: wer
+      value: 9.0
+---
+
+# SWRA (SWARA)
+
+`SWRA (SWARA)` is a Speech to Text Transformer (S2T) model trained by @binarybardakshat for automatic speech recognition (ASR). The S2T model was proposed in [this paper](https://arxiv.org/abs/2010.05171) and released in [this repository](https://github.com/pytorch/fairseq/tree/master/examples/speech_to_text).
+
+## Model Description
+
+SWRA (SWARA) is an end-to-end sequence-to-sequence transformer model. It is trained with standard autoregressive cross-entropy loss and generates the transcripts autoregressively.
+
+## Intended Uses & Limitations
+
+This model can be used for end-to-end speech recognition (ASR). See the [model hub](https://huggingface.co/models?filter=speech_to_text) to look for other S2T checkpoints.
+
+### How to Use
+
+As this is a standard sequence-to-sequence transformer model, you can use the `generate` method to generate the transcripts by passing the speech features to the model.
+
+*Note: The `Speech2TextProcessor` object uses [torchaudio](https://github.com/pytorch/audio) to extract the filter bank features. Make sure to install the `torchaudio` package before running this example.*
+
+*Note: The feature extractor depends on [torchaudio](https://github.com/pytorch/audio) and the tokenizer depends on [sentencepiece](https://github.com/google/sentencepiece), so be sure to install those packages before running the examples.*
+
+You could either install those as extra speech dependencies with `pip install transformers"[speech, sentencepiece]"` or install the packages separately with `pip install torchaudio sentencepiece`.
+
+```python
+import torch
+from transformers import Speech2TextProcessor, Speech2TextForConditionalGeneration
+from datasets import load_dataset
+
+model = Speech2TextForConditionalGeneration.from_pretrained("binarybardakshat/swra-swara")
+processor = Speech2TextProcessor.from_pretrained("binarybardakshat/swra-swara")
+
+ds = load_dataset(
+    "patrickvonplaten/librispeech_asr_dummy",
+    "clean",
+    split="validation"
+)
+
+input_features = processor(
+    ds[0]["audio"]["array"],
+    sampling_rate=16_000,
+    return_tensors="pt"
+).input_features  # Batch size 1
+generated_ids = model.generate(input_features=input_features)
+
+transcription = processor.batch_decode(generated_ids)
+
+#### Evaluation on LibriSpeech Test
+
+The following script shows how to evaluate this model on the [LibriSpeech](https://huggingface.co/datasets/librispeech_asr)
+*"clean"* and *"other"* test dataset.
+
+```python
+from datasets import load_dataset
+from evaluate import load
+from transformers import Speech2TextForConditionalGeneration, Speech2TextProcessor
+
+librispeech_eval = load_dataset("librispeech_asr", "clean", split="test")  # change to "other" for other test dataset
+wer = load("wer")
+
+model = Speech2TextForConditionalGeneration.from_pretrained("facebook/s2t-small-librispeech-asr").to("cuda")
+processor = Speech2TextProcessor.from_pretrained("facebook/s2t-small-librispeech-asr", do_upper_case=True)
+
+def map_to_pred(batch):
+    features = processor(batch["audio"]["array"], sampling_rate=16000, padding=True, return_tensors="pt")
+    input_features = features.input_features.to("cuda")
+    attention_mask = features.attention_mask.to("cuda")
+
+    gen_tokens = model.generate(input_features=input_features, attention_mask=attention_mask)
+    batch["transcription"] = processor.batch_decode(gen_tokens, skip_special_tokens=True)[0]
+    return batch
+
+result = librispeech_eval.map(map_to_pred, remove_columns=["audio"])
+
+print("WER:", wer.compute(predictions=result["transcription"], references=result["text"]))
+```
+
+*Result (WER)*:
+
+| "clean" | "other" |
+|:-------:|:-------:|
+| 4.3     | 9.0     |
+
+
+
+## Training data
+
+The S2T-SMALL-LIBRISPEECH-ASR is trained on [LibriSpeech ASR Corpus](https://www.openslr.org/12), a dataset consisting of
+approximately 1000 hours of 16kHz read English speech.
+
+
+## Training procedure
+
+### Preprocessing
+
+The speech data is pre-processed by extracting Kaldi-compliant 80-channel log mel-filter bank features automatically from
+WAV/FLAC audio files via PyKaldi or torchaudio. Further utterance-level CMVN (cepstral mean and variance normalization)
+is applied to each example.
+
+The texts are lowercased and tokenized using SentencePiece and a vocabulary size of 10,000.
+
+
+### Training
+
+The model is trained with standard autoregressive cross-entropy loss and using [SpecAugment](https://arxiv.org/abs/1904.08779).
+The encoder receives speech features, and the decoder generates the transcripts autoregressively.
+
+
+### BibTeX entry and citation info
+
+```bibtex
+@inproceedings{wang2020fairseqs2t,
+  title = {fairseq S2T: Fast Speech-to-Text Modeling with fairseq},
+  author = {Changhan Wang and Yun Tang and Xutai Ma and Anne Wu and Dmytro Okhonko and Juan Pino},
+  booktitle = {Proceedings of the 2020 Conference of the Asian Chapter of the Association for Computational Linguistics (AACL): System Demonstrations},
+  year = {2020},
+}
+
+```

config.json

@@ -0,0 +1,46 @@
+{
+  "_name_or_path": "hf_models_fb/s2t-small-librispeech-asr",
+  "activation_dropout": 0.1,
+  "activation_function": "relu",
+  "architectures": [
+    "Speech2TextForConditionalGeneration"
+  ],
+  "attention_dropout": 0.1,
+  "bos_token_id": 0,
+  "classifier_dropout": 0.0,
+  "conv_channels": 1024,
+  "conv_kernel_sizes": [
+    5,
+    5
+  ],
+  "d_model": 256,
+  "decoder_attention_heads": 4,
+  "decoder_ffn_dim": 2048,
+  "decoder_layerdrop": 0.0,
+  "decoder_layers": 6,
+  "decoder_start_token_id": 2,
+  "dropout": 0.1,
+  "early_stopping": true,
+  "encoder_attention_heads": 4,
+  "encoder_ffn_dim": 2048,
+  "encoder_layerdrop": 0.0,
+  "encoder_layers": 12,
+  "eos_token_id": 2,
+  "gradient_checkpointing": false,
+  "init_std": 0.02,
+  "input_channels": 1,
+  "input_feat_per_channel": 80,
+  "is_encoder_decoder": true,
+  "max_length": 200,
+  "max_source_positions": 6000,
+  "max_target_positions": 1024,
+  "model_type": "speech_to_text",
+  "num_beams": 5,
+  "num_conv_layers": 2,
+  "num_hidden_layers": 12,
+  "pad_token_id": 1,
+  "scale_embedding": true,
+  "transformers_version": "4.4.0.dev0",
+  "use_cache": true,
+  "vocab_size": 10000
+}

generation_config.json

@@ -0,0 +1,11 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 0,
+  "decoder_start_token_id": 2,
+  "early_stopping": true,
+  "eos_token_id": 2,
+  "max_length": 200,
+  "num_beams": 5,
+  "pad_token_id": 1,
+  "transformers_version": "4.27.0.dev0"
+}

gitattributes

@@ -0,0 +1,17 @@
+*.bin.* filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tar.gz filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+model.safetensors filter=lfs diff=lfs merge=lfs -text

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7d2b5fd0d9072cf00d3599363653a91f725d24e50a6b9ece8e4cb0837ba1969f
+size 118185584

preprocessor_config.json

@@ -0,0 +1,11 @@
+{
+  "do_ceptral_normalize": true,
+  "feature_size": 80,
+  "normalize_means": true,
+  "normalize_vars": true,
+  "num_mel_bins": 80,
+  "padding_side": "right",
+  "padding_value": 0.0,
+  "return_attention_mask": true,
+  "sampling_rate": 16000
+}

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:95be85b800e626fa6063bf30bd40874b3a426fc12b0393b7046546e470fcc535
+size 118267196

sentencepiece.bpe.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:052a168787a9160b4b2ba54e4995e9600298812c34191ca3f70cea51cd4f5c1e
+size 416684

special_tokens_map.json

@@ -0,0 +1 @@
+{"bos_token": "<s>", "eos_token": "</s>", "unk_token": "<unk>", "pad_token": "<pad>"}

tf_model.h5

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2faac8e98aaa3808196dab18955801120c7aab1c6d4d17ea788fefd1cd37aaa8
+size 128800472

tokenizer_config.json

@@ -0,0 +1 @@
+{"bos_token": "<s>", "eos_token": "</s>", "unk_token": "<unk>", "pad_token": "<pad>", "do_upper_case": false, "do_lower_case": true, "tgt_lang": null, "lang_codes": null, "special_tokens_map_file": "/home/suraj/.cache/huggingface/transformers/f39f1499e9c4d2b3e803e3cad8a31c4cf3e626e1c69197d4cd6921e5c07007f9.9d6cd81ef646692fb1c169a880161ea1cb95f49694f220aced9b704b457e51dd", "tokenizer_file": null, "name_or_path": "hf_models_fb/s2t-small-librispeech-asr"}