primeline

/

whisper-large-v3-turbo-german

like 19

Follow

primeLine AI Services 10

Cool release and congrats on finetuning :)

Some remarks on the evaluation:

First of all thanks for sharing them. I could only verify this since you were so transparent about all of this. So thanks for that. Currently all Whisper Models except yours get punished for using ß vs. ss which is fairly common in german. Also punctuation is punished which is not always clear in spoken language especially. Just generally experimenting a bit with the evals they are highly dependent on which types of normalization one does include. With the commonly used Whisper normalizer evals look like this:

******************** common_voice_19_0 ********************
openai-whisper-large-v3-turbo: 4.08
openai-whisper-large-v3: 3.73
primeline-whisper-large-v3-german: 2.83
nyrahealth-CrisperWhisper: 2.07
primeline-whisper-large-v3-turbo-german: 3.40

******************** multilingual librispeech ********************
openai-whisper-large-v3-turbo: 3.23
openai-whisper-large-v3: 2.82
primeline-whisper-large-v3-german: 2.10
nyrahealth-CrisperWhisper: 2.64
primeline-whisper-large-v3-turbo-german: 2.09

******************** Tuda-De ********************
openai-whisper-large-v3-turbo: 8.69
openai-whisper-large-v3: 8.31
primeline-whisper-large-v3-german: 8.54
nyrahealth-CrisperWhisper: 5.04
primeline-whisper-large-v3-turbo-german: 6.70

******************** All ********************
openai-whisper-large-v3-turbo: 3.76
openai-whisper-large-v3: 3.37
primeline-whisper-large-v3-german: 2.64
nyrahealth-CrisperWhisper: 2.58
primeline-whisper-large-v3-turbo-german: 2.72

So feel free to run them yourself or comment in and out certain normalizations from my wer_standardize function and using wer_standardize in calculate_wer. Maybe you find a way to display a fairer evaluation or multiple ones that could reflect various use cases.
Commonvoice itself is problematic to run evaluations on anyway since test sets are not static between releases :)

import jiwer
from collections import defaultdict
import datasets
from datasets import load_dataset
from jiwer import wer, cer, wer_standardize_contiguous
import re
from whisper_normalizer.english import EnglishTextNormalizer
english_normalizer = EnglishTextNormalizer()
def wer_standardize2(text):
#    # This function should implement the standardization logic
#    # For this example, we'll use a simple lowercase transformation
    return english_normalizer(text)



def wer_standardize(text):
    # Convert to lowercase
    text = text.lower()
    
    # Replace ß with ss
    text = text.replace('ß', 'ss')
    
    # Remove words enclosed in []
    text = re.sub(r'\[.*?\]', '', text)

    #remove quotes
    text = re.sub(r'[\'"]', '', text)
    
    # Remove punctuation and hyphens
    text = re.sub(r'[^\w\s]', '', text)

    # Remove trailing spaces and reduce multiple spaces to single space
    text = re.sub(r'\s+', ' ', text.strip())
    
    return text

#change to standardize one and comment in and out different standardizations
def calculate_wer(references, hypotheses):
    references = [wer_standardize2(ref) for ref in references]
    hypotheses = [wer_standardize2(hyp) for hyp in hypotheses]
    return wer(references,
               hypotheses)

def process_dataset(dataset):
    models = [
        'openai-whisper-large-v3-turbo',
        'openai-whisper-large-v3',
        'primeline-whisper-large-v3-german',
        'nyrahealth-CrisperWhisper',
        'primeline-whisper-large-v3-turbo-german',
    ]

    set_names = ["All"]
    results = defaultdict(lambda: defaultdict(lambda: {'references': [], 'predictions': []}))

    for i in range(len(dataset)):
        item = dataset[i]
        dataset_source = str(item['from'])
        reference = item['references']
        set_names.append(dataset_source)

        for model in models:
            prediction = item.get(model, '')
            if prediction:  # Skip empty transcriptions
                results[dataset_source][model]['references'].append(reference)
                results[dataset_source][model]['predictions'].append(prediction)
                results["All"][model]['references'].append(reference)
                results["All"][model]['predictions'].append(prediction)
    return results, set(set_names)

def print_results(results, set_names):
    for dataset_source in set_names:
        print("\n\n")
        print("*" * 20, dataset_source, "*" * 20)
        for model in results[dataset_source]:
            refs = results[dataset_source][model]['references']
            preds = results[dataset_source][model]['predictions']
            if refs and preds:  # Check if lists are not empty
                error_rate = calculate_wer(refs, preds) * 100
                print(f"{model}: {error_rate:.2f}")
        print("*" * 50)

# Process the 'train' split of the dataset
# Load the dataset
ds = load_dataset("flozi00/asr-german-mixed-evals")
dataset = ds['train']
results, set_names = process_dataset(dataset)
print_results(results, set_names)
```