<jupyter_start><jupyter_text>Quantize Speech Recognition Models with OpenVINO Post-Training Optimization Tool ​This tutorial demonstrates how to apply INT8 quantization to the speech recognition model known as [Wav2Vec2](https://huggingface.co/docs/transformers/model_doc/wav2vec2), using the [Post-Training Optimization Tool API (POT API)](https://docs.openvino.ai/latest/pot_compression_api_README.html) (part of the [OpenVINO Toolkit](https://docs.openvino.ai/)). We will use a fine-tuned [Wav2Vec2-Base-960h](https://huggingface.co/facebook/wav2vec2-base-960h) [PyTorch](https://pytorch.org/) model trained on the [LibriSpeech ASR corpus](https://www.openslr.org/12). The tutorial is designed to be extendable to custom models and datasets. It consists of the following steps:- Download and prepare the Wav2Vec2 model and LibriSpeech dataset- Define data loading and accuracy validation functionality- Prepare the model for quantization- Run optimization pipeline- Compare performance of the original and quantized models  Imports<jupyter_code>import os
import sys
import time
import re
import numpy as np
import torch
import tarfile
from pathlib import Path
from itertools import groupby
import soundfile as sf
import IPython.display as ipd

from transformers import Wav2Vec2ForCTC
from openvino.runtime import Core
from openvino.tools.pot import Metric, DataLoader, IEEngine, \
    load_model, save_model, compress_model_weights, create_pipeline

sys.path.append("../utils")
from notebook_utils import download_file<jupyter_output><empty_output><jupyter_text>Settings<jupyter_code># Set the data and model directories, model source URL and model filename
DATA_DIR = "data"
MODEL_DIR = "model"

os.makedirs(DATA_DIR, exist_ok=True)
os.makedirs(MODEL_DIR, exist_ok=True)<jupyter_output><empty_output><jupyter_text>Prepare the ModelNext steps include:- Download and unpack pre-trained Wav2Vec2 model- Convert model to ONNX- Run OpenVINO Model Optimizer tool to convert the model from the ONNX representation to the OpenVINO Intermediate Representation (IR)<jupyter_code>download_file("https://huggingface.co/facebook/wav2vec2-base-960h/resolve/main/pytorch_model.bin", directory=Path(MODEL_DIR) / 'pytorch', show_progress=True)
download_file("https://huggingface.co/facebook/wav2vec2-base-960h/resolve/main/config.json", directory=Path(MODEL_DIR) / 'pytorch', show_progress=False)<jupyter_output><empty_output><jupyter_text>Import all dependencies to load the original PyTorch model and convert it to the ONNX representation.<jupyter_code>BATCH_SIZE = 1
MAX_SEQ_LENGTH = 30480


def export_model_to_onnx(model, path):
    with torch.no_grad():
        default_input = torch.zeros([1, MAX_SEQ_LENGTH], dtype=torch.float)
        inputs = {
            "inputs": default_input
        }
        symbolic_names = {0: "batch_size", 1: "sequence_len"}
        torch.onnx.export(
            model,
            (inputs["inputs"]),
            path,
            opset_version=11,
            input_names=["inputs"],
            output_names=["logits"],
            dynamic_axes={
                "inputs": symbolic_names,
                "logits": symbolic_names,
            },
        )
        print("ONNX model saved to {}".format(path))


torch_model = Wav2Vec2ForCTC.from_pretrained(Path(MODEL_DIR) / 'pytorch')
onnx_model_path = Path(MODEL_DIR) / "wav2vec2_base.onnx"
if not onnx_model_path.exists():
    export_model_to_onnx(torch_model, onnx_model_path)<jupyter_output><empty_output><jupyter_text>Convert the ONNX Model to OpenVINO IR<jupyter_code>ir_model_xml = onnx_model_path.with_suffix(".xml")
ir_model_bin = onnx_model_path.with_suffix(".bin")

if not ir_model_xml.exists():
    !mo --input_model $onnx_model_path --output_dir $MODEL_DIR --input_shape [1,-1] --data_type FP16<jupyter_output><empty_output><jupyter_text>Prepare LibriSpeech DatasetThe code below will download and unpack the archive with 'test-clean' subset of LibriSpeech Dataset<jupyter_code>download_file("http://openslr.elda.org/resources/12/test-clean.tar.gz", directory=DATA_DIR, show_progress=True)

if not os.path.exists(f'{DATA_DIR}/LibriSpeech'):
    with tarfile.open(f"{DATA_DIR}/test-clean.tar.gz") as tar:
        tar.extractall(path=DATA_DIR)<jupyter_output><empty_output><jupyter_text>Define DataLoader for POTIn this step, we need to define `DataLoader` based on POT API. It will be used to collect statistics for quantization and run model evaluation.Wav2Vec2 model accepts a raw waveform of the speech signal as input and produces vocabulary class estimations as output. Since dataset containsaudio files in flac-format, we use 'soundfile' package to convert them to waveform. > NOTE: Consider increase `samples_limit` to get more precise results, suggested value is `300` or more, it will take longer time to process.<jupyter_code>class LibriSpeechDataLoader(DataLoader):
    samples_limit = 50

    @staticmethod
    def read_flac(file_name):
        speech, samplerate = sf.read(file_name)
        assert samplerate == 16000, "read_flac: only 16kHz supported!"
        return speech

    # Required methods
    def __init__(self, config):
        """Constructor
        :param config: data loader specific config
        """
        super().__init__(config)
        self._data_dir = config["data_source"]
        self._ds = []
        self._prepare_dataset()

    def __len__(self):
        """Returns size of the dataset"""
        return len(self._ds)

    def __getitem__(self, index):
        """
        Returns annotation, data and metadata at the specified index.
        Possible formats:
        (index, annotation), data
        (index, annotation), data, metadata
        """
        label = self._ds[index][0]
        inputs = {'inputs': np.expand_dims(self._ds[index][1], axis=0)}
        return label, inputs

    # Methods specific to the current implementation
    def _prepare_dataset(self):
        pattern = re.compile(r'([0-9\-]+)\s+(.+)')
        data_folder = Path(self._data_dir)
        txts = list(data_folder.glob('**/*.txt'))
        counter = 0
        for txt in txts:
            content = txt.open().readlines()
            for line in content:
                res = pattern.search(line)
                if not res:
                    continue
                name = res.group(1)
                transcript = res.group(2)
                fname = txt.parent / name
                fname = fname.with_suffix('.flac')
                identifier = str(fname.relative_to(data_folder))
                self._ds.append(((counter, transcript.upper()), LibriSpeechDataLoader.read_flac(os.path.join(self._data_dir, identifier))))
                counter += 1
                if counter >= self.samples_limit:
                    # Limit exceeded
                    return<jupyter_output><empty_output><jupyter_text>Define WER Metric CalculationAt this step the `Metric` interface for WER metric is implemented. It is used for validating the accuracy of the model. `WER` stands for Word Error Rate, you can find more details on the [Wiki page](https://en.wikipedia.org/wiki/Word_error_rate).<jupyter_code>class MetricWER(Metric):
    alphabet = [
        "<pad>", "<s>", "</s>", "<unk>", "|",
        "e", "t", "a", "o", "n", "i", "h", "s", "r", "d", "l", "u",
        "m", "w", "c", "f", "g", "y", "p", "b", "v", "k", "'", "x", "j", "q", "z"]
    words_delimiter = '|'
    pad_token = '<pad>'

    @staticmethod
    def decode_logits(logits):
        decoding_vocab = dict(enumerate(MetricWER.alphabet))
        token_ids = np.squeeze(np.argmax(logits, -1))
        tokens = [decoding_vocab[idx] for idx in token_ids]
        tokens = [token_group[0] for token_group in groupby(tokens)]
        tokens = [t for t in tokens if t != MetricWER.pad_token]
        res_string = ''.join([t if t != MetricWER.words_delimiter else ' ' for t in tokens]).strip()
        res_string = ' '.join(res_string.split(' '))
        res_string = res_string.lower()
        return res_string

    # Required methods
    def __init__(self):
        super().__init__()
        self._name = "WER"
        self._sum_score = 0
        self._sum_words = 0
        self._cur_score = 0
        self._decoding_vocab = dict(enumerate(self.alphabet))

    @property
    def value(self):
        """Returns accuracy metric value for the last model output."""
        return {self._name: self._cur_score}

    @property
    def avg_value(self):
        """Returns accuracy metric value for all model outputs."""
        return {self._name: self._sum_score / self._sum_words if self._sum_words != 0 else 0}

    def update(self, output, target):
        """
        Updates prediction matches.

        :param output: model output
        :param target: annotations
        """
        decoded = [self.decode_logits(i) for i in output]
        target = [i.lower() for i in target]
        assert len(output) == len(target), "sizes of output and target mismatch!"
        for i in range(len(output)):
            self._get_metric_per_sample(decoded[i], target[i])

    def reset(self):
        """
        Resets collected matches
        """
        self._sum_score = 0
        self._sum_words = 0

    def get_attributes(self):
        """
        Returns a dictionary of metric attributes {metric_name: {attribute_name: value}}.
        Required attributes: 'direction': 'higher-better' or 'higher-worse'
                             'type': metric type
        """
        return {self._name: {"direction": "higher-worse", "type": "WER"}}

    # Methods specific to the current implementation
    def _get_metric_per_sample(self, annotation, prediction):
        cur_score = self._editdistance_eval(annotation.split(), prediction.split())
        cur_words = len(annotation.split())

        self._sum_score += cur_score
        self._sum_words += cur_words
        self._cur_score = cur_score / cur_words

        result = cur_score / cur_words if cur_words != 0 else 0
        return result

    def _editdistance_eval(self, source, target):
        n, m = len(source), len(target)

        distance = np.zeros((n + 1, m + 1), dtype=int)
        distance[:, 0] = np.arange(0, n + 1)
        distance[0, :] = np.arange(0, m + 1)

        for i in range(1, n + 1):
            for j in range(1, m + 1):
                cost = 0 if source[i - 1] == target[j - 1] else 1

                distance[i][j] = min(distance[i - 1][j] + 1,
                                     distance[i][j - 1] + 1,
                                     distance[i - 1][j - 1] + cost)
        return distance[n][m]<jupyter_output><empty_output><jupyter_text>Run Quantization PipelineHere we define a configuration for our quantization pipeline and run it. Please note that we use built-in `IEEngine` implementation of `Engine` interface from the POT API for model inference.<jupyter_code>model_config = {"model_name": "wav2vec2_base", "model": ir_model_xml, "weights": ir_model_bin}

engine_config = {"device": "CPU"}

dataset_config = {"data_source": os.path.join(DATA_DIR, "LibriSpeech/test-clean")}

algorithms = [
    {
        "name": "DefaultQuantization",
        "params": {
            "target_device": "ANY",
            "model_type": "transformer",
            "preset": "performance",
            "stat_subset_size": 300,
            "activations": {
                "range_estimator": {
                    "min": {
                        "aggregator": "min",
                        "type": "min"
                    },
                    "max": {
                        "aggregator": "mean",
                        "type": "quantile",
                        "outlier_prob": 0.0001
                    }
                }
            },
            "ignored": {
                "scope": ["214"]
            }
        }
    }
]

# Step 1: Load the model
model = load_model(model_config=model_config)

# Step 2: Initialize the data loader
data_loader = LibriSpeechDataLoader(config=dataset_config)

# Step 3 (Optional. Required for AccuracyAwareQuantization): Initialize the metric
metric = MetricWER()

# Step 4: Initialize the engine for metric calculation and statistics collection
engine = IEEngine(config=engine_config, data_loader=data_loader, metric=metric)

# Step 5: Create a pipeline of compression algorithms
pipeline = create_pipeline(algo_config=algorithms, engine=engine)

# Step 6 (Optional): Evaluate the original model. Print the results
start_time = time.perf_counter()
fp_results = pipeline.evaluate(model=model)
end_time = time.perf_counter()
print(f"Evaluation finished in {end_time - start_time:.2f} seconds")
if fp_results:
    print("FP16 model results:")
    for name, value in fp_results.items():
        print(f"{name}: {value:.5f}")
# Step 7: Execute the pipeline
print(f"Quantizing model with {algorithms[0]['params']['preset']} preset and {algorithms[0]['name']}")
start_time = time.perf_counter()
compressed_model = pipeline.run(model=model)
end_time = time.perf_counter()
print(f"Quantization finished in {end_time - start_time:.2f} seconds")

# Step 8 (Optional): Compress model weights to quantized precision
#                    in order to reduce the size of final .bin file
compress_model_weights(model=compressed_model)

# Step 9: Save the compressed model to the desired path
compressed_model_paths = save_model(model=compressed_model, save_path=MODEL_DIR, model_name="quantized_wav2vec2_base")
compressed_model_xml = compressed_model_paths[0]["model"]
# Step 10 (Optional): Evaluate the compressed model and print the results
int_results = pipeline.evaluate(model=compressed_model)

if int_results:
    print("INT8 model results:")
    for name, value in int_results.items():
        print(f"{name}: {value:.5f}")<jupyter_output><empty_output><jupyter_text>Model Usage Example with Inference PipelineBoth initial (FP16) and quntized (INT8) models are exactly the same in the useWe start with taking one example from the dataset to show inference steps for it<jupyter_code>audio = LibriSpeechDataLoader.read_flac(f'{DATA_DIR}/LibriSpeech/test-clean/121/127105/121-127105-0017.flac')

ipd.Audio(audio, rate=16000)<jupyter_output><empty_output><jupyter_text>Next, we load quantized model to the inference pipeline<jupyter_code>ie = Core()

model = ie.read_model(compressed_model_xml)
compiled_model = ie.compile_model(model=model, device_name='CPU')

input_data = np.expand_dims(audio, axis=0)
output_layer = compiled_model.outputs[0]<jupyter_output><empty_output><jupyter_text>Time to make a prediction<jupyter_code>predictions = compiled_model([input_data])[output_layer]<jupyter_output><empty_output><jupyter_text>Now we just need to decode predicted probabilites to text using tokenizer from MetricWER classAlternatively, you can use built-in tokenizer Wav2Vec2Processor from `transformers` package<jupyter_code>predicted_text = MetricWER.decode_logits(predictions)

predicted_text<jupyter_output><empty_output><jupyter_text>Compare Performance of the Original and Quantized ModelsFinally, we will measure the inference performance of the FP16 and INT8 models. To do this, we use [Benchmark Tool](https://docs.openvino.ai/latest/openvino_inference_engine_tools_benchmark_tool_README.html) - OpenVINO's inference performance measurement tool.> NOTE: For more accurate performance, we recommended running `benchmark_app` in a terminal/command prompt after closing other applications. Run `benchmark_app -m model.xml -d CPU` to benchmark async inference on CPU for one minute. Change `CPU` to `GPU` to benchmark on GPU. Run `benchmark_app --help` to see an overview of all command line options.<jupyter_code># Inference FP16 model (IR)
! benchmark_app -m $ir_model_xml -shape [1,30480] -d CPU -api async
# Inference INT8 model (IR)
! benchmark_app -m $compressed_model_xml -shape [1,30480] -d CPU -api async<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10746042757141956, 'fraction_numerical': 0.012486598978369174, 'mean_word_length': 3.3470394736842106, 'pattern_counts': {'":': 27, '<': 62, '<?xml version=': 0, '>': 65, 'https://': 10, 'lorem ipsum': 0, 'www.': 1, 'xml': 9}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 11, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '564209', 'license': 'Apache-2.0', 'n_tokens_mistral': 4977, 'n_tokens_neox': 4757, 'n_words': 1308, 'path': 'notebooks/107-speech-recognition-quantization/107-speech-recognition-quantization.ipynb', 'repo_name': 'yoyowz/openvino_notebooks', 'size': '15857'}