codingutf8 2021 the huggingface inc team licensed under the apache license version 2 0 the license you may not use this file except in compliance with the license you may obtain a copy of the license at http www apache orglicenseslicense2 0 unless required by applicable law or agreed to in writing software distributed under the license is distributed on an as is basis without warranties or conditions of any kind either express or implied see the license for the specific language governing permissions and limitations under the license speech processor class for wav2vec2 output type of wav2vec2decoderwithlm with transcription args text list of str or str decoded logits in text from usually the speech transcription logitscore list of float or float total logit score of the beams associated with produced text lmscore list of float fused lmscore of the beams associated with produced text wordoffsets list of listdictstr unionint str or listdictstr unionint str offsets of the decoded words in combination with sampling rate and model downsampling rate word offsets can be used to compute time stamps for each word featureextractorclass wav2vec2featureextractor tokenizerclass wav2vec2ctctokenizer def init self featureextractor featureextractionmixin tokenizer pretrainedtokenizerbase decoder beamsearchdecoderctc from pyctcdecode import beamsearchdecoderctc super initfeatureextractor tokenizer if not isinstancedecoder beamsearchdecoderctc raise valueerrorfdecoder has to be of type beamsearchdecoderctc class but is typedecoder make sure that decoder s alphabet and tokenizer s vocab match in content missingdecodertokens self getmissingalphabettokensdecoder tokenizer if lenmissingdecodertokens 0 raise valueerror fthe tokens missingdecodertokens are defined in the tokenizer s vocabulary but not in the decoder s alphabet fmake sure to include missingdecodertokens in the decoder s alphabet self decoder decoder self currentprocessor self featureextractor self intargetcontextmanager false def savepretrainedself savedirectory super savepretrainedsavedirectory self decoder savetodirsavedirectory classmethod def frompretrainedcls pretrainedmodelnameorpath kwargs r instantiate a wav2vec2processorwithlm from a pretrained wav2vec2 processor tip this class method is simply calling wav2vec2featureextractor s featureextractionutils featureextractionmixin frompretrained wav2vec2ctctokenizer s tokenizationutilsbase pretrainedtokenizerbase frompretrained and pyctcdecode beamsearchdecoderctc loadfromhfhub please refer to the docstrings of the methods above for more information tip args pretrainedmodelnameorpath str or os pathlike this can be either a string the model id of a pretrained featureextractor hosted inside a model repo on huggingface co valid model ids can be located at the rootlevel like bertbaseuncased or namespaced under a user or organization name like dbmdzbertbasegermancased a path to a directory containing a feature extractor file saved using the sequencefeatureextractor savepretrained method e g mymodeldirectory a path or url to a saved feature extractor json file e g mymodeldirectorypreprocessorconfig json kwargs additional keyword arguments passed along to both sequencefeatureextractor and pretrainedtokenizer beamsearchdecoderctc has no auto class snapshotdownload has no trustremotecode flag make sure that only relevant filenames are downloaded set language model attributes make sure that decoder s alphabet and tokenizer s vocab match in content we need to make sure that all of the tokenizer s except the special tokens are present in the decoder s alphabet retrieve missing alphabet token from decoder replace special tokens are any of the extra tokens no special tokenizer tokens when used in normal mode this method forwards all its arguments to wav2vec2featureextractor s wav2vec2featureextractor call and returns its output if used in the context wav2vec2processorwithlm astargetprocessor this method forwards all its arguments to wav2vec2ctctokenizer s wav2vec2ctctokenizer call please refer to the docstring of the above two methods for more information for backward compatibility when used in normal mode this method forwards all its arguments to wav2vec2featureextractor s wav2vec2featureextractor pad and returns its output if used in the context wav2vec2processorwithlm astargetprocessor this method forwards all its arguments to wav2vec2ctctokenizer s wav2vec2ctctokenizer pad please refer to the docstring of the above two methods for more information for backward compatibility batch decode output logits to audio transcription with language model support tip this function makes use of python s multiprocessing currently multiprocessing is available only on unix systems see this issuehttps github comkenshotechnologiespyctcdecodeissues65 if you are decoding multiple batches consider creating a pool and passing it to batchdecode otherwise batchdecode will be very slow since it will create a fresh pool for each call see usage example below tip args logits np ndarray the logits output vector of the model representing the log probabilities for each token pool multiprocessing pool optional an optional usermanaged pool if not set one will be automatically created and closed the pool should be instantiated after wav2vec2processorwithlm otherwise the lm won t be available to the pool s subprocesses tip currently only pools created with a fork context can be used if a spawn pool is passed it will be ignored and sequential decoding will be used instead tip numprocesses int optional if pool is not set number of processes on which the function should be parallelized over defaults to the number of available cpus beamwidth int optional maximum number of beams at each step in decoding defaults to pyctcdecode s defaultbeamwidth beamprunelogp int optional beams that are much worse than best beam will be pruned defaults to pyctcdecode s defaultprunelogp tokenminlogp int optional tokens below this logp are skipped unless they are argmax of frame defaults to pyctcdecode s defaultmintokenlogp hotwords liststr optional list of words with extra importance can be oov for lm hotwordweight int optional weight factor for hotword importance defaults to pyctcdecode s defaulthotwordweight alpha float optional weight for language model during shallow fusion beta float optional weight for length score adjustment of during scoring unkscoreoffset float optional amount of log score offset for unknown tokens lmscoreboundary bool optional whether to have kenlm respect boundaries when scoring outputwordoffsets bool optional defaults to false whether or not to output word offsets word offsets can be used in combination with the sampling rate and model downsampling rate to compute the timestamps of transcribed words nbest int optional defaults to 1 number of best hypotheses to return if nbest is greater than 1 the returned text will be a list of lists of strings logitscore will be a list of lists of floats and lmscore will be a list of lists of floats where the length of the outer list will correspond to the batch size and the length of the inner list will correspond to the number of returned hypotheses the value should be 1 tip please take a look at the example of wav2vec2processorwithlm decode to better understand how to make use of outputwordoffsets wav2vec2processorwithlm batchdecode works the same way with batched output tip returns models wav2vec2 wav2vec2decoderwithlmoutput example see decoding multiple audiosdecodingmultipleaudios set defaults reset params at every forward call it s just a set method in pyctcdecode create multiprocessing pool and list numpy arrays filter out logits padding create a pool if necessary while also using it as a context manager to close itself fork is safe to use only on unix see contexts and start methods section on multiprocessing s docs https docs python org3librarymultiprocessing htmlcontextsandstartmethods pool is managed by the user so we don t need to close it pyctcdecode extract text and scores wordoffsets appendword t0 startoffset t10 endoffset t11 for t in d01 decode output logits to audio transcription with language model support args logits np ndarray the logits output vector of the model representing the log probabilities for each token beamwidth int optional maximum number of beams at each step in decoding defaults to pyctcdecode s defaultbeamwidth beamprunelogp int optional a threshold to prune beams with logprobs less than bestbeamlogp beamprunelogp the value should be 0 defaults to pyctcdecode s defaultprunelogp tokenminlogp int optional tokens with logprobs below tokenminlogp are skipped unless they are have the maximum logprob for an utterance defaults to pyctcdecode s defaultmintokenlogp hotwords liststr optional list of words with extra importance which can be missing from the lm s vocabulary e g huggingface hotwordweight int optional weight multiplier that boosts hotword scores defaults to pyctcdecode s defaulthotwordweight alpha float optional weight for language model during shallow fusion beta float optional weight for length score adjustment of during scoring unkscoreoffset float optional amount of log score offset for unknown tokens lmscoreboundary bool optional whether to have kenlm respect boundaries when scoring outputwordoffsets bool optional defaults to false whether or not to output word offsets word offsets can be used in combination with the sampling rate and model downsampling rate to compute the timestamps of transcribed words nbest int optional defaults to 1 number of best hypotheses to return if nbest is greater than 1 the returned text will be a list of strings logitscore will be a list of floats and lmscore will be a list of floats where the length of these lists will correspond to the number of returned hypotheses the value should be 1 tip please take a look at the example below to better understand how to make use of outputwordoffsets tip returns models wav2vec2 wav2vec2decoderwithlmoutput example python let s see how to retrieve time steps for a model from transformers import autotokenizer autoprocessor automodelforctc from datasets import loaddataset import datasets import torch import model feature extractor tokenizer model automodelforctc frompretrainedpatrickvonplatenwav2vec2base100hwithlm processor autoprocessor frompretrainedpatrickvonplatenwav2vec2base100hwithlm load first sample of english commonvoice dataset loaddatasetmozillafoundationcommonvoice110 en splittrain streamingtrue dataset dataset castcolumnaudio datasets audiosamplingrate16000 datasetiter iterdataset sample nextdatasetiter forward sample through model to get greedily predicted transcription ids inputvalues processorsampleaudioarray returntensorspt inputvalues with torch nograd logits modelinputvalues logits0 cpu numpy retrieve word stamps analogous commands for outputcharoffsets outputs processor decodelogits outputwordoffsetstrue compute timeoffset in seconds as product of downsampling ratio and samplingrate timeoffset model config inputstologitsratio processor featureextractor samplingrate wordoffsets word dword starttime rounddstartoffset timeoffset 2 endtime rounddendoffset timeoffset 2 for d in outputs wordoffsets compare word offsets with audio entrain0commonvoiceen19121553 mp3 online on the dataset viewer https huggingface codatasetsmozillafoundationcommonvoice110vieweren wordoffsets 4 word the starttime 0 68 endtime 0 78 word track starttime 0 88 endtime 1 1 word appears starttime 1 18 endtime 1 66 word on starttime 1 86 endtime 1 92 from pyctcdecode constants import defaultbeamwidth defaulthotwordweight defaultmintokenlogp defaultprunelogp set defaults beamwidth beamwidth if beamwidth is not none else defaultbeamwidth beamprunelogp beamprunelogp if beamprunelogp is not none else defaultprunelogp tokenminlogp tokenminlogp if tokenminlogp is not none else defaultmintokenlogp hotwordweight hotwordweight if hotwordweight is not none else defaulthotwordweight reset params at every forward call it s just a set method in pyctcdecode self decoder resetparams alphaalpha betabeta unkscoreoffsetunkscoreoffset lmscoreboundarylmscoreboundary pyctcdecode decodedbeams self decoder decodebeams logits beamwidthbeamwidth beamprunelogpbeamprunelogp tokenminlogptokenminlogp hotwordshotwords hotwordweighthotwordweight wordoffsets none if outputwordoffsets wordoffsets word word startoffset startoffset endoffset endoffset for word startoffset endoffset in beam2 for beam in decodedbeams logitscores beam2 for beam in decodedbeams lmscores beam1 for beam in decodedbeams hypotheses beam0 for beam in decodedbeams if nbest lendecodedbeams logger info nbest size is larger than the number of generated hypotheses all hypotheses will be returned if nbest 1 return wav2vec2decoderwithlmoutput texthypotheses0 logitscorelogitscores0 lmscorelmscores0 wordoffsetswordoffsets0 if wordoffsets is not none else none else return wav2vec2decoderwithlmoutput texthypotheses nbest logitscorelogitscores nbest lmscorelmscores nbest wordoffsetswordoffsets nbest if wordoffsets is not none else none contextmanager def astargetprocessorself warnings warn astargetprocessor is deprecated and will be removed in v5 of transformers you can process your labels by using the argument text of the regular call method either in the same call as your audio inputs or in a separate call self intargetcontextmanager true self currentprocessor self tokenizer yield self currentprocessor self featureextractor self intargetcontextmanager false coding utf 8 2021 the huggingface inc team licensed under the apache license version 2 0 the license you may not use this file except in compliance with the license you may obtain a copy of the license at http www apache org licenses license 2 0 unless required by applicable law or agreed to in writing software distributed under the license is distributed on an as is basis without warranties or conditions of any kind either express or implied see the license for the specific language governing permissions and limitations under the license speech processor class for wav2vec2 output type of wav2vec2decoderwithlm with transcription args text list of str or str decoded logits in text from usually the speech transcription logit_score list of float or float total logit score of the beams associated with produced text lm_score list of float fused lm_score of the beams associated with produced text word_offsets list of list dict str union int str or list dict str union int str offsets of the decoded words in combination with sampling rate and model downsampling rate word offsets can be used to compute time stamps for each word constructs a wav2vec2 processor which wraps a wav2vec2 feature extractor a wav2vec2 ctc tokenizer and a decoder with language model support into a single processor for language model boosted speech recognition decoding args feature_extractor wav2vec2featureextractor an instance of wav2vec2featureextractor the feature extractor is a required input tokenizer wav2vec2ctctokenizer an instance of wav2vec2ctctokenizer the tokenizer is a required input decoder pyctcdecode beamsearchdecoderctc an instance of pyctcdecode beamsearchdecoderctc the decoder is a required input make sure that decoder s alphabet and tokenizer s vocab match in content instantiate a wav2vec2processorwithlm from a pretrained wav2vec2 processor tip this class method is simply calling wav2vec2featureextractor s feature_extraction_utils featureextractionmixin from_pretrained wav2vec2ctctokenizer s tokenization_utils_base pretrainedtokenizerbase from_pretrained and pyctcdecode beamsearchdecoderctc load_from_hf_hub please refer to the docstrings of the methods above for more information tip args pretrained_model_name_or_path str or os pathlike this can be either a string the model id of a pretrained feature_extractor hosted inside a model repo on huggingface co valid model ids can be located at the root level like bert base uncased or namespaced under a user or organization name like dbmdz bert base german cased a path to a directory containing a feature extractor file saved using the sequencefeatureextractor save_pretrained method e g my_model_directory a path or url to a saved feature extractor json file e g my_model_directory preprocessor_config json kwargs additional keyword arguments passed along to both sequencefeatureextractor and pretrainedtokenizer beamsearchdecoderctc has no auto class snapshot_download has no trust_remote_code flag make sure that only relevant filenames are downloaded set language model attributes make sure that decoder s alphabet and tokenizer s vocab match in content we need to make sure that all of the tokenizer s except the special tokens are present in the decoder s alphabet retrieve missing alphabet token from decoder replace special tokens are any of the extra tokens no special tokenizer tokens when used in normal mode this method forwards all its arguments to wav2vec2featureextractor s wav2vec2featureextractor __call__ and returns its output if used in the context wav2vec2processorwithlm as_target_processor this method forwards all its arguments to wav2vec2ctctokenizer s wav2vec2ctctokenizer __call__ please refer to the docstring of the above two methods for more information for backward compatibility when used in normal mode this method forwards all its arguments to wav2vec2featureextractor s wav2vec2featureextractor pad and returns its output if used in the context wav2vec2processorwithlm as_target_processor this method forwards all its arguments to wav2vec2ctctokenizer s wav2vec2ctctokenizer pad please refer to the docstring of the above two methods for more information for backward compatibility batch decode output logits to audio transcription with language model support tip this function makes use of python s multiprocessing currently multiprocessing is available only on unix systems see this issue https github com kensho technologies pyctcdecode issues 65 if you are decoding multiple batches consider creating a pool and passing it to batch_decode otherwise batch_decode will be very slow since it will create a fresh pool for each call see usage example below tip args logits np ndarray the logits output vector of the model representing the log probabilities for each token pool multiprocessing pool optional an optional user managed pool if not set one will be automatically created and closed the pool should be instantiated after wav2vec2processorwithlm otherwise the lm won t be available to the pool s sub processes tip currently only pools created with a fork context can be used if a spawn pool is passed it will be ignored and sequential decoding will be used instead tip num_processes int optional if pool is not set number of processes on which the function should be parallelized over defaults to the number of available cpus beam_width int optional maximum number of beams at each step in decoding defaults to pyctcdecode s default_beam_width beam_prune_logp int optional beams that are much worse than best beam will be pruned defaults to pyctcdecode s default_prune_logp token_min_logp int optional tokens below this logp are skipped unless they are argmax of frame defaults to pyctcdecode s default_min_token_logp hotwords list str optional list of words with extra importance can be oov for lm hotword_weight int optional weight factor for hotword importance defaults to pyctcdecode s default_hotword_weight alpha float optional weight for language model during shallow fusion beta float optional weight for length score adjustment of during scoring unk_score_offset float optional amount of log score offset for unknown tokens lm_score_boundary bool optional whether to have kenlm respect boundaries when scoring output_word_offsets bool optional defaults to false whether or not to output word offsets word offsets can be used in combination with the sampling rate and model downsampling rate to compute the time stamps of transcribed words n_best int optional defaults to 1 number of best hypotheses to return if n_best is greater than 1 the returned text will be a list of lists of strings logit_score will be a list of lists of floats and lm_score will be a list of lists of floats where the length of the outer list will correspond to the batch size and the length of the inner list will correspond to the number of returned hypotheses the value should be 1 tip please take a look at the example of wav2vec2processorwithlm decode to better understand how to make use of output_word_offsets wav2vec2processorwithlm batch_decode works the same way with batched output tip returns models wav2vec2 wav2vec2decoderwithlmoutput example see decoding multiple audios decoding multiple audios set defaults reset params at every forward call it s just a set method in pyctcdecode create multiprocessing pool and list numpy arrays filter out logits padding create a pool if necessary while also using it as a context manager to close itself fork is safe to use only on unix see contexts and start methods section on multiprocessing s docs https docs python org 3 library multiprocessing html contexts and start methods pool is managed by the user so we don t need to close it pyctcdecode extract text and scores word_offsets append word t 0 start_offset t 1 0 end_offset t 1 1 for t in d 0 1 decode output logits to audio transcription with language model support args logits np ndarray the logits output vector of the model representing the log probabilities for each token beam_width int optional maximum number of beams at each step in decoding defaults to pyctcdecode s default_beam_width beam_prune_logp int optional a threshold to prune beams with log probs less than best_beam_logp beam_prune_logp the value should be 0 defaults to pyctcdecode s default_prune_logp token_min_logp int optional tokens with log probs below token_min_logp are skipped unless they are have the maximum log prob for an utterance defaults to pyctcdecode s default_min_token_logp hotwords list str optional list of words with extra importance which can be missing from the lm s vocabulary e g huggingface hotword_weight int optional weight multiplier that boosts hotword scores defaults to pyctcdecode s default_hotword_weight alpha float optional weight for language model during shallow fusion beta float optional weight for length score adjustment of during scoring unk_score_offset float optional amount of log score offset for unknown tokens lm_score_boundary bool optional whether to have kenlm respect boundaries when scoring output_word_offsets bool optional defaults to false whether or not to output word offsets word offsets can be used in combination with the sampling rate and model downsampling rate to compute the time stamps of transcribed words n_best int optional defaults to 1 number of best hypotheses to return if n_best is greater than 1 the returned text will be a list of strings logit_score will be a list of floats and lm_score will be a list of floats where the length of these lists will correspond to the number of returned hypotheses the value should be 1 tip please take a look at the example below to better understand how to make use of output_word_offsets tip returns models wav2vec2 wav2vec2decoderwithlmoutput example python let s see how to retrieve time steps for a model from transformers import autotokenizer autoprocessor automodelforctc from datasets import load_dataset import datasets import torch import model feature extractor tokenizer model automodelforctc from_pretrained patrickvonplaten wav2vec2 base 100h with lm processor autoprocessor from_pretrained patrickvonplaten wav2vec2 base 100h with lm load first sample of english common_voice dataset load_dataset mozilla foundation common_voice_11_0 en split train streaming true dataset dataset cast_column audio datasets audio sampling_rate 16_000 dataset_iter iter dataset sample next dataset_iter forward sample through model to get greedily predicted transcription ids input_values processor sample audio array return_tensors pt input_values with torch no_grad logits model input_values logits 0 cpu numpy retrieve word stamps analogous commands for output_char_offsets outputs processor decode logits output_word_offsets true compute time_offset in seconds as product of downsampling ratio and sampling_rate time_offset model config inputs_to_logits_ratio processor feature_extractor sampling_rate word_offsets word d word start_time round d start_offset time_offset 2 end_time round d end_offset time_offset 2 for d in outputs word_offsets compare word offsets with audio en_train_0 common_voice_en_19121553 mp3 online on the dataset viewer https huggingface co datasets mozilla foundation common_voice_11_0 viewer en word_offsets 4 word the start_time 0 68 end_time 0 78 word track start_time 0 88 end_time 1 1 word appears start_time 1 18 end_time 1 66 word on start_time 1 86 end_time 1 92 set defaults reset params at every forward call it s just a set method in pyctcdecode pyctcdecode temporarily sets the processor for processing the target useful for encoding the labels when fine tuning wav2vec2
import os
import warnings
from contextlib import contextmanager, nullcontext
from dataclasses import dataclass
from multiprocessing import Pool, get_context, get_start_method
from typing import TYPE_CHECKING, Dict, Iterable, List, Optional, Union

import numpy as np

from ...processing_utils import ProcessorMixin
from ...utils import ModelOutput, logging, requires_backends


logger = logging.get_logger(__name__)


if TYPE_CHECKING:
    from pyctcdecode import BeamSearchDecoderCTC

    from ...feature_extraction_utils import FeatureExtractionMixin
    from ...tokenization_utils import PreTrainedTokenizerBase


ListOfDict = List[Dict[str, Union[int, str]]]


@dataclass
class Wav2Vec2DecoderWithLMOutput(ModelOutput):
    

    text: Union[List[List[str]], List[str], str]
    logit_score: Union[List[List[float]], List[float], float] = None
    lm_score: Union[List[List[float]], List[float], float] = None
    word_offsets: Union[List[List[ListOfDict]], List[ListOfDict], ListOfDict] = None


class Wav2Vec2ProcessorWithLM(ProcessorMixin):
    r

    feature_extractor_class = "Wav2Vec2FeatureExtractor"
    tokenizer_class = "Wav2Vec2CTCTokenizer"

    def __init__(
        self,
        feature_extractor: "FeatureExtractionMixin",
        tokenizer: "PreTrainedTokenizerBase",
        decoder: "BeamSearchDecoderCTC",
    ):
        from pyctcdecode import BeamSearchDecoderCTC

        super().__init__(feature_extractor, tokenizer)
        if not isinstance(decoder, BeamSearchDecoderCTC):
            raise ValueError(f"`decoder` has to be of type {BeamSearchDecoderCTC.__class__}, but is {type(decoder)}")

        
        missing_decoder_tokens = self.get_missing_alphabet_tokens(decoder, tokenizer)
        if len(missing_decoder_tokens) > 0:
            raise ValueError(
                f"The tokens {missing_decoder_tokens} are defined in the tokenizer's "
                "vocabulary, but not in the decoder's alphabet. "
                f"Make sure to include {missing_decoder_tokens} in the decoder's alphabet."
            )

        self.decoder = decoder
        self.current_processor = self.feature_extractor
        self._in_target_context_manager = False

    def save_pretrained(self, save_directory):
        super().save_pretrained(save_directory)
        self.decoder.save_to_dir(save_directory)

    @classmethod
    def from_pretrained(cls, pretrained_model_name_or_path, **kwargs):
        r
        requires_backends(cls, "pyctcdecode")
        from pyctcdecode import BeamSearchDecoderCTC

        feature_extractor, tokenizer = super()._get_arguments_from_pretrained(pretrained_model_name_or_path, **kwargs)

        if os.path.isdir(pretrained_model_name_or_path) or os.path.isfile(pretrained_model_name_or_path):
            decoder = BeamSearchDecoderCTC.load_from_dir(pretrained_model_name_or_path)
        else:
            
            kwargs.pop("_from_auto", None)
            
            kwargs.pop("trust_remote_code", None)

            
            language_model_filenames = os.path.join(BeamSearchDecoderCTC._LANGUAGE_MODEL_SERIALIZED_DIRECTORY, "*")
            alphabet_filename = BeamSearchDecoderCTC._ALPHABET_SERIALIZED_FILENAME
            allow_patterns = [language_model_filenames, alphabet_filename]

            decoder = BeamSearchDecoderCTC.load_from_hf_hub(
                pretrained_model_name_or_path, allow_patterns=allow_patterns, **kwargs
            )

        
        for attribute in ["alpha", "beta", "unk_score_offset", "score_boundary"]:
            value = kwargs.pop(attribute, None)

            if value is not None:
                cls._set_language_model_attribute(decoder, attribute, value)

        
        missing_decoder_tokens = cls.get_missing_alphabet_tokens(decoder, tokenizer)
        if len(missing_decoder_tokens) > 0:
            raise ValueError(
                f"The tokens {missing_decoder_tokens} are defined in the tokenizer's "
                "vocabulary, but not in the decoder's alphabet. "
                f"Make sure to include {missing_decoder_tokens} in the decoder's alphabet."
            )

        return cls(feature_extractor=feature_extractor, tokenizer=tokenizer, decoder=decoder)

    @staticmethod
    def _set_language_model_attribute(decoder: "BeamSearchDecoderCTC", attribute: str, value: float):
        setattr(decoder.model_container[decoder._model_key], attribute, value)

    @property
    def language_model(self):
        return self.decoder.model_container[self.decoder._model_key]

    @staticmethod
    def get_missing_alphabet_tokens(decoder, tokenizer):
        from pyctcdecode.alphabet import BLANK_TOKEN_PTN, UNK_TOKEN, UNK_TOKEN_PTN

        
        
        
        tokenizer_vocab_list = list(tokenizer.get_vocab().keys())

        
        for i, token in enumerate(tokenizer_vocab_list):
            if BLANK_TOKEN_PTN.match(token):
                tokenizer_vocab_list[i] = ""
            if token == tokenizer.word_delimiter_token:
                tokenizer_vocab_list[i] = " "
            if UNK_TOKEN_PTN.match(token):
                tokenizer_vocab_list[i] = UNK_TOKEN

        
        missing_tokens = set(tokenizer_vocab_list) - set(decoder._alphabet.labels)

        return missing_tokens

    def __call__(self, *args, **kwargs):
        
        
        if self._in_target_context_manager:
            return self.current_processor(*args, **kwargs)

        if "raw_speech" in kwargs:
            warnings.warn("Using `raw_speech` as a keyword argument is deprecated. Use `audio` instead.")
            audio = kwargs.pop("raw_speech")
        else:
            audio = kwargs.pop("audio", None)
        sampling_rate = kwargs.pop("sampling_rate", None)
        text = kwargs.pop("text", None)
        if len(args) > 0:
            audio = args[0]
            args = args[1:]

        if audio is None and text is None:
            raise ValueError("You need to specify either an `audio` or `text` input to process.")

        if audio is not None:
            inputs = self.feature_extractor(audio, *args, sampling_rate=sampling_rate, **kwargs)
        if text is not None:
            encodings = self.tokenizer(text, **kwargs)

        if text is None:
            return inputs
        elif audio is None:
            return encodings
        else:
            inputs["labels"] = encodings["input_ids"]
            return inputs

    def pad(self, *args, **kwargs):
        
        
        if self._in_target_context_manager:
            return self.current_processor.pad(*args, **kwargs)

        input_features = kwargs.pop("input_features", None)
        labels = kwargs.pop("labels", None)
        if len(args) > 0:
            input_features = args[0]
            args = args[1:]

        if input_features is not None:
            input_features = self.feature_extractor.pad(input_features, *args, **kwargs)
        if labels is not None:
            labels = self.tokenizer.pad(labels, **kwargs)

        if labels is None:
            return input_features
        elif input_features is None:
            return labels
        else:
            input_features["labels"] = labels["input_ids"]
            return input_features

    def batch_decode(
        self,
        logits: np.ndarray,
        pool: Optional[Pool] = None,
        num_processes: Optional[int] = None,
        beam_width: Optional[int] = None,
        beam_prune_logp: Optional[float] = None,
        token_min_logp: Optional[float] = None,
        hotwords: Optional[Iterable[str]] = None,
        hotword_weight: Optional[float] = None,
        alpha: Optional[float] = None,
        beta: Optional[float] = None,
        unk_score_offset: Optional[float] = None,
        lm_score_boundary: Optional[bool] = None,
        output_word_offsets: bool = False,
        n_best: int = 1,
    ):
        

        from pyctcdecode.constants import (
            DEFAULT_BEAM_WIDTH,
            DEFAULT_HOTWORD_WEIGHT,
            DEFAULT_MIN_TOKEN_LOGP,
            DEFAULT_PRUNE_LOGP,
        )

        
        beam_width = beam_width if beam_width is not None else DEFAULT_BEAM_WIDTH
        beam_prune_logp = beam_prune_logp if beam_prune_logp is not None else DEFAULT_PRUNE_LOGP
        token_min_logp = token_min_logp if token_min_logp is not None else DEFAULT_MIN_TOKEN_LOGP
        hotword_weight = hotword_weight if hotword_weight is not None else DEFAULT_HOTWORD_WEIGHT

        
        self.decoder.reset_params(
            alpha=alpha, beta=beta, unk_score_offset=unk_score_offset, lm_score_boundary=lm_score_boundary
        )

        
        
        logits_list = [array[(array != -100.0).all(axis=-1)] for array in logits]

        
        if pool is None:
            
            
            default_context = get_start_method()

            if default_context == "fork":
                cm = pool = get_context().Pool(num_processes)
            else:
                logger.warning(
                    "Parallel batch decoding is not currently supported in this platform. "
                    "Falling back to sequential decoding."
                )
                cm = nullcontext()
        else:
            
            cm = nullcontext()

            if num_processes is not None:
                logger.warning(
                    "Parameter `num_process` was passed, but it will be ignored since `pool` was also specified."
                )

        
        with cm:
            decoded_beams = self.decoder.decode_beams_batch(
                pool=pool,
                logits_list=logits_list,
                beam_width=beam_width,
                beam_prune_logp=beam_prune_logp,
                token_min_logp=token_min_logp,
                hotwords=hotwords,
                hotword_weight=hotword_weight,
            )

        
        batch_texts, logit_scores, lm_scores, word_offsets = [], [], [], []

        for d in decoded_beams:
            batch_texts.append([beam[0] for beam in d])
            logit_scores.append([beam[-2] for beam in d])
            lm_scores.append([beam[-1] for beam in d])

            

            word_offsets.append(
                [
                    [
                        {"word": word, "start_offset": start_offset, "end_offset": end_offset}
                        for word, (start_offset, end_offset) in beam[1]
                    ]
                    for beam in d
                ]
            )

        word_offsets = word_offsets if output_word_offsets else None

        if n_best == 1:
            return Wav2Vec2DecoderWithLMOutput(
                text=[hyps[0] for hyps in batch_texts],
                logit_score=[hyps[0] for hyps in logit_scores],
                lm_score=[hyps[0] for hyps in lm_scores],
                word_offsets=[hyps[0] for hyps in word_offsets] if word_offsets is not None else None,
            )
        else:
            return Wav2Vec2DecoderWithLMOutput(
                text=[hyps[:n_best] for hyps in batch_texts],
                logit_score=[hyps[:n_best] for hyps in logit_scores],
                lm_score=[hyps[:n_best] for hyps in lm_scores],
                word_offsets=[hyps[:n_best] for hyps in word_offsets] if word_offsets is not None else None,
            )

    def decode(
        self,
        logits: np.ndarray,
        beam_width: Optional[int] = None,
        beam_prune_logp: Optional[float] = None,
        token_min_logp: Optional[float] = None,
        hotwords: Optional[Iterable[str]] = None,
        hotword_weight: Optional[float] = None,
        alpha: Optional[float] = None,
        beta: Optional[float] = None,
        unk_score_offset: Optional[float] = None,
        lm_score_boundary: Optional[bool] = None,
        output_word_offsets: bool = False,
        n_best: int = 1,
    ):
        

        from pyctcdecode.constants import (
            DEFAULT_BEAM_WIDTH,
            DEFAULT_HOTWORD_WEIGHT,
            DEFAULT_MIN_TOKEN_LOGP,
            DEFAULT_PRUNE_LOGP,
        )

        
        beam_width = beam_width if beam_width is not None else DEFAULT_BEAM_WIDTH
        beam_prune_logp = beam_prune_logp if beam_prune_logp is not None else DEFAULT_PRUNE_LOGP
        token_min_logp = token_min_logp if token_min_logp is not None else DEFAULT_MIN_TOKEN_LOGP
        hotword_weight = hotword_weight if hotword_weight is not None else DEFAULT_HOTWORD_WEIGHT

        
        self.decoder.reset_params(
            alpha=alpha, beta=beta, unk_score_offset=unk_score_offset, lm_score_boundary=lm_score_boundary
        )

        
        decoded_beams = self.decoder.decode_beams(
            logits,
            beam_width=beam_width,
            beam_prune_logp=beam_prune_logp,
            token_min_logp=token_min_logp,
            hotwords=hotwords,
            hotword_weight=hotword_weight,
        )

        word_offsets = None
        if output_word_offsets:
            word_offsets = [
                [
                    {"word": word, "start_offset": start_offset, "end_offset": end_offset}
                    for word, (start_offset, end_offset) in beam[2]
                ]
                for beam in decoded_beams
            ]
        logit_scores = [beam[-2] for beam in decoded_beams]

        lm_scores = [beam[-1] for beam in decoded_beams]

        hypotheses = [beam[0] for beam in decoded_beams]

        if n_best > len(decoded_beams):
            logger.info(
                "N-best size is larger than the number of generated hypotheses, all hypotheses will be returned."
            )

        if n_best == 1:
            return Wav2Vec2DecoderWithLMOutput(
                text=hypotheses[0],
                logit_score=logit_scores[0],
                lm_score=lm_scores[0],
                word_offsets=word_offsets[0] if word_offsets is not None else None,
            )
        else:
            return Wav2Vec2DecoderWithLMOutput(
                text=hypotheses[:n_best],
                logit_score=logit_scores[:n_best],
                lm_score=lm_scores[:n_best],
                word_offsets=word_offsets[:n_best] if word_offsets is not None else None,
            )

    @contextmanager
    def as_target_processor(self):
        
        warnings.warn(
            "`as_target_processor` is deprecated and will be removed in v5 of Transformers. You can process your "
            "labels by using the argument `text` of the regular `__call__` method (either in the same call as "
            "your audio inputs, or in a separate call."
        )
        self._in_target_context_manager = True
        self.current_processor = self.tokenizer
        yield
        self.current_processor = self.feature_extractor
        self._in_target_context_manager = False