2023 the huggingface team licensed under the apache license version 2 0 the license you may not use this file except in compliance with the license you may obtain a copy of the license at http www apache orglicenseslicense2 0 unless required by applicable law or agreed to in writing software distributed under the license is distributed on an as is basis without warranties or conditions of any kind either express or implied see the license for the specific language governing permissions and limitations under the license from typing import list union texttoaudio generation pipeline using any automodelfortexttowaveform or automodelfortexttospectrogram this pipeline generates an audio file from an input text and optional other conditional inputs example python from transformers import pipeline pipe pipelinemodelsunobarksmall output pipehey it s huggingface on the phone audio outputaudio samplingrate outputsamplingrate learn more about the basics of using a pipeline in the pipeline tutorial pipelinetutorial tip you can specify parameters passed to the model by using texttoaudiopipeline call forwardparams or texttoaudiopipeline call generatekwargs example python from transformers import pipeline musicgenerator pipelinetasktexttoaudio modelfacebookmusicgensmall frameworkpt diversify the music generation by adding randomness with a high temperature and set a maximum music length generatekwargs dosample true temperature 0 7 maxnewtokens 35 outputs musicgeneratortechno music with high melodic riffs generatekwargsgeneratekwargs tip this pipeline can currently be loaded from pipeline using the following task identifiers texttospeech or texttoaudio see the list of available models on huggingface comodelshttps huggingface comodels filtertexttospeech get samplingrate from config and generation config bark tokenizer is called with barkprocessor which uses those kwargs priority is given to kwargs we expect some kwargs to be additional tensors which need to be on the right device we expect some kwargs to be additional tensors which need to be on the right device generatekwargs get priority over forwardparams in that case the output is a spectrogram that needs to be converted into a waveform generates speechaudio from the inputs see the texttoaudiopipeline documentation for more information args textinputs str or liststr the texts to generate forwardparams dict optional parameters passed to the model generationforward method forwardparams are always passed to the underlying model generatekwargs dict optional the dictionary of adhoc parametrization of generateconfig to be used for the generation call for a complete overview of generate check the following guidehttps huggingface codocstransformersenmainclassestextgeneration generatekwargs are only passed to the underlying model if the latter is a generative model return a dict or a list of dict the dictionaries have two keys audio np ndarray of shape nbchannels audiolength the generated audio waveform samplingrate int the sampling rate of the generated audio waveform 2023 the huggingface team licensed under the apache license version 2 0 the license you may not use this file except in compliance with the license you may obtain a copy of the license at http www apache org licenses license 2 0 unless required by applicable law or agreed to in writing software distributed under the license is distributed on an as is basis without warranties or conditions of any kind either express or implied see the license for the specific language governing permissions and limitations under the license from typing import list union text to audio generation pipeline using any automodelfortexttowaveform or automodelfortexttospectrogram this pipeline generates an audio file from an input text and optional other conditional inputs example python from transformers import pipeline pipe pipeline model suno bark small output pipe hey it s huggingface on the phone audio output audio sampling_rate output sampling_rate learn more about the basics of using a pipeline in the pipeline tutorial pipeline_tutorial tip you can specify parameters passed to the model by using texttoaudiopipeline __call__ forward_params or texttoaudiopipeline __call__ generate_kwargs example python from transformers import pipeline music_generator pipeline task text to audio model facebook musicgen small framework pt diversify the music generation by adding randomness with a high temperature and set a maximum music length generate_kwargs do_sample true temperature 0 7 max_new_tokens 35 outputs music_generator techno music with high melodic riffs generate_kwargs generate_kwargs tip this pipeline can currently be loaded from pipeline using the following task identifiers text to speech or text to audio see the list of available models on huggingface co models https huggingface co models filter text to speech get sampling_rate from config and generation config bark tokenizer is called with barkprocessor which uses those kwargs priority is given to kwargs we expect some kwargs to be additional tensors which need to be on the right device we expect some kwargs to be additional tensors which need to be on the right device generate_kwargs get priority over forward_params you re using the texttoaudiopipeline with a forward only model but generate_kwargs is non empty for forward only tta models please use forward_params instead of of generate_kwargs for reference here are the generate_kwargs used here generate_kwargs keys in that case the output is a spectrogram that needs to be converted into a waveform generates speech audio from the inputs see the texttoaudiopipeline documentation for more information args text_inputs str or list str the text s to generate forward_params dict optional parameters passed to the model generation forward method forward_params are always passed to the underlying model generate_kwargs dict optional the dictionary of ad hoc parametrization of generate_config to be used for the generation call for a complete overview of generate check the following guide https huggingface co docs transformers en main_classes text_generation generate_kwargs are only passed to the underlying model if the latter is a generative model return a dict or a list of dict the dictionaries have two keys audio np ndarray of shape nb_channels audio_length the generated audio waveform sampling_rate int the sampling rate of the generated audio waveform
from typing import List, Union

from ..utils import is_torch_available
from .base import Pipeline


if is_torch_available():
    from ..models.auto.modeling_auto import MODEL_FOR_TEXT_TO_SPECTROGRAM_MAPPING
    from ..models.speecht5.modeling_speecht5 import SpeechT5HifiGan

DEFAULT_VOCODER_ID = "microsoft/speecht5_hifigan"


class TextToAudioPipeline(Pipeline):
    

    def __init__(self, *args, vocoder=None, sampling_rate=None, **kwargs):
        super().__init__(*args, **kwargs)

        if self.framework == "tf":
            raise ValueError("The TextToAudioPipeline is only available in PyTorch.")

        self.vocoder = None
        if self.model.__class__ in MODEL_FOR_TEXT_TO_SPECTROGRAM_MAPPING.values():
            self.vocoder = (
                SpeechT5HifiGan.from_pretrained(DEFAULT_VOCODER_ID).to(self.model.device)
                if vocoder is None
                else vocoder
            )

        self.sampling_rate = sampling_rate
        if self.vocoder is not None:
            self.sampling_rate = self.vocoder.config.sampling_rate

        if self.sampling_rate is None:
            

            config = self.model.config
            gen_config = self.model.__dict__.get("generation_config", None)
            if gen_config is not None:
                config.update(gen_config.to_dict())

            for sampling_rate_name in ["sample_rate", "sampling_rate"]:
                sampling_rate = getattr(config, sampling_rate_name, None)
                if sampling_rate is not None:
                    self.sampling_rate = sampling_rate

    def preprocess(self, text, **kwargs):
        if isinstance(text, str):
            text = [text]

        if self.model.config.model_type == "bark":
            
            new_kwargs = {
                "max_length": self.model.generation_config.semantic_config.get("max_input_semantic_length", 256),
                "add_special_tokens": False,
                "return_attention_mask": True,
                "return_token_type_ids": False,
                "padding": "max_length",
            }

            
            new_kwargs.update(kwargs)

            kwargs = new_kwargs

        output = self.tokenizer(text, **kwargs, return_tensors="pt")

        return output

    def _forward(self, model_inputs, **kwargs):
        
        kwargs = self._ensure_tensor_on_device(kwargs, device=self.device)
        forward_params = kwargs["forward_params"]
        generate_kwargs = kwargs["generate_kwargs"]

        if self.model.can_generate():
            
            generate_kwargs = self._ensure_tensor_on_device(generate_kwargs, device=self.device)

            
            forward_params.update(generate_kwargs)

            output = self.model.generate(**model_inputs, **forward_params)
        else:
            if len(generate_kwargs):
                raise ValueError(
                    f
                )
            output = self.model(**model_inputs, **forward_params)[0]

        if self.vocoder is not None:
            
            output = self.vocoder(output)

        return output

    def __call__(self, text_inputs: Union[str, List[str]], **forward_params):
        
        return super().__call__(text_inputs, **forward_params)

    def _sanitize_parameters(
        self,
        preprocess_params=None,
        forward_params=None,
        generate_kwargs=None,
    ):
        params = {
            "forward_params": forward_params if forward_params else {},
            "generate_kwargs": generate_kwargs if generate_kwargs else {},
        }

        if preprocess_params is None:
            preprocess_params = {}
        postprocess_params = {}

        return preprocess_params, params, postprocess_params

    def postprocess(self, waveform):
        output_dict = {}

        output_dict["audio"] = waveform.cpu().float().numpy()
        output_dict["sampling_rate"] = self.sampling_rate

        return output_dict