# What does this PR do?

Until recently, there was only one TTS model in Transformers. Recent ([Bark](https://huggingface.co/docs/transformers/model_doc/bark)) and future ([FastSpeechConformer2](https://github.com/huggingface/transformers/pull/23439)) additions have and will further enrich the number of TTS models in Transformers. 
This may be the best time to add a text-to-speech pipeline to Transformers.

This PR tentatively proposes:

- The addition of a text-to-speech pipeline whose design could be modified in line with future TTS additions.
- Add a class AutoModelForTextToSpeech
- Add a `processor` task to the pipeline code to facilitate use of the `processor`.

My conception of the architecture for now:

- Backward compatibility with [FastSpeechConformer2](https://github.com/huggingface/transformers/pull/23439), retaining the ability to use its hacked `generate_speech` method.
- Future compatibility with future TTS models, counting on the fact that these models will use a `generate` method to generate audio.
- Possible compatibility with other TTA (text-to-audio) models such as [MusicGen](https://huggingface.co/docs/transformers/model_doc/musicgen).

What I'm counting on:
- future models should have a `generate` method, even if they are not AR models per se (for the moment, [FastSpeechConformer2](https://github.com/huggingface/transformers/pull/23439) is not AR and has no such method) or counts on an additional head model ([FastSpeechConformer2](https://github.com/huggingface/transformers/pull/23439) needs a vocoder on top to pass from a spectrogram to an audio - see [discussion here](https://github.com/huggingface/transformers/pull/23439#discussion_r1258660411)).
- future models will use a `Processor` even if they only use a tokenizer, to allow easy use of other conditional inputs such as audio or speaker embeddings.  And the processor must be added to `PROCESSOR_MAPPING` (not the case of [MusicGen](https://huggingface.co/docs/transformers/model_doc/musicgen) atm).

I'm open to further discuss the architecture and to make some changes!

EDIT: for reference, I've made another design choice following internal discussions. It is discussed [here](https://github.com/huggingface/transformers/pull/24952/#pullrequestreview-1556507240).

Fixes #22487

*Note:* I was inspired by @LysandreJik draft of a [TTS pipeline](https://huggingface.co/lysandre/text-to-speech-pipeline/blob/main/tts.py).

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? [LINK](https://github.com/huggingface/transformers/issues/22487#issuecomment-1496312713)
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Hey @sanchit-gandhi and @Narsil, I think you're the right people to talk to before the core review!

Add Text-To-Speech pipeline. # What does this PR do?

Until recently, there was only one TTS model in Transformers. Recent ([Bark](https://huggingface.co/docs/transformers/model_doc/bark)) and future ([FastSpeechConformer2](https://github.com/huggingface/transformers/pull/23439)) additions have and will further enrich the number of TTS models in Transformers. 
This may be the best time to add a text-to-speech pipeline to Transformers.

This PR tentatively proposes:

- The addition of a text-to-speech pipeline whose design could be modified in line with future TTS additions.
- Add a class AutoModelForTextToSpeech
- Add a `processor` task to the pipeline code to facilitate use of the `processor`.

My conception of the architecture for now:

- Backward compatibility with [FastSpeechConformer2](https://github.com/huggingface/transformers/pull/23439), retaining the ability to use its hacked `generate_speech` method.
- Future compatibility with future TTS models, counting on the fact that these models will use a `generate` method to generate audio.
- Possible compatibility with other TTA (text-to-audio) models such as [MusicGen](https://huggingface.co/docs/transformers/model_doc/musicgen).

What I'm counting on:
- future models should have a `generate` method, even if they are not AR models per se (for the moment, [FastSpeechConformer2](https://github.com/huggingface/transformers/pull/23439) is not AR and has no such method) or counts on an additional head model ([FastSpeechConformer2](https://github.com/huggingface/transformers/pull/23439) needs a vocoder on top to pass from a spectrogram to an audio - see [discussion here](https://github.com/huggingface/transformers/pull/23439#discussion_r1258660411)).
- future models will use a `Processor` even if they only use a tokenizer, to allow easy use of other conditional inputs such as audio or speaker embeddings.  And the processor must be added to `PROCESSOR_MAPPING` (not the case of [MusicGen](https://huggingface.co/docs/transformers/model_doc/musicgen) atm).

I'm open to further discuss the architecture and to make some changes!

EDIT: for reference, I've made another design choice following internal discussions. It is discussed [here](https://github.com/huggingface/transformers/pull/24952/#pullrequestreview-1556507240).

Fixes #22487

*Note:* I was inspired by @LysandreJik draft of a [TTS pipeline](https://huggingface.co/lysandre/text-to-speech-pipeline/blob/main/tts.py).

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? [LINK](https://github.com/huggingface/transformers/issues/22487#issuecomment-1496312713)
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Hey @sanchit-gandhi and @Narsil, I think you're the right people to talk to before the core review!

# Troubleshooting

This is a document explaining how to deal with various issues on Circle-CI. The entries may include actually solutions or pointers to Issues that cover those.

## Circle CI

* pytest worker runs out of resident RAM and gets killed by `cgroups`: https://github.com/huggingface/transformers/issues/11408

# Troubleshooting

This is a document explaining how to deal with various issues on Circle-CI. The entries may include actually solutions or pointers to Issues that cover those.

## Circle CI

* pytest worker runs out of resident RAM and gets killed by `cgroups`: https://github.com/huggingface/transformers/issues/11408
