# What does this PR do?

Until recently, there was only one TTS model in Transformers. Recent ([Bark](https://huggingface.co/docs/transformers/model_doc/bark)) and future ([FastSpeechConformer2](https://github.com/huggingface/transformers/pull/23439)) additions have and will further enrich the number of TTS models in Transformers. 
This may be the best time to add a text-to-speech pipeline to Transformers.

This PR tentatively proposes:

- The addition of a text-to-speech pipeline whose design could be modified in line with future TTS additions.
- Add a class AutoModelForTextToSpeech
- Add a `processor` task to the pipeline code to facilitate use of the `processor`.

My conception of the architecture for now:

- Backward compatibility with [FastSpeechConformer2](https://github.com/huggingface/transformers/pull/23439), retaining the ability to use its hacked `generate_speech` method.
- Future compatibility with future TTS models, counting on the fact that these models will use a `generate` method to generate audio.
- Possible compatibility with other TTA (text-to-audio) models such as [MusicGen](https://huggingface.co/docs/transformers/model_doc/musicgen).

What I'm counting on:
- future models should have a `generate` method, even if they are not AR models per se (for the moment, [FastSpeechConformer2](https://github.com/huggingface/transformers/pull/23439) is not AR and has no such method) or counts on an additional head model ([FastSpeechConformer2](https://github.com/huggingface/transformers/pull/23439) needs a vocoder on top to pass from a spectrogram to an audio - see [discussion here](https://github.com/huggingface/transformers/pull/23439#discussion_r1258660411)).
- future models will use a `Processor` even if they only use a tokenizer, to allow easy use of other conditional inputs such as audio or speaker embeddings.  And the processor must be added to `PROCESSOR_MAPPING` (not the case of [MusicGen](https://huggingface.co/docs/transformers/model_doc/musicgen) atm).

I'm open to further discuss the architecture and to make some changes!

EDIT: for reference, I've made another design choice following internal discussions. It is discussed [here](https://github.com/huggingface/transformers/pull/24952/#pullrequestreview-1556507240).

Fixes #22487

*Note:* I was inspired by @LysandreJik draft of a [TTS pipeline](https://huggingface.co/lysandre/text-to-speech-pipeline/blob/main/tts.py).

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? [LINK](https://github.com/huggingface/transformers/issues/22487#issuecomment-1496312713)
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Hey @sanchit-gandhi and @Narsil, I think you're the right people to talk to before the core review!

Add Text-To-Speech pipeline. # What does this PR do?

Until recently, there was only one TTS model in Transformers. Recent ([Bark](https://huggingface.co/docs/transformers/model_doc/bark)) and future ([FastSpeechConformer2](https://github.com/huggingface/transformers/pull/23439)) additions have and will further enrich the number of TTS models in Transformers. 
This may be the best time to add a text-to-speech pipeline to Transformers.

This PR tentatively proposes:

- The addition of a text-to-speech pipeline whose design could be modified in line with future TTS additions.
- Add a class AutoModelForTextToSpeech
- Add a `processor` task to the pipeline code to facilitate use of the `processor`.

My conception of the architecture for now:

- Backward compatibility with [FastSpeechConformer2](https://github.com/huggingface/transformers/pull/23439), retaining the ability to use its hacked `generate_speech` method.
- Future compatibility with future TTS models, counting on the fact that these models will use a `generate` method to generate audio.
- Possible compatibility with other TTA (text-to-audio) models such as [MusicGen](https://huggingface.co/docs/transformers/model_doc/musicgen).

What I'm counting on:
- future models should have a `generate` method, even if they are not AR models per se (for the moment, [FastSpeechConformer2](https://github.com/huggingface/transformers/pull/23439) is not AR and has no such method) or counts on an additional head model ([FastSpeechConformer2](https://github.com/huggingface/transformers/pull/23439) needs a vocoder on top to pass from a spectrogram to an audio - see [discussion here](https://github.com/huggingface/transformers/pull/23439#discussion_r1258660411)).
- future models will use a `Processor` even if they only use a tokenizer, to allow easy use of other conditional inputs such as audio or speaker embeddings.  And the processor must be added to `PROCESSOR_MAPPING` (not the case of [MusicGen](https://huggingface.co/docs/transformers/model_doc/musicgen) atm).

I'm open to further discuss the architecture and to make some changes!

EDIT: for reference, I've made another design choice following internal discussions. It is discussed [here](https://github.com/huggingface/transformers/pull/24952/#pullrequestreview-1556507240).

Fixes #22487

*Note:* I was inspired by @LysandreJik draft of a [TTS pipeline](https://huggingface.co/lysandre/text-to-speech-pipeline/blob/main/tts.py).

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? [LINK](https://github.com/huggingface/transformers/issues/22487#issuecomment-1496312713)
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Hey @sanchit-gandhi and @Narsil, I think you're the right people to talk to before the core review!

<!--Copyright 2022 NVIDIA and The HuggingFace Team. All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
specific language governing permissions and limitations under the License.

⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
rendered properly in your Markdown viewer.

-->

# GroupViT

## Overview

The GroupViT model was proposed in [GroupViT: Semantic Segmentation Emerges from Text Supervision](https://arxiv.org/abs/2202.11094) by Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, Xiaolong Wang.
Inspired by [CLIP](clip), GroupViT is a vision-language model that can perform zero-shot semantic segmentation on any given vocabulary categories.

The abstract from the paper is the following:

*Grouping and recognition are important components of visual scene understanding, e.g., for object detection and semantic segmentation. With end-to-end deep learning systems, grouping of image regions usually happens implicitly via top-down supervision from pixel-level recognition labels. Instead, in this paper, we propose to bring back the grouping mechanism into deep networks, which allows semantic segments to emerge automatically with only text supervision. We propose a hierarchical Grouping Vision Transformer (GroupViT), which goes beyond the regular grid structure representation and learns to group image regions into progressively larger arbitrary-shaped segments. We train GroupViT jointly with a text encoder on a large-scale image-text dataset via contrastive losses. With only text supervision and without any pixel-level annotations, GroupViT learns to group together semantic regions and successfully transfers to the task of semantic segmentation in a zero-shot manner, i.e., without any further fine-tuning. It achieves a zero-shot accuracy of 52.3% mIoU on the PASCAL VOC 2012 and 22.4% mIoU on PASCAL Context datasets, and performs competitively to state-of-the-art transfer-learning methods requiring greater levels of supervision.*

Tips:

- You may specify `output_segmentation=True` in the forward of `GroupViTModel` to get the segmentation logits of input texts. 

This model was contributed by [xvjiarui](https://huggingface.co/xvjiarui). The TensorFlow version was contributed by [ariG23498](https://huggingface.co/ariG23498) with the help of [Yih-Dar SHIEH](https://huggingface.co/ydshieh), [Amy Roberts](https://huggingface.co/amyeroberts), and [Joao Gante](https://huggingface.co/joaogante).
The original code can be found [here](https://github.com/NVlabs/GroupViT).

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with GroupViT.

- The quickest way to get started with GroupViT is by checking the [example notebooks](https://github.com/xvjiarui/GroupViT/blob/main/demo/GroupViT_hf_inference_notebook.ipynb) (which showcase zero-shot segmentation inference).
- One can also check out the [HuggingFace Spaces demo](https://huggingface.co/spaces/xvjiarui/GroupViT) to play with GroupViT. 

## GroupViTConfig

[[autodoc]] GroupViTConfig
    - from_text_vision_configs

## GroupViTTextConfig

[[autodoc]] GroupViTTextConfig

## GroupViTVisionConfig

[[autodoc]] GroupViTVisionConfig

## GroupViTModel

[[autodoc]] GroupViTModel
    - forward
    - get_text_features
    - get_image_features

## GroupViTTextModel

[[autodoc]] GroupViTTextModel
    - forward

## GroupViTVisionModel

[[autodoc]] GroupViTVisionModel
    - forward

## TFGroupViTModel

[[autodoc]] TFGroupViTModel
    - call
    - get_text_features
    - get_image_features

## TFGroupViTTextModel

[[autodoc]] TFGroupViTTextModel
    - call

## TFGroupViTVisionModel

[[autodoc]] TFGroupViTVisionModel
    - call
<!--Copyright 2022 NVIDIA and The HuggingFace Team. All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
specific language governing permissions and limitations under the License.

⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
rendered properly in your Markdown viewer.

-->

# GroupViT

## Overview

The GroupViT model was proposed in [GroupViT: Semantic Segmentation Emerges from Text Supervision](https://arxiv.org/abs/2202.11094) by Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, Xiaolong Wang.
Inspired by [CLIP](clip), GroupViT is a vision-language model that can perform zero-shot semantic segmentation on any given vocabulary categories.

The abstract from the paper is the following:

*Grouping and recognition are important components of visual scene understanding, e.g., for object detection and semantic segmentation. With end-to-end deep learning systems, grouping of image regions usually happens implicitly via top-down supervision from pixel-level recognition labels. Instead, in this paper, we propose to bring back the grouping mechanism into deep networks, which allows semantic segments to emerge automatically with only text supervision. We propose a hierarchical Grouping Vision Transformer (GroupViT), which goes beyond the regular grid structure representation and learns to group image regions into progressively larger arbitrary-shaped segments. We train GroupViT jointly with a text encoder on a large-scale image-text dataset via contrastive losses. With only text supervision and without any pixel-level annotations, GroupViT learns to group together semantic regions and successfully transfers to the task of semantic segmentation in a zero-shot manner, i.e., without any further fine-tuning. It achieves a zero-shot accuracy of 52.3% mIoU on the PASCAL VOC 2012 and 22.4% mIoU on PASCAL Context datasets, and performs competitively to state-of-the-art transfer-learning methods requiring greater levels of supervision.*

Tips:

- You may specify `output_segmentation=True` in the forward of `GroupViTModel` to get the segmentation logits of input texts. 

This model was contributed by [xvjiarui](https://huggingface.co/xvjiarui). The TensorFlow version was contributed by [ariG23498](https://huggingface.co/ariG23498) with the help of [Yih-Dar SHIEH](https://huggingface.co/ydshieh), [Amy Roberts](https://huggingface.co/amyeroberts), and [Joao Gante](https://huggingface.co/joaogante).
The original code can be found [here](https://github.com/NVlabs/GroupViT).

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with GroupViT.

- The quickest way to get started with GroupViT is by checking the [example notebooks](https://github.com/xvjiarui/GroupViT/blob/main/demo/GroupViT_hf_inference_notebook.ipynb) (which showcase zero-shot segmentation inference).
- One can also check out the [HuggingFace Spaces demo](https://huggingface.co/spaces/xvjiarui/GroupViT) to play with GroupViT. 

## GroupViTConfig

[[autodoc]] GroupViTConfig
    - from_text_vision_configs

## GroupViTTextConfig

[[autodoc]] GroupViTTextConfig

## GroupViTVisionConfig

[[autodoc]] GroupViTVisionConfig

## GroupViTModel

[[autodoc]] GroupViTModel
    - forward
    - get_text_features
    - get_image_features

## GroupViTTextModel

[[autodoc]] GroupViTTextModel
    - forward

## GroupViTVisionModel

[[autodoc]] GroupViTVisionModel
    - forward

## TFGroupViTModel

[[autodoc]] TFGroupViTModel
    - call
    - get_text_features
    - get_image_features

## TFGroupViTTextModel

[[autodoc]] TFGroupViTTextModel
    - call

## TFGroupViTVisionModel

[[autodoc]] TFGroupViTVisionModel
    - call