# T5ForSequenceClassification
**T5ForSequenceClassification** adapts the original [T5](https://github.com/google-research/text-to-text-transfer-transformer) architecture for sequence classification tasks.

T5 was originally built for text-to-text tasks and excels in it.
It can handle any NLP task if it has been converted to a text-to-text format, including sequence classification task!
You can find [here](https://huggingface.co/google/flan-t5-base?text=Premise%3A++At+my+age+you+will+probably+have+learnt+one+lesson.+Hypothesis%3A++It%27s+not+certain+how+many+lessons+you%27ll+learn+by+your+thirties.+Does+the+premise+entail+the+hypothesis%3F) how the original T5 is used for sequence classification task. 

Our motivations for building **T5ForSequenceClassification** is that the full original T5 architecture is not needed for most NLU tasks. Indeed, NLU tasks generally do not require to generate text and thus a large decoder is unnecessary.
By removing the decoder we can *half the original number of parameters* (thus half the computation cost) and *efficiently optimize* the network for the given task.

## Table of Contents

0. [Usage](#usage)
1. [Why use T5ForSequenceClassification?](#why-use-t5forsequenceclassification)
2. [T5ForClassification vs T5](#t5forclassification-vs-t5)
3. [Results](#results)

## Usage
**T5ForSequenceClassification** supports the task of zero-shot classification.
It can direclty be used for:
- topic classification
- intent recognition
- boolean question answering
- sentiment analysis
- and any other task which goal is to clasify a text...

Since the *T5ForClassification* class is currently not supported by the transformers library, you cannot direclty use this model on the Hub.
To use **T5ForSequenceClassification**, you will have to install additional packages and model weights.
You can find instructions [here](https://github.com/AntoineBlanot/zero-nlp).


## Why use T5ForSequenceClassification?
Models based on the [BERT](https://huggingface.co/bert-large-uncased) architecture like [RoBERTa](https://huggingface.co/roberta-large) and [DeBERTa](https://huggingface.co/microsoft/deberta-v2-xxlarge) have shown very strong performance on sequence classification task and are still widely used today.
However, those models only scale up to ~1.5B parameters (DeBERTa xxlarge) resulting in a limited knowledge compare to bigger models.
On the other hand, models based on the T5 architecture scale up to ~11B parameters (t5-xxl) and innovations with this architecture are very recent and keeps improving ([mT5](https://huggingface.co/google/mt5-xxl), [Flan-T5](https://huggingface.co/google/flan-t5-xxl), [UL2](https://huggingface.co/google/ul2), [Flan-UL2](https://huggingface.co/google/flan-ul2), and probably more...) 

## T5ForClassification vs T5
**T5ForClassification** Architecture:
- Encoder: same as original T5
- Decoder: only the first layer (for pooling purpose)
- Classification head: simple Linear layer on top of the decoder

Benefits and Drawbacks:
- (**+**) Keeps T5 encoding strength
- (**+**) Parameters size is half
- (**+**) Interpretable outputs (class logits)
- (**+**) No generation mistakes and faster prediction (no generation latency)
- (**-**) Looses text-to-text ability

## Results
Results on the validation data of **training tasks**:
| Dataset | Accuracy | F1 |
|:-------:|:--------:|:--:|
| MNLI (m)| 0.923 | 0.923 |
| MNLI (mm) | 0.922 | 0.922 |
| SNLI | 0.942 | 0.942 |
| SciTail | 0.966 | 0.647 |

Results on validation data of **unseen tasks** (zero-shot):
| Dataset | Accuracy | F1 |
|:-------:|:--------:|:--:|
| ?| ? | ? |

Special thanks to [philschmid](https://huggingface.co/philschmid) for making a Flan-T5-xxl [checkpoint](https://huggingface.co/philschmid/flan-t5-xxl-sharded-fp16) in fp16.

{"language": ["en"], "datasets": ["multi_nli", "snli", "scitail"], "metrics": ["accuracy", "f1"], "pipeline_tag": "zero-shot-classification", "model-index": [{"name": "AntoineBlanot/flan-t5-xxl-classif-3way", "results": [{"task": {"type": "nli", "name": "Natural Language Inference"}, "dataset": {"name": "MultiNLI", "type": "multi_nli", "split": "validation_matched"}, "metrics": [{"type": "accuracy", "value": 0.9230769230769231, "name": "Validation (matched) accuracy"}, {"type": "f1", "value": 0.9225172687920663, "name": "Validation (matched) f1"}]}, {"task": {"type": "nli", "name": "Natural Language Inference"}, "dataset": {"name": "MultiNLI", "type": "multi_nli", "split": "validation_mismatched"}, "metrics": [{"type": "accuracy", "value": 0.9222945484133441, "name": "Validation (mismatched) accuracy"}, {"type": "f1", "value": 0.9216699467726924, "name": "Validation (mismatched) f1"}]}, {"task": {"type": "nli", "name": "Natural Language Inference"}, "dataset": {"name": "SNLI", "type": "snli", "split": "validation"}, "metrics": [{"type": "accuracy", "value": 0.9418817313554155, "name": "Validation accuracy"}, {"type": "f1", "value": 0.9416213776111287, "name": "Validation f1"}]}, {"task": {"type": "nli", "name": "Natural Language Inference"}, "dataset": {"name": "SciTail", "type": "scitail", "split": "validation"}, "metrics": [{"type": "accuracy", "value": 0.9662576687116564, "name": "Validation accuracy"}, {"type": "f1", "value": 0.6471347983817357, "name": "Validation f1"}]}]}]}
transformers pytorch t5 zero-shot-classification en dataset:multi_nli dataset:snli dataset:scitail model-index endpoints_compatible text-generation-inference region:us
TAGS
#transformers #pytorch #t5 #zero-shot-classification #en #dataset-multi_nli #dataset-snli #dataset-scitail #model-index #endpoints_compatible #text-generation-inference #region-us 

T5ForSequenceClassification
===========================


T5ForSequenceClassification adapts the original T5 architecture for sequence classification tasks.


T5 was originally built for text-to-text tasks and excels in it.
It can handle any NLP task if it has been converted to a text-to-text format, including sequence classification task!
You can find here how the original T5 is used for sequence classification task.


Our motivations for building T5ForSequenceClassification is that the full original T5 architecture is not needed for most NLU tasks. Indeed, NLU tasks generally do not require to generate text and thus a large decoder is unnecessary.
By removing the decoder we can *half the original number of parameters* (thus half the computation cost) and *efficiently optimize* the network for the given task.


Table of Contents
-----------------


0. Usage
1. Why use T5ForSequenceClassification?
2. T5ForClassification vs T5
3. Results


Usage
-----


T5ForSequenceClassification supports the task of zero-shot classification.
It can direclty be used for:


* topic classification
* intent recognition
* boolean question answering
* sentiment analysis
* and any other task which goal is to clasify a text...


Since the *T5ForClassification* class is currently not supported by the transformers library, you cannot direclty use this model on the Hub.
To use T5ForSequenceClassification, you will have to install additional packages and model weights.
You can find instructions here.


Why use T5ForSequenceClassification?
------------------------------------


Models based on the BERT architecture like RoBERTa and DeBERTa have shown very strong performance on sequence classification task and are still widely used today.
However, those models only scale up to ~1.5B parameters (DeBERTa xxlarge) resulting in a limited knowledge compare to bigger models.
On the other hand, models based on the T5 architecture scale up to ~11B parameters (t5-xxl) and innovations with this architecture are very recent and keeps improving (mT5, Flan-T5, UL2, Flan-UL2, and probably more...)


T5ForClassification vs T5
-------------------------


T5ForClassification Architecture:


* Encoder: same as original T5
* Decoder: only the first layer (for pooling purpose)
* Classification head: simple Linear layer on top of the decoder


Benefits and Drawbacks:


* (+) Keeps T5 encoding strength
* (+) Parameters size is half
* (+) Interpretable outputs (class logits)
* (+) No generation mistakes and faster prediction (no generation latency)
* (-) Looses text-to-text ability


Results
-------


Results on the validation data of training tasks:



Results on validation data of unseen tasks (zero-shot):



Special thanks to philschmid for making a Flan-T5-xxl checkpoint in fp16.



TAGS
#transformers #pytorch #t5 #zero-shot-classification #en #dataset-multi_nli #dataset-snli #dataset-scitail #model-index #endpoints_compatible #text-generation-inference #region-us 
