t5 was introduced by JMLR:v21:20-074.
It is an encoder-decoder Transformer architecture and has a multilingual version, mt5 [1]}.
It is trained on Colossal Clean Crawled Corpus (C4) and achieved sota on the SQuAD qa dataset, where it generates the answer token by token.
A simplified version of layer normalization is used such that no additive bias is used, in contrast to the original Transformer.
The self-attention of the decoder is a form of causal or autoregressive self-attention.
All the tasks considered for the model are cast into a unified text-to-text format, in terms of input and output.
This approach, despite its benefits, is known to suffer from prediction issues [2]}, [3]}.
Maximum likelihood is the training objective for all the tasks and a task-prefix is specified in the input before the model is fed, in order to identify the task at hand. The base version of the model has about 220M parameters.
