We compare our results of T5-L (800M parameters) fine-tuned on various dataset sizes to those of GPT3 (175B, over 200 times larger than T5-L) using in-context learning, as reported by [1]}.Our work has 6 datasets that properly overlap with the original GPT3 paper. ARC-E and PIQA are also used to evaluate GPT3, but in the open QA format.
Table REF  shows a wide performance gap between GPT3 and T5-L on open QA datasets, which cannot be bridged by additional labeled examples, as observed in our main experiments.
However, for classification and extractive QA tasks, even a few hundred labeled examples are often enough for T5-L to catch up with GPT3's performance and even exceed it.
In BoolQ, for example, just collecting 96 additional examples is tantamount to adding 200 times more parameters to the model.
This result demonstrates that while performance may improve along the parameter axis in classification and extractive QA tasks, a small amount of labeled training data can also go a long way.
<TABLE>