While models can benefit from both parameters and labeled data in many tasks, scaling up language models to hundreds of billions of parameters may restrict the ability to fine-tune, as GPT3-scale models are typically available only as a service to most practitioners and researchers [1]}.
Given this data-parameter trade-off, how many labeled examples are 175B parameters worth?
