In our work, we are interested in scaling along second axis i.e. model size first. The reasoning behind this choice is two-folds: a) training on large data requires large enough model in order to take advantage of the data scale and discover properties present in the dataset, and b) model size appears to be a strong lever for low-shot learning  [1]} and we are interested in pushing these limits further.
