Scaling up the size of vision models has been thede factostandard to obtain more powerful visual representations. In this work, we discuss the point beyond which larger vision models arenotnecessary. First, we demonstrate the power ofScaling onScales (S2) , whereby a pre-trained and frozen smaller vision model (e.g., ViT-B or ViT-L) , run over multiple image scales, can outperform larger models (e.g., ViT-H or ViT-G)  on classification, segmentation, depth estimation, Multimodal LLM (MLLM)  benchmarks, and robotic manipulation. Notably, S2achieves state-of-the-art performance in detailed understanding of MLLM on the V∗benchmark, surpassing models such as GPT-4V. We examine the conditions under which S2is a preferred scaling approach compared to scaling on model size. While larger models have the advantage of better generalization on hard examples, we show that features of larger vision models can be well approximated by those of multi-scale smaller models. This suggests most, if not all, of the representations learned by current large pre-trained models can also be obtained from multi-scale smaller models. Our results show that a multi-scale smaller model has comparable learning capacity to a larger model, and pre-training smaller models with S2can match or even exceed the advantage of larger models. We release a Python package that can apply S2on any vision model withone line of code:https://github.com/bfshi/scaling_on_scales.