Nevertheless, we argue that this paper does not show that _all_ small models should suffer from saturation, but rather that the saturation of small language models is symptomatic of a limitation that may affect language models that are based on a relatively small hidden dimension.