Initializing from a single pretrained model can produce a range of in-domain generalization behaviors [1]}, [2]}, [3]}, [4]}. However, variation on performance in diagnostic sets is even more substantial, from social biases [3]} to unusual paraphrases [6]}, [7]}. [8]} found that a wide variety of decision boundaries were expressed within a low-loss volume, and [9]} further found that there is diversity in boundaries during OOD generalization, far off the data manifold. Our work contributes to the literature on how diverse functions can have similar loss by linking sets of models that share low dimensional subspaces to particular OOD generalization behavior. We also expand beyond visualizing decision boundaries by focusing on the geography of basins aligned with specific interpretable strategies, particularly syntax-aware heuristics.
