While our results strongly suggest that linear mode connectivity breaks down between heuristic and generalizing models, they should nonetheless be connected, at least by a nonlinear path of constant training loss [1]}, [2]}. We consider the behavior of HANS generalization along this path, which maintains near-constant ID loss. These paths were easy to identify with Riemannian curves and segmented lines [3]}, and exhibited poor HANS-LO generalization the further along the path the parameters fall, as shown in Fig. REF .
