We thank Jean-Stanislas Denain, Erik Jones, Ezra Karger, Jacob Pfau and Ruiqi Zhong for helpful discussions, and Jean-Stanislas Denain, Owain Evans, Dan Hendrycks, Horace He and Andy Zou for comments and feedbacks on an early draft of the paper. DH was supported by an award from the C3.ai Digital Transformation Institute. FZ was supported by NSF award CCF-2311648. JS was supported by the National Science Foundation SaTC CORE Award No. 1804794 and the Simons Foundation.