When running the full layer-stealing attack, we confirmed that our extracted weights are nearly identical to the actual weights, with error \(<7\cdot 10^{-4}\), up to an \(h\times h\) matrix product as discussed previously. Table 4 reports the RMS between our extracted weight matrix and the actual model weights, after "aligning" the two by an \(h\times h\) transform.