Proof of Lemma 4.2(ii).: As a gentle warmup, we prove (ii) under the additional assumption that the model does not use normalization layers (LayerNorm or RMSNorm) in its architecture. To prove (ii) we show it is possible to find a two distinct sets of model parameters \(\theta,\theta^{\prime}\) with different embedding projection matrices that result in identical API outputs.