To confirm that the method from Appendix B.2.1 works, we test whether we can detect whether the GPT-2, Pythia and LLAMA architectures use LayerNorm or RMSNorm from their logit outputs alone. We found that the technique required two adjustments before it worked on models with lower than 32-bit precision (it always worked with 32-bit precision). i) We do not subtract \(\mathcal{O}\left(p_{0}\right)\) from logits queries, but instead subtract the mean logits over all queries, i.e. \(\frac{1}{n}\sum_{i=1}^{n}\mathcal{O}\left(p_{i}\right)\). Since the average of several points in a common affine subspace still lie on that affine subspace, this doesn't change the conclusions from Appendix B.2.1. ii) We additionally found it helped to calculate this mean in lower precision, before casting to 64-bit precision to calculate the compact SVD.