In Figure 8, we compare the results of the head bottleneck experiment of the Pythia-1B model in Section 5.1 to the \(W\)-error on the head of the same model as the bottleneck dimension \(d\) evolves. It shows that the loss gap grows slowly with the \(W\)-error, implying that even when the allowed rank would lead to a poor approximation of \(W\), the performance can still remain acceptable. We notice that the performance starts decreasing when the \(W\)-error outgrows 0.6.