We begin with a simpler case where \(g_{\theta}\) does not have residual connections but a fully connected (FC) final layer. In this case, for any invertible \(h\times h\) matrix \(\mathbf{S}\), we have that \(g_{\theta}\left(p\right)=\mathbf{S}_{g\theta^{\prime}}\left(p\right)\) where \(\theta^{\prime}\) is the same as \(\theta\) except that the weights of the final FC layer are pre-multiplied by \(\mathbf{S}^{-1}\). Hence, if \(g_{\theta}\) has a final FC layer, it is impossible to distinguish between the embedding projection layer \(\mathbf{W}\) acting on \(g_{\theta}\) and the embedding projection layer \(\mathbf{W}\cdot\mathbf{S}\) acting on \(g_{\theta^{\prime}}\), given access to the output of the API \(\mathcal{O}\) only.