
where \(\mathbf {W} = \mathbf {W}_L\)  is the last-layer linear classifier and \(\mathbf {h}_{k,i} =\mathbf {h}(\mathbf {x}_{k,i})=\phi _{\mathbf {\theta }}(\mathbf {x}_{k,i})\)  denotes a deep hierarchical representation (or feature) of the input \(\mathbf {x}_{k,i}\) . Here, for a \(L\) -layer deep network \(\psi _{\mathbf {\Theta }}(\mathbf {x})\) , each layer is composed of an affine transformation, followed by a nonlinear activation \(\sigma (\cdot )\)  and normalization functions (e.g., BatchNorm [1]}). We use \(\mathbf {\Theta }\)  to denote all the network parameters of \(\psi _{\mathbf {\Theta }}(\mathbf {x})\)  and \(\mathbf {\theta }\)  to denote the network parameters of \(\phi _{\mathbf {\theta }}(\mathbf {x})\) . Additionally, we use
\(\mathbf {H} \;=\; \begin{bmatrix}\mathbf {H}_1 & \mathbf {H}_2 & \cdots & \mathbf {H}_K\end{bmatrix} \in \mathbb {R}^{d \times N},\quad \mathbf {H}_k \;=\; \begin{bmatrix}\mathbf {h}_{k,1} & \cdots & \mathbf {h}_{k,n}\end{bmatrix} \in \mathbb {R}^{d \times n},\; 1\le k\le K,\) 
