Online Learning in Teacher-Student Setup [1]}: We use a two-layer perceptron which has \(N\)  input units, \(M\)  hidden units and 1 output unit as the teacher network to generate labels for i.i.d Gaussian input, \(\mathbf {x}^t= (x_1^t,\ldots ,x_N^t)\)  (\(x_i^t \in \mathcal {N}(0,1)\)  \(\forall i \in \lbrace 1,\ldots ,N\rbrace \) ). Let \(\theta ^* = \lbrace \mathbf {w}^*(\in \mathbb {R}^{M \times N}), v^* \in \mathbb {R}^M\rbrace \)  denote the fixed parameters of the teacher network. The label \(y^t\)  of the input \(\mathbf {x}^t\)  (\(t = 1,2,\ldots \) ) is given as,
