No noise is added to the labels generated by the teacher network, i.e., \(\sigma = 0\)  in (REF ).

With the above assumptions, authors of  [1]} gave a closed form of the GE as follows:
\(\epsilon _{g} = f_1(Q) + f_2(T) - f_3(R,Q,T)\) 
where,
\(f_1(Q) & = \frac{1}{\pi } \sum _{i,k} v_i v_k \arcsin \frac{Q_{ik}}{\sqrt{1+Q_{ii}} \sqrt{1+Q_{kk}}}\\f_2(T) & = \frac{1}{\pi } \sum _{n,m} v_n^* v_m^* \arcsin \frac{T_{nm}}{\sqrt{1+T_{nn}} \sqrt{1+T_{mm}}}\\f_3(R,Q,T) & = \frac{2}{\pi } \sum _{i,n} v_i v_n^* \arcsin \frac{R_{in}}{\sqrt{1+Q_{ii}} \sqrt{1+T_{nn}}}\) 

where \(Q,R,T\)  are the order parameters as defined in section .
Properties of DPP Kernel
In section  we see that each node in the hidden layer of a student network carries certain amount of information about the training data and it is captured in a vector form. We create an information matrix by accumulating the information vectors of these hidden nodes. For simplicity of theoretical analysis, we have considered the kernel as the inner product of the information matrix. In the thermodynamic limit, the inner product is divided by the input dimension. Formally, if \(\mathbf {h}_i\)  and \(\mathbf {h}_j\)  are the information at \(i^{th}\)  hidden node and \(j^{th}\)  hidden node respectively, then
\(L_{ij} = \frac{1}{N} \frac{1}{n} \mathbf {h}_i^T \mathbf {h}_j\) 
where \(n\)  is the total number of training examples. It can be seen that the analysis for the kernel defined in Section  and REF  are similar. Note that all analyses are for the student network trying learn from the teacher network. Refer to Table REF  for details of notations.
Lemma 1 
Assume (A1) - (A7). Then the expected kernel of DPP Node for the hidden layer is the order parameter \(Q\) .

[Proof of Lemma REF ]
For the two-layer teacher-student setup, the hidden layer gets information \((\mathbf {h}_1,\ldots ,\mathbf {h}_{K})\)  from the input layer, where \(\mathbf {h}_i= (h_{i1},\ldots ,h_{in})\) 
and \(h_{ij} (= t_j^T\mathbf {w}_i)\)  is the information at \(i^{th}\)  hidden node on \(j^{th}\)  input data (\(t_j\) ). Hence,
\(\mathbf {h}_i^T\mathbf {h}_j = \sum _{k=1}^n h_{ik} h_{jk} = \sum _{k=1}^n t_k^T\mathbf {w}_i \cdot t_k^T\mathbf {w}_j = \sum _{k=1}^n \mathbf {w}_i^T t_k \cdot t_k^T\mathbf {w}_j = \sum _{k=1}^n \mathbf {w}_i^T (t_k t_k^T) \mathbf {w}_j\) 
But for the given input distribution (i.i.d. Gaussian), \(\mathbb {E}[t_k t_k^T] = \mathbf {I}_{N\times N}.\)  Hence, \(\raisebox {0.5ex}{\scalebox {0.8}{\displaystyle \lim _{N \rightarrow \infty }\;}} \mathbb {E}[L_{ij} ] =\raisebox {0.5ex}{\scalebox {0.8}{\displaystyle \lim _{N \rightarrow \infty }\;}}\mathbb {E}[ \frac{1}{N}\frac{1}{n}\mathbf {h}_i^T\mathbf {h}_j]= \raisebox {0.5ex}{\scalebox {0.8}{\displaystyle \lim _{N \rightarrow \infty }\;}} \frac{1}{N} \mathbf {w}_i^T \mathbf {w}_j=Q_{ij}\) , and we have the lemma.
From [1]} we know that \(Q\)  is a block diagonal matrix where each “block" (or “group" used interchangeably henceforth) refers to the set of student hidden nodes that represent (explian/learn) one particular teacher hidden node.
Lemma 2 
Assume (A1) - (A7). Then the expected kernel of DPP Edge for the \(s^{th}\)  hidden node is
\(\mathbb {E}[L_{ij}^s] ={\left\lbrace \begin{array}{ll}\frac{1}{N} w_{is}^2 & \text{ if } i=j \\0 & \text{ otherwise }\end{array}\right.}\) 

[Proof of Lemma REF ]
For the \(s^{th}\)  hidden node, we choose a subset of incoming edges w.r.t. a DPP. The representation for the \(j^{th}\)  incoming edge is \(\mathbf {u}_j^s = (u_{1j}^s,\ldots ,u_{nj}^s),\)  where \(u_{kj}^s=t_{kj}w_{js}.\)  Hence
\(L_{ij}^s = \frac{1}{N} \frac{1}{n} \sum _{k=1}^n u_{ki}^s u_{kj}^s = \frac{1}{N} \frac{1}{n} \sum _{k=1}^n t_{ki}w_{is} t_{kj}w_{js}\) 
As the input is drawn from i.i.d. Gaussian distribution, it can be seen that \(\mathbb {E}[t_{ki} t_{kj}] = 1 \iff i=j\) . Hence we have
\(\mathbb {E}[L_{ij}^s] ={\left\lbrace \begin{array}{ll}\frac{1}{N} w_{is}^2 & \text{ if } i=j \\0 & \text{ otherwise }\end{array}\right.}\) 
And we have the lemma.
This lemma basically states that the DPP edge method under the teacher student setup has a diagonal kernel.

Proof of the Theorems
[Proof of Theorem  and ]
Let \(H_R = \left\lbrace  h_{i_1}, \ldots , h_{i_{k_n}} \right\rbrace \)  be the set of selected nodes by DPP Node pruning method. Recall from [1]} that every student hidden node specializes in learning a teacher node. Denote \(t(h)\)  to be the teacher node learnt by \(h\) . \(S_{m}\subseteq H_R\)  be the set of selected hidden nodes of the pruned network which learnt the \(m^{th}\)  teacher node , i.e., \(S_m = \lbrace h \in H_R | t(h) = t_m \rbrace \)  (\(t_m\)  is the \(m^{th}\)  teacher node). Hence, \(prn = |\lbrace {1}(|S_m|>0)| 1 \le m \le M \rbrace | \)  is the number of teacher nodes explained by the pruned network and W.L.O.G. we can assume that \(t_1, \ldots , t_{prn}\)  are those set of teacher nodes. Let \(l_1, \ldots , l_{prn}\)  be the number of student nodes in the pruned network which learn the corresponding teacher node. Note that, \(\sum _{i=1}^{prn} l_i = k_n\)  and \(l_i\le Z\)  (where \(Z\)  is the number of student nodes dedicated to learn a single teacher node in the unpruned network) for all \(i\) . Applying Lemma REF  directly we can see that the GE for the pruned network is
\(\frac{(v^*)^2}{6}\left[ \sum _{i=1}^{prn}\left(1-\frac{l_i}{Z} \right)^2\right] + \frac{(M-prn)(v^*)^2}{6}\) 
The first part of (REF ) is the GE for the group whose corresponding teacher node is partially explained and the second part accounts for the GE due to unexplained teacher nodes (number of such teacher nodes are \(M-prn\) ). From Lemma REF  we know that the expected kernel matrix for DPP Node pruning is the order parameter \(Q\)  and it becomes a block diagonal matrix after the training converges, where size of each block is \(Z\)  (which is also the number of student nodes dedicated to learn a single teacher node in the unpruned network). Because of the block diagonal property of the DPP kernel matrix, at most 1 student hidden node will be chosen from each block, i.e., \(l_i =1\)  \(\forall i\) . Hence, \(prn = k_n\) . From Lemma REF  we can see that the GE of node pruned network only depends on the number of student node survived in each block after pruning, and, for DPP node pruning, it is always 1 (given \(k_n \le M\) ). This is why there is no expectation in the GE term. So for DPP node pruning the GE is,
\(\epsilon _{k_n}^{DPP\: Node}(f) = (v^*)^2 \left[ \frac{k_n}{6}\left( 1 - \frac{1}{Z}\right)^2 + \frac{M-k_n}{6} \right].\) 
Each of the \(k_n\)  student nodes in the pruned network learns a different teacher node. Consider one such teacher node and call it \(t_i\) . In the unpruned network, there are \(Z\)  student hidden nodes which learn a single teacher node \(t_i\) , only one of which survives after DPP node pruning. The first part of the error is due to the removal of student nodes (\(Z-1\)  student nodes for each \(t_i\) ). However, these errors can be retrieved by reweighting the survived student node. On the contrary, there are \(M-k_n\)  teacher nodes which don't have any representative (some student hidden node from the set of student nodes which specialized in this particular teacher node) in the pruned network. And the error (second part of the GE) due to those nodes can not be retrieved even after reweighting. Hence, the GE after reweighting becomes,
\((M-k_n) \times \frac{(v^*)^2 }{6}\) 
Thus, we have the Theorem .
Next, we will prove Theorem . We will show, for any network pruned by Random Node, the GE is more than the expected GE of DPP Node pruning. Recall the randomly pruned network \(f\)  discussed in the beginning of the proof. From Lemma REF  we can see that for node pruning the GE only depends on the number of nodes survived in each block. From (REF ) we have,
\(&\;\;\;\;\;\; \epsilon _{k_n}^{Rand\: Node}(f) \nonumber \\& = \frac{(v^*)^2}{6}\left[ \sum _{i=1}^{prn}\left(1-\frac{l_i}{Z} \right)^2\right] + \frac{(M-prn)(v^*)^2}{6}\nonumber \\& = \frac{(M-k_n)(v^*)^2}{6} +\sum _{i=1}^{prn} \left[ (l_i-1)\frac{(v^*)^2}{6} +\frac{(v^*)^2}{6} \left(1 - \frac{l_i}{Z}\right)^2 \right] \nonumber \\& \ge \frac{(M-k_n)(v^*)^2}{6} + \sum _{i=1}^{prn} l_i \frac{(v^*)^2}{6} \left(1-\frac{1}{Z}\right)^2 \\& = \frac{(M-k_n)(v^*)^2}{6} + l \frac{(v^*)^2}{6} \left(1-\frac{1}{Z}\right)^2\nonumber \\& = \epsilon _{k_n}^{DPP\: Node}(f) \nonumber \) 

where (REF ) follows from the inequality below:
\((l_i-1)\frac{(v^*)^2}{6} +\frac{(v^*)^2}{6} \left(1 - \frac{l_i}{Z}\right)^2 = l_i \frac{(v^*)^2}{6} \left[ 1 +\frac{1}{Z^2}-\frac{2}{Z}\right] \ge l_i \frac{(v^*)^2}{6} \left(1-\frac{1}{Z}\right)^2\) 
which proves the first part of Theorem . The proof for the reweighted network is similar.
In case of importance node pruning, the nodes with lowest absolute value of outgoing edges are dropped. Following [1]} the outgoing weights of all the hidden teacher nodes are equal (we call it \(v^*\) ). Also, from Lemma REF  we see that the sum of the weights of the outgoing edges of the student nodes which learn the same teacher node add up to the outgoing edge weight of the corresponding teacher hidden node. Moreover, we assume the ansatz \(v_i = v_j\)  when \(i,j \in G_n\) , where \(G_n\)  denotes the set of student nodes which learn the same teacher node \(t_n\) . Hence, we can see that all the outgoing edges are approximately similar. We also verify this fact experimentally. Therefore, this defines an approximately uniform distribution on the set of hidden nodes. Hence, this is almost same as random node pruning and so the result follows from Theorem .
Remark 3 The comparison between performance of importance node pruning and DIVNET depends on the fact that all the outgoing edges of the teacher hidden nodes are equal. However, when the outgoing weights are not equal the importance pruning first selects student hidden nodes from a group whose corresponding teacher node has the highest weight. Once all the student nodes are selected from that group then it selects the group whose corresponding teacher node has second highest outgoing edge weight and the process continues. Because of this approach, even without reweighting a complete information about the teacher node is preserved in the pruned network. However, in DPP node pruning one candidate from each group (representing a particular teacher node) is selected first. But if a member is selected from a group then the reweighting method can recover the complete lost information for the corresponding group. Hence, DIVNET is able to preserve information about more number of teacher hidden nodes than importance pruning which results in better performance.

[Proof of Theorem REF ]
In this theorem, we will give the GE of the expected network pruned by the Random Edge method. Pruning is performed on the edges between input layer and the hidden layer. Hence, the order parameter changes. From Lemma REF , we have the order parameters of the expected network (call these \(Q^{\prime },R^{\prime },T^{\prime }\) ). However, the weights of the second layer remain unchanged. Putting these values in (REF ), () and () we have,
\(f_1(Q^{\prime }) & = \frac{1}{\pi } \sum _{i,k} v_i v_k \arcsin \frac{Q^{\prime }_{ik}}{\sqrt{1+Q^{\prime }_{ii}} \sqrt{1+Q^{\prime }_{kk}}}\nonumber \\& = \frac{M(v^*)^2}{\pi } \arcsin \frac{c^2 }{1+c} +\frac{M(v^*)^2}{Z\pi }\left[\arcsin \frac{c}{1+c}-\arcsin \frac{c^2 }{1+c}\right]\) 

and,
\(f_3(R^{\prime },Q^{\prime },T^{\prime }) & = \frac{2}{\pi } \sum _{i,n} v_i v_n^* \arcsin \frac{R^{\prime }_{in}}{\sqrt{1+Q^{\prime }_{ii}} \sqrt{1+T^{\prime }_{nn}}}\nonumber \\& = \frac{2 M (v^*)^2}{\pi } \arcsin \frac{c }{\sqrt{2(1+c)}}\) 

Therefore, the GE of the expected network after Random Edge pruning is,
\(\frac{M(v^*)^2}{\pi }\left[ \arcsin \frac{c^2 }{1+c} + \frac{\pi }{6} - 2 \arcsin \frac{c }{\sqrt{2(1+c)}} \right] +\frac{M(v^*)^2}{Z\pi }\left[\arcsin \frac{c}{1+c}-\arcsin \frac{c^2 }{1+c}\right]\) 
This proves the first part of the theorem.
<FIGURE>[Proof of Theorem REF ]
Theorem  and REF  provide the closed form of the GE after DPP node pruning and random node pruning respectively. Using this closed form we plot \(\epsilon _{k_n}^{DPP\: Node}(f) - \epsilon _{c}^{Rand\: edge}(f)\)  in Figure REF  A. Here \(k_n\)  and \(c\)  satisfy (REF ), i.e., parameter count is same after two kinds of pruning. We can see for \(Z\ge 4\)  this value is \(\ge 0\)  given \(0\le c \le 1.0/Z,\)  which proves the theorem.
Remark 4 Our results hold for \(Z\ge 4\) , where \(Z\)  is the number of student nodes which learn the same teacher node. This is because in DPP node pruning at most 1 student node survives per group. As a result for larger \(Z\)  the lost information per group is higher (in the scale of \(\left(1-\frac{1}{Z}\right)^2\) ).

Next we state the impossibility result as discussed in Section . We will show that, no reweighting scheme in the second layer for random edge pruning which is based on scaling can beat DPP node pruning after reweighting. Formally we have the following:
Theorem 3 
Assume \((A1)-(A7)\) . Let \(k_n\)  and \(c\)  satisfy (REF ), and \(0 \le c \le \frac{1}{Z}\)  and \(Z\ge 4\) . Assume the reweighting scheme for random edge in second layer such that, \(\hat{v}_i = A v_i\) . Then \(\forall A \in \mathbb {R}\)  we have,

\(\hat{\epsilon }_{k_n}^{DPP\:Node}\left(f \right) \le \hat{\epsilon }_{c}^{Rand\:Edge}\left(\mathbb {E}\left[f\right] \right)\) 
[Proof of Theorem REF ]
From Theorem  we know that the GE after rewighting the DPP node pruned network is
\(\frac{(v^*)^2}{6} \left(M-k_n\right) = \frac{M (v^*)^2}{6} \left(1-Zc\right)\) 
where \(c\)  satisfies (REF ). Now for the given reweighting scheme in the hypothesis the GE for random edge pruning will be,
\(\frac{M(v^*)^2}{\pi }\left[ A^2 \left(\frac{1}{Z} \arcsin \frac{c}{1+c} +\left(1-\frac{1}{Z}\right) \arcsin \frac{c^2 }{1+c} \right) + \frac{\pi }{6} - 2 A \arcsin \frac{c }{\sqrt{2(1+c)}} \right]\) 
(REF ) can be viewed as a quadratic equation of \(A\)  whose minimum correspond to the best reweighting scheme in the scaling family. In Figure REF  B we compare this minimum with (REF ). Formally we plotted \(\hat{\epsilon }_{k_n}^{DPP\:Node}\left(f \right) - \hat{\epsilon }_{c}^{Rand\:Edge}\left(\mathbb {E}\left[f\right] \right).\)  It can be seen that this value is \(-ve\)  for all \(0 \le c \le \frac{1}{Z}\) , which implies GE of reweighted DPP node pruned network is always lower than reweighted random edge pruned network.
Proof of Lemmas
Lemma 3 
Assume (A1)-(A7). Let \(t_1, \ldots , t_{M}\)  denote the teacher hidden nodes and \(l_1, \ldots , l_{M}\)  denote the number of student hidden nodes in a node pruned network which learnt the corresponding teacher node. If \(\sum _{m=1}^M l_m \le M,\)  then the GE of this node pruned network is,
\(\frac{(v^*)^2}{6}\left[ \sum _{m=1}^{M}\left(1-\frac{l_m}{Z} \right)^2\right]. \) 

Let \(G_1,\ldots ,G_M\)  be the subsets of student nodes such that all student nodes in \(G_m\)  learn the \(m^{th}\)  teacher node. From the assumption we have, \(|G_m| = Z\)  for all \(m\) . After pruning, a subset \(P_m \subseteq G_m\)  is chosen, and \(|P_m| = l_m\) . Denote the order parameters of the pruned network as \(Q^{\prime },R^{\prime },T^{\prime }\) . For node pruning we can see that
\(Q^{\prime }_{ik} ={\left\lbrace \begin{array}{ll}Q_{ik} & \text{ if } \exists m \text{ s.t. } h_i \in P_m \text{ and } h_k \in P_m \\0 & \text{ otherwise }\end{array}\right.}\) 
Also, for the unpruned network we have
\(Q_{ik} ={\left\lbrace \begin{array}{ll}1 & \text{ if } \exists m \text{ s.t. } h_i \in G_m \text{ and } h_k \in G_m \\0 & \text{ otherwise }\end{array}\right.}\) 
Now from (REF ) we can break down the GE into three parts. From (REF ), () and () we have,.
\(f_1(Q^{\prime }) & = \frac{1}{\pi } \sum _{i,k} v_i v_k \arcsin \frac{Q^{\prime }_{ik}}{\sqrt{1+Q^{\prime }_{ii}} \sqrt{1+Q^{\prime }_{kk}}},\nonumber \\& = \frac{1}{\pi } \sum _{n=1}^{M} \sum _{i,k \in P_n} v_i v_k \arcsin \frac{1}{2},\\& = \frac{1}{\pi } \sum _{n=1}^{M} \sum _{i,k \in P_n} v_i v_k \frac{\pi }{6},\nonumber \\& = \frac{1}{6} \sum _{n=1}^{M} \left(\sum _{i \in P_{n}} v_i\right)^2,\nonumber \\& = \frac{(v^*)^2}{6} \sum _{n=1}^{M} \left(\frac{l_i}{Z}\right)^2\) 

(REF ) follows from the fact that \(h_i\)  and \(h_k\)  belong to the same group \(G_n\) . So we have,
\(\frac{Q^{\prime }_{ik}}{\sqrt{1+Q^{\prime }_{ii}} \sqrt{1+Q^{\prime }_{kk}}} = \frac{1}{\sqrt{2}\sqrt{2}} = \frac{1}{2}\) 
We can also see that () follows from Lemma REF  and the ansatz \(v_i = v_j\)  when \(i,j \in G_n\) . The order parameters \(T_{nm}\)  doesn't change after pruning, and so we have,
\(f_2(T^{\prime }) & = \frac{1}{\pi } \sum _{n,m} v_n^* v_m^* \arcsin \frac{T_{nm}}{\sqrt{1+T_{nn}} \sqrt{1+T_{mm}}},\nonumber \\& = \frac{1}{6} \sum _{n=1}^{M} (v^*)^2\) 

And similarly,
\(f_3(R^{\prime },Q^{\prime },T^{\prime }) & = \frac{2}{\pi } \sum _{i,n} v_i v_n^* \arcsin \frac{R^{\prime }_{in}}{\sqrt{1+Q^{\prime }_{ii}} \sqrt{1+T^{\prime }_{nn}}},\nonumber \\& = \frac{2}{\pi } \sum _{n=1}^{M} v_n ^* \sum _{i \in P_n} v_i \arcsin \frac{1}{2},\nonumber \\& = \frac{2}{6} \sum _{n=1}^{M} v_n ^* \sum _{i \in P_n} v_i.\) 

Then from (),(REF ) and (REF ) the GE of node pruning is,
\(\frac{(v^*)^2}{6}\left[ \sum _{m=1}^{M}\left(1-\frac{l_m}{Z} \right)^2\right].\) 
Hence we have the lemma.
Intuitively, this lemma states that for teacher hidden node \(t_n\)  if \(l_n\)  student hidden nodes survive after node pruning, then the fraction of information lost due to the deletion of nodes is \(1-\frac{l_n}{Z}\) , where \(Z\)  is the number of student nodes learn a particular teacher node in the unpruned network.
Lemma 4 
Let \(v^*\)  denotes the weight of the second layer of the teacher network and \(\lbrace v_1,\cdots ,v_K\rbrace \)  be the weights of the student network after convergence. Then in the noiseless case for all \(n\)  we have,
\(v^* = \sum _{i\in G_n} v_i\) 

[Proof of Lemma REF ]
From \((S36)\)  of [1]} we have,
\(\frac{dv_i}{dt} & = \eta _v \left[ \sum _{n=1}^{M} v_n ^* I_2(i,n) -\sum _{j=1}^{K} v_j I_2(i,j) \right] \\& = \eta _v \arcsin \frac{1}{2} \left[ v^* -\sum _{j \in G_{n}} v_j \right]\) 

Hence, a fixed point (in terms of \(v_i\) 's) of the ODE is,
\(\lbrace (v_1,\ldots ,v_K) | \sum _{i \in G_n} v_i = v^* ,\forall 1\le n\le M)\rbrace \) 
Intuitively, this lemma states that the sum of the outgoing edges of the student hidden nodes which learn a particular teacher hidden node is approximately equal to the weight of the outgoing edge of that teacher hidden node.
Lemma 5 
Let \(Q,R,T\)  are the order parameters of the unpruned network, and \(Q^{\prime },R^{\prime },T^{\prime }\)  are the respective order parameters after applying the Random Edge pruning where \(c\)  fraction of the edges are kept. Then we have the following:


\(\mathbb {E}[Q^{\prime }_{ik}] ={\left\lbrace \begin{array}{ll}c Q_{ik} & \text{ if } i = k \\c^2 Q_{ik} & \text{ otherwise }\end{array}\right.}\) 

\(\mathbb {E}[R^{\prime }_{st}] = c R_{st}\)

\(T^{\prime }_{mn} = T_{mn} \)


In case of Random Edge pruning each edge is kept with probability \(c\) . Then we have,
\( \mathbb {E}[Q^{\prime }_{st}] =\frac{1}{N} \sum _{i=1}^N c \cdot w_{is} \times c \cdot w_{it} = c^2 \frac{1}{N} \sum _{i=1}^N w_{is} w_{it} = c^2 Q_{st} \) 
and
\(\mathbb {E}[Q^{\prime }_{ss}] =\frac{1}{N} \sum _{i=1}^N c^2 \cdot w_{ss}^2= c^2 Q_{ss}.\) 
Similarly,
\(\mathbb {E}[R^{\prime }_{st}] =\frac{1}{N} \sum _{i=1}^N c \cdot w_{is} w^*_{it} = c R_{st}\) 
The teacher node is not affected by the pruning. So \(T\)  is not modified by the pruning process. This proves the lemma.
Intuitively, this lemma states that the order parameters of the pruned network using random edge pruning is a scaled version of the order parameters of the unpruned networks. However, the scaling of diagonal elements are different from that of off-diagonal elements (for more see Figure REF  A).

Simulation Details
In total, 10 rounds of simulations are run for each of the 6 pruning methods in Section REF , and we report the average and standard deviations (as error bars). The standard deviations are negligible (in the magnitude of \(10^{-3}\) ). A round is the entire process of generating a new teacher network with datasets, training the student from scratch, performing pruning and finally testing with the pruned network. For DPP and random methods, we sampled 100 masks per round and reported the average performance in each round. Given \(M = 2\)  and \(K = 6\) , we tried pruning with \([1, 2, 3, 4, 5]\)  nodes (and the equivalent number of edges) left in the student, respectively. We keep the total number of weights same to compare different pruning methods. The node-to-edge ratio, given \(N = 500\) , \(K = 6\) , and \(M = 2\) , is [\(1:83, 2:166, 3:250, 4:333, 5:417\) ]. This is calculated, for the teacher-student setup (single output node) specifically, as \(k_e = \frac{k_n(1 + N) - K}{K}\) . We grid-searched \(\eta \)  in the range of [\(0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7\) ] and found \(0.50\)  to be the optimal. We used \(\beta = 0.3\)  for all DPP node and edge kernel calculations in all simulations.

Hyperparameters for Real Datasets
Besides the hyperparameters and setup we proposed in Section REF  on the synthetic dataset, we report the hyperparameters used for the results on the MNIST and CIFAR10. As stated in Section REF , we used that exact same experiment setup (network architectures, error thresholds, etc.) as in [6]} for fair and consistent comparisons. We used SGD optimizers, a learning rate of \(0.001\) , and a momentum of \(0.9\)  for traning on both datasets. For  MNIST, the training batch size was 1000. For  CIFAR10, the training batch size was 128. All pruning methods were performed 10 times, and we report the means and standard deviations in Figure REF  (with reweighting) and Figure REF  (without reweighting).
The node-to-edge ratio for pruning, which keeps the number of parameters in the pruned network the same, is \([397:614, 472:921, 548:1228, 623:1536, 699:1843, 774:2150, 849:2457, 925:2764]\)  for CIFAR10 and \([256:156, 287:235, 317:313, 348:392, 378:470, 409:548, 439:627, 470:705]\)  for MNIST, given the network architecture in Table 1 of [6]}. These ratios correspond to \(20\%\)  to \(90\%\)  of the edges left for each node, as shown on the x-axis of Figure REF  and Figure REF . These node-to-edge ratios are calculated based on the conversion equation in Section REF . We used \(\beta = 10 / |T|\)  where \(|T|\)  is the size of the training dataset for all DPP node and edge kernel calculations on real data, following the choice of [6]}.

Tables and Figures
Table REF  shows the experimental results on the synthetic data with the setup discussed in Section REF . For all the node-to-edge ratios in (REF ), given \(K = 6\)  and \(M = 2\) , we calculated the mean square GEs for both the noiseless and noisy case (\(\sigma = 0.25\) ). We sampled 100 masks per simulation, and there are in total 10 rounds of simulations. As mentioned in Section REF , DPP methods are stable, and the standard deviations are in the magnitude of \(10^{-3}\)  for all ratios.
We observe that, DPP edge pruning in the noisy case with \(83\%\)  of the parameters outperforms the original unpruned loss in simulations (Table REF ). Similar behaviour for DPP edge pruning is also observed for real data (see Figure REF ) when reweighting is applied. We relate this loosely to the lottery ticket hypothesis. However, we clearly acknowledge the fact that our pruned model is not reinitialized and retrained, thereby deviating from the original hypothesis.
<TABLE><FIGURE><FIGURE><FIGURE>