Our main contributions are: (1) We use GE bounds on the teacher-student framework to compare different pruning methods, which to the best of our knowledge, is the first theoretical advance in understanding pruning methods. (2) We prove that DPP node pruning outperforms random and importance node pruning methods, previously shown by  [1]} empirically. (3) We also theoretically show that random edge pruning performs better than DPP node pruning which is consistent with empirical observations from pruning literature that sparse models outperform dense models [2]}. (4) Finally, we propose DPP edge pruning method for feedforward networks which uses synaptic diversity to remove redundant edges and outperforms competing methods on real data. We also observe that DPP edge pruning finds a sparse network which performs better than the unpruned dense network, hence loosely connecting it to the Lottery Ticket Hypothesis [3]}.
