Deep neural networks have achieved impressive results in a wide variety of applications such as classification [1]}, [2]}, image processing [3]}, [4]}, natural language processing [5]}, [6]}, [7]}, etc. Most of these networks use millions and sometimes even billions of parameters which makes inference computationally expensive and memory intensive [5]}. To address this, researchers explore pruning techniques with the primary goal of comparing performance on real datasets without any theoretical guarantees. In this work, we take a step to bridge the gap between empirical observations from pruning studies and their theoretical justification. We compare the quality of different pruned feedforward neural networks under the teacher-student framework [9]}, [10]}, [11]}, [12]} in the thermodynamic limit (input dimension goes to infinity) using generalization error bounds, a theoretical measure of performance of machine learning models on unseen test data [13]} .
