Pruning neural networks for wiring length efficiency is considered. Three techniques are proposed and experimentally tested: distance-based regularization, nested-rank pruning, and layer-by-layer bipartite matching. The first two algorithms are used in the training and pruning phases, respectively, and the third is used in the arranging neurons phase. Experiments show that distance-based regularization with weight based pruning tends to perform the best, with or without layer-by-layer bipartite matching. These results suggest that these techniques may be useful in creating neural networks for implementation in widely deployed specialized circuits. In this paper we consider the novel problem of learning accurate neural networks that have low total wiring length because this corresponds to energy consumption in the fundamental limit. We introduce weight-distance regularization, nested rank pruning, and layer-by-layer bipartite matching and show through ablation studies that all of these algorithms are effective, and can even reach state-of-the-art compression ratios. The results suggests that these techniques may be worth the computational effort if the neural network is to be widely deployed, if significantly lower energy is worth the slight decrease in accuracy, or if the application is to be deployed on either a specialized circuit or general purpose processor. Table 2 : Average and standard deviation over four trials for Street View House Numbers task on both the wiring length metric (energy) and remaining edges metric (edges). We note that with the appropriate hyperparameter setting our algorithm outperforms the baseline weight based techniques (p=0) often on both the energy and number of remaining edges metric. Table 3 : Results of applying the bipartite matching algorithm on the best performing weight-based pruning network and best performing distance-based regularization method before and after applying layer-by-layer bipartite matching. Average and standard deviation over 4 trials presented.
Three new algorithms with ablation studies to prune neural network to optimize for wiring length, as opposed to number of remaining weights.
{"src": "scitldr-AIC_train", "paper_id": "rygpbR2Pi7"}