ForestPrune: Compact Depth-Controlled Tree Ensembles
Tree ensembles are versatile supervised learning algorithms that achieve
state-of-the-art performance. These models are extremely powerful but can grow
to enormous sizes. As a result, tree ensembles are often post-processed to
reduce memory footprint and improve interpretability. In this paper, we present
ForestPrune, a novel optimization framework that can post-process tree
ensembles by pruning depth layers from individual trees. We also develop a new
block coordinate descent method to efficiently obtain high-quality solutions to
optimization problems under this framework. The number of nodes in a decision
tree increases exponentially with tree depth, so pruning deep trees can
drastically improve model parsimony. ForestPrune can substantially reduce the
space complexity of an ensemble for a minimal cost to performance. The
framework supports various weighting schemes and contains just a single
hyperparameter to tune. In our experiments, we observe that ForestPrune can
reduce model size 20-fold with negligible performance loss.