Flexible Modeling and Multitask Learning using Differentiable Tree Ensembles
Decision tree ensembles are widely used and competitive learning models.
Despite their success, popular toolkits for learning tree ensembles have
limited modeling capabilities. For instance, these toolkits support a limited
number of loss functions and are restricted to single task learning. We propose
a flexible framework for learning tree ensembles, which goes beyond existing
toolkits to support arbitrary loss functions, missing responses, and multi-task
learning. Our framework builds on differentiable (a.k.a. soft) tree ensembles,
which can be trained using first-order methods. However, unlike classical
trees, differentiable trees are difficult to scale. We therefore propose a
novel tensor-based formulation of differentiable trees that allows for
efficient vectorization on GPUs. We perform experiments on a collection of 28
real open-source and proprietary datasets, which demonstrate that our framework
can lead to 100x more compact and 23% more expressive tree ensembles than those
by popular toolkits.