Multi-layered representation is believed to be the key ingredient of deep neural networks especially in cognitive tasks like computer vision. While non-differentiable models such as gradient boosting decision trees (GBDTs) are the dominant methods for modeling discrete or tabular data, they are hard to incorporate with such representation learning ability. In this work, we propose the multi-layered GBDT forest (mGBDTs), with an explicit emphasis on exploring the ability to learn hierarchical representations by stacking several layers of regression GBDTs as its building block. The model can be jointly trained by a variant of target propagation across layers, without the need to derive back-propagation nor differentiability. Experiments and visualizations confirmed the effectiveness of the model in terms of performance and representation learning ability.
There is still no universal theory in explaining why a deep model works better than a shallow one. Many of the current attempts for this question are based on the conjecture that it is the hierarchical distributed representations learned from data are the driven forces behind the effectiveness of deep models. Similar works such as conjectured that better representations can be exploited to produce faster-mixing Markov chains, therefore, a deeper model always helps. Tishby and Zaslavsky @cite_1 treated the hidden layers as a successive refinement of relevant information and a deeper structure helps to speed up such process exponentially. Nevertheless, it seems for a deep model to work well, it is critical to obtain a better feature re-representation from intermediate layers.
@cite_1
2952742172
It has previously been hypothesized, and supported with some experimental evidence, that deeper representations, when well trained, tend to do a better job at disentangling the underlying factors of variation. We study the following related conjecture: better representations, in the sense of better disentangling, can be exploited to produce faster-mixing Markov chains. Consequently, mixing would be more efficient at higher levels of representation. To better understand why and how this is happening, we propose a secondary conjecture: the higher-level samples fill more uniformly the space they occupy and the high-density manifolds tend to unfold when represented at higher levels. The paper discusses these hypotheses and tests them experimentally through visualization and measurements of mixing and interpolating between samples.