<commit_before>---
layout:     post
title:      "Reading Group: Deep Neural Decision Forests"
date:       2015-12-02
summary:    Thoughts and experiments on dNDFs
permalink:  /deep-neural-decision-forests/
categories: ["reading group"]
tags: ["reading group", "deep learning"]
---

_<NAME>, <NAME>, <NAME>, <NAME>_

**Table of Contents**

- [About This Post](#about-this-post)
- [Summary](#summary)
- [Top Thoughts](#top-thoughts)
- [MNIST Experiments](#mnist-experiments)
	- [I. Softmax Attention on Weight Matrix](#i-softmax-attention-on-weight-matrix)
	- [II. 2-Step Optimization](#ii-2-step-optimization)
	- [III. Softmax Attention on Probability Matrix](#iii-softmax-attention-on-probability-matrix)
	- [IV. Tree Attention on Probability Matrix](#iv-tree-attention-on-probability-matrix)
	- [V. In-network Ensembling](#v-in-network-ensembling)
	- [VI. Conclusions](#vi-conclusions)
- [CIFAR Experiments](#cifar-experiments)
- [And More](#and-more)

# About This Post

The following blog post was the collaborative output of the our DL reading group. We seek to be thoughtful about every aspect of our reading group and iteratively discover/measure/optimize the parameters of its success. More is written [here](/towards-a-better-reading-group/) about what we have learnt. Please email <a href="mailto:<EMAIL>?Subject=Reading%20Group" target="_top"><EMAIL></a> if you are interested in cutting edge optimization of DL algorithms and reading groups. (:

# Summary

Paper: [Deep Neural Decision Forests (dNDFs)](http://research.microsoft.com/apps/pubs/default.aspx?id=255952), <NAME>, <NAME>, <NAME>, <NAME>, ICCV 2015

The function spaces of neural networks and decision trees are quite different: the former is piece-wise linear while the latter learns sequences of hierarchical conditional rules. Accordingly, it feels intuitively appealing to combine those function spaces to leverage the strengths of both.

* Each activation $$f_n$$ of the final layer of the neural network corresponds to the routing probability $$d_n$$ of going left or right in the decision tree ![dNDF Forest](/assets/dndf-forest.png)

* $$d_n(x; \theta) = \sigma(f_n(x; \theta)) = P(1_{l \leftarrow n})$$ ![dNDF Activation to Probability](/assets/dndf-activation-to-probability.png)

* Each leaf is associated with a probability vector $$\pi_i$$ over the classes, e.g. {cat : 0.84, dog : 0.06, bird : 0.10} ![dNDF Probability Vector](/assets/dndf-probability-vector.png)

* The probability of a leaf node is the product of the (routing) probabilities of taking each left/right choice

* The output of a deep NDF is the weighted sum of each probability vector $$\pi_i$$ with its routing probability

As such, the network output is a soft attention over a constant set of probability vectors where that attention happens to be computed with a tree. The $$d_i$$ are trained using regular gradient descent while the optimal probability vectors can be solved using convex optimization before each epoch.

# Top Thoughts<commit_msg>fix wording<commit_after>---
layout:     post
title:      "Reading Group: Deep Neural Decision Forests"
date:       2015-12-02
summary:    Thoughts and experiments on dNDFs
permalink:  /deep-neural-decision-forests/
categories: ["reading group"]
tags: ["reading group", "deep learning"]
---

_<NAME>, <NAME>, <NAME>, <NAME>_

**Table of Contents**

- [About This Post](#about-this-post)
- [Summary](#summary)
- [Top Thoughts](#top-thoughts)
- [MNIST Experiments](#mnist-experiments)
	- [I. Softmax Attention on Weight Matrix](#i-softmax-attention-on-weight-matrix)
	- [II. 2-Step Optimization](#ii-2-step-optimization)
	- [III. Softmax Attention on Probability Matrix](#iii-softmax-attention-on-probability-matrix)
	- [IV. Tree Attention on Probability Matrix](#iv-tree-attention-on-probability-matrix)
	- [V. In-network Ensembling](#v-in-network-ensembling)
	- [VI. Conclusions](#vi-conclusions)
- [CIFAR Experiments](#cifar-experiments)
- [And More](#and-more)

# About This Post

The following blog post was the collaborative output of the our DL reading group. We seek to be thoughtful about every aspect of our reading group and iteratively discover/measure/optimize the parameters of its success. More is written [here](/towards-a-better-reading-group/) about what we have learnt. Please email <a href="mailto:<EMAIL>?Subject=Reading%20Group" target="_top"><EMAIL></a> if you are interested in cutting edge optimization of DL algorithms and reading groups. (:

# Summary

Paper: [Deep Neural Decision Forests (dNDFs)](http://research.microsoft.com/apps/pubs/default.aspx?id=255952), <NAME>, <NAME>, <NAME>, <NAME>, ICCV 2015

The function spaces of neural networks and decision trees are quite different: the former is piece-wise linear while the latter learns sequences of hierarchical conditional rules. Accordingly, it feels intuitively appealing to combine those function spaces to leverage the strengths of both.

* Each activation $$f_n$$ of the final layer of the neural network corresponds to the routing probability $$d_n$$ of going left or right in the decision tree ![dNDF Forest](/assets/dndf-forest.png)

* $$d_n(x; \theta) = \sigma(f_n(x; \theta)) = P(1_{l \leftarrow n})$$ ![dNDF Activation to Probability](/assets/dndf-activation-to-probability.png)

* Each leaf is associated with a probability vector $$\pi_i$$ over the classes, e.g. {cat : 0.84, dog : 0.06, bird : 0.10} ![dNDF Probability Vector](/assets/dndf-probability-vector.png)

* The probability of a leaf node is the product of the (routing) probabilities of taking each left/right choice

* The output of a deep NDF is the weighted sum of each probability vector $$\pi_i$$ with its routing probability

As such, the network output is a form of soft attention over a constant (but optimized) set of probability vectors where that attention happens to be computed with a tree. The $$d_i$$ are trained using regular gradient descent while the optimal probability vectors can be solved using convex optimization before each epoch.

# Top Thoughts
{'fraction_non_alphanumeric': 0.11107427055702918, 'fraction_numerical': 0.010610079575596816, 'mean_word_length': 4.7566793893129775, 'pattern_counts': {'":': 0, '<': 27, '<?xml version=': 0, '>': 27, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2644604', 'n_tokens_mistral': 1872, 'n_tokens_neox': 1691, 'n_words': 686}