<commit_before>---
layout: post-light-feature
title: The Deterministic Information Bottleneck (DIB)
description: "How does the brain decide what to remember and what to forget?"
category: project
modified: 2014-12-16
tags: [prediction, compression, information bottleneck, information theory, machine learning, memory, learning, projects]
draft: true
comments: true
image:
  thumb: lijiang.jpg
---
Throughout your day, you are constantly bombarded with sensory information - the sound of other peoples' voices speaking to you, the sight of cars gliding by at a crosswalk, the smells of the food on your plate. You cannot possibly remember everything, so how does your brain decide what to keep and what to throw away? We propose that a useful metric for importance is predictivity - the most useful bits of sensory input are those that contain the most information about the future. The intuition is that the primary reason to collect information is to guide future actions, and information is only relevant for that process to the extent that it pertains to the future.

For example, imagine you are on a date. Try as you might, you won't be able to remember everything the other person says, but you might pay particular attention to details such as their favorite bands or types of foods, because these bits of information will help you predict their reactions to future date suggestions (and in particular, choose date options they will enjoy).

To summarize, we view sensory filtering as a tradeoff between compression and prediction - remember as few details as possible, but make sure the ones you do remember are the most useful for prediction. This approach was originally suggested in [Tishby, Pereira, & Bialek 2000](http://www.cs.huji.ac.il/labs/learning/Papers/allerton.pdf) and was dubbed the "information bottleneck" (IB) problem. Those authors formalized this tradeoff as a mathematical optimization problem using an information theoretic cost function. The optimization is done over the stochastic encoding distribution $q(r|x)$, where x and r are random variables representing past observations and internal representation, respectively. For example, in a typical neuroscience experiment, x might represent the location of a moving bar stimulus, and r might represent the spiking activity of a population of retinal ganglion cells. Algorithmically, the optimization is done via an iterative algorithm, except in a few simple cases where an analytic solution is available.

Our contribution is to introduce an alternative formulation of the IB method, which we call the deterministic information bottleneck (DIB). We motivate our formulation by arguing for an alternative to the IB cost function, which better represents the biologically-relevant goal of minimizing required memory resources. Then, we show that this seemingly minor change to the IB cost function has the dramatic effect of converting the stochastic encoding distribution $q(r|x)$ into a deterministic encoding function $r(x)$. Next, we propose an iterative algorithm for solving the DIB problem. Currently, we are working on comparing the IB and DIB methods on a variety of synthetic datasets. We are also testing their relative relevance to the brain by comparing how well each approach describes the coding of visual stimuli by populations of retinal ganglion cells. (For an IB-only analysis, see [Palmer et al. 2013](http://arxiv.org/abs/1307.0225).)

This work stems from discussions with [<NAME>](http://cbl.eng.cam.ac.uk/Public/Turner/WebHome) that we had while preparing to give a tutorial on the IB at [Cambridge's CBL](http://cbl.eng.cam.ac.uk/Public/WebHome). It is now joint work with [<NAME>](http://www.physics.northwestern.edu/people/personalpages/DavidSchwab.html) at Northwestern, [<NAME>](http://pondside.uchicago.edu/oba/faculty/palmer_s.html) at UChicago, and [<NAME>](http://www.princeton.edu/~wbialek/wbialek.html) at Princeton.<commit_msg>replacing | with \|<commit_after>---
layout: post-light-feature
title: The Deterministic Information Bottleneck (DIB)
description: "How does the brain decide what to remember and what to forget?"
category: project
modified: 2014-12-16
tags: [prediction, compression, information bottleneck, information theory, machine learning, memory, learning, projects]
draft: true
comments: true
image:
  thumb: lijiang.jpg
---
Throughout your day, you are constantly bombarded with sensory information - the sound of other peoples' voices speaking to you, the sight of cars gliding by at a crosswalk, the smells of the food on your plate. You cannot possibly remember everything, so how does your brain decide what to keep and what to throw away? We propose that a useful metric for importance is predictivity - the most useful bits of sensory input are those that contain the most information about the future. The intuition is that the primary reason to collect information is to guide future actions, and information is only relevant for that process to the extent that it pertains to the future.

For example, imagine you are on a date. Try as you might, you won't be able to remember everything the other person says, but you might pay particular attention to details such as their favorite bands or types of foods, because these bits of information will help you predict their reactions to future date suggestions (and in particular, choose date options they will enjoy).

To summarize, we view sensory filtering as a tradeoff between compression and prediction - remember as few details as possible, but make sure the ones you do remember are the most useful for prediction. This approach was originally suggested in [Tishby, Pereira, & Bialek 2000](http://www.cs.huji.ac.il/labs/learning/Papers/allerton.pdf) and was dubbed the "information bottleneck" (IB) problem. Those authors formalized this tradeoff as a mathematical optimization problem using an information theoretic cost function. The optimization is done over the stochastic encoding distribution $q(r\|x)$, where x and r are random variables representing past observations and internal representation, respectively. For example, in a typical neuroscience experiment, x might represent the location of a moving bar stimulus, and r might represent the spiking activity of a population of retinal ganglion cells. Algorithmically, the optimization is done via an iterative algorithm, except in a few simple cases where an analytic solution is available.

Our contribution is to introduce an alternative formulation of the IB method, which we call the deterministic information bottleneck (DIB). We motivate our formulation by arguing for an alternative to the IB cost function, which better represents the biologically-relevant goal of minimizing required memory resources. Then, we show that this seemingly minor change to the IB cost function has the dramatic effect of converting the stochastic encoding distribution $q(r\|x)$ into a deterministic encoding function $r(x)$. Next, we propose an iterative algorithm for solving the DIB problem. Currently, we are working on comparing the IB and DIB methods on a variety of synthetic datasets. We are also testing their relative relevance to the brain by comparing how well each approach describes the coding of visual stimuli by populations of retinal ganglion cells. (For an IB-only analysis, see [Palmer et al. 2013](http://arxiv.org/abs/1307.0225).)

This work stems from discussions with [<NAME>](http://cbl.eng.cam.ac.uk/Public/Turner/WebHome) that we had while preparing to give a tutorial on the IB at [Cambridge's CBL](http://cbl.eng.cam.ac.uk/Public/WebHome). It is now joint work with [<NAME>](http://www.physics.northwestern.edu/people/personalpages/DavidSchwab.html) at Northwestern, [<NAME>](http://pondside.uchicago.edu/oba/faculty/palmer_s.html) at UChicago, and [<NAME>](http://www.princeton.edu/~wbialek/wbialek.html) at Princeton.
{'fraction_non_alphanumeric': 0.057341772151898736, 'fraction_numerical': 0.0060759493670886075, 'mean_word_length': 5.054406130268199, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 11, 'https://': 0, 'lorem ipsum': 0, 'www.': 6, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2064995', 'n_tokens_mistral': 1939, 'n_tokens_neox': 1771, 'n_words': 1091}