Optimal stimulus and noise distributions for information transmission
  via suprathreshold stochastic resonance
Optimal Stimulus and Noise Distributions for Information
Transmission via Suprathreshold Stochastic Resonance
Mark D. McDonnell,1, ∗ Nigel G. Stocks,2 and Derek Abbott1
1 School of Electrical and Electronic Engineering & Centre for Biomedical Engineering,
The University of Adelaide, SA 5005, Australia
2School of Engineering, The University of Warwick,
Coventry CV4 7AL, United Kingdom
(Dated: November 4, 2018)
Abstract
Suprathreshold stochastic resonance (SSR) is a form of noise enhanced signal transmission that
occurs in a parallel array of independently noisy identical threshold nonlinearities, including model
neurons. Unlike most forms of stochastic resonance, the output response to suprathreshold random
input signals of arbitrary magnitude is improved by the presence of even small amounts of noise.
In this paper the information transmission performance of SSR in the limit of a large array size is
considered. Using a relationship between Shannon’s mutual information and Fisher information,
a sufficient condition for optimality, i.e. channel capacity, is derived. It is shown that capacity is
achieved when the signal distribution is Jeffrey’s prior, as formed from the noise distribution, or
when the noise distribution depends on the signal distribution via a cosine relationship. These
results provide theoretical verification and justification for previous work in both computational
neuroscience and electronics.
PACS numbers: 02.50.-r,05.40.Ca,89.70.+c,87.19.La
∗Electronic address: mmcdonne@eleceng.adelaide.edu.au (all correspondence)
http://arxiv.org/abs/0704.0673v1
mailto:mmcdonne@eleceng.adelaide.edu.au (all correspondence)
I. INTRODUCTION
The term ‘stochastic resonance’ describes the situation where a system’s response to some
signal is optimized by the presence of random noise, rather than its absence. It occurs in a
wide variety of nonlinear physical [1] and biological [2] systems.
In many of the systems and models in which stochastic resonance (SR) has been observed,
the essential nonlinearity is a single static threshold, e.g. [3, 4, 5, 6]. It is generally thought
that SR cannot occur in such systems for suprathreshold signals, meaning that the amplitude
of the input signal needs to be restricted to values smaller than the amplitude of the threshold
for SR to occur [7].
However, the 1999 discovery of a novel form of SR—known as suprathreshold stochastic
resonance (SSR)—showed that this is not always true [8]. SSR occurs in an array of identical
threshold nonlinearities, each of which are subject to independently random additive noise.
We refer to this array as the SSR model—see Fig. 1. In this model SR occurs regardless of
whether the input signal is entirely subthreshold or not. Furthermore, SSR occurs even for
very large input SNRs. This is a further difference to conventional SR, for which the signal
is required to be weak compared to the noise.
SSR is a form of aperiodic stochastic resonance [4, 9, 10] that was first shown to occur by
calculating Shannon’s average mutual information for the SSR model [8]. It was subsequently
found that the performance achievable via SSR is maximized when all threshold values are set
to the signal mean [11], and that for sufficiently small input SNRs, modifying the thresholds
in the model cannot improve information transfer [12].
The SSR model was originally motivated as a model for parallel sensory neurons, such
as those synapsing with hair cells in the inner ear [13]. Although the basic SSR model is
non-dynamical, and does not model the many complexities of real neurons, each threshold
nonlinearity is equivalent to a Pitts-McCulloch neuron model, and encapsulates the neural
coding properties we are interested in—i.e. the generation of action potentials in response
to a noisy aperiodic random stimulus. The small input SNRs we focus on are biologically
relevant [14], particularly so for hair cells, which are subject to substantial Brownian mo-
tion [15]. This leads to much randomness in the release of neurotransmitters at synapses
with afferent neurons leading to the cochlear nucleus.
Further justification of the SSR model’s relevance to neural coding is discussed in [16,
17, 18], and by extensions of the model to include more biologically realistic neural features.
For example, the parallel array has been modified to consist of parallel FitzHugh-Nagumo
neuron models [19], leaky integrate-and-fire neuron models [16, 17] and Hodgkin-Huxley
models [16], and for the case of signal-dependent (multiplicative) noise [18]. In all cases the
same qualitative results as for the simple threshold model were obtained. The SSR effect has
also led to a proposal for improving the performance of cochlear implants for suprathreshold
stimuli [13], based on the idea that the natural randomness present in functioning cochlear
hair cells is missing in patients requiring implants [20].
The purpose of this paper is to analyze, in a general manner, the information theoretic
upper limits of performance of the SSR model. This requires allowing the array size, N , to
approach infinity. Previous work has discussed the scaling of the mutual information through
the SSR model with N for specific cases, and found conditions for which the maximum
mutual information—i.e. channel capacity—occurs [11, 16, 21]. In a neural coding context,
the question of ‘what is the optimal stimulus distribution?’ for a given noise distribution is
discussed numerically for the SSR model in [16].
In Sec. II, we significantly extend the results in [11, 16, 21], by showing that the mu-
tual information and output entropy can both be written in terms of simple relative en-
tropy expressions—see Eqs. (21) and (22). This leads to a very general sufficient condition,
Eq. (25), for achieving capacity in the large N regime that can be achieved either by op-
timizing the signal distribution for a given noise distribution, or optimizing the noise for a
given signal. Given the neuroscience motivation for studying the SSR model, this result is
potentially highly significant in computational neuroscience, where both optimal stimulus
distributions, and optimal tuning curves are often considered [16, 22].
Furthermore, the optimal signal for the special case of uniform noise is shown to be
the arcsine distribution (a special case of the Beta distribution), which has a relatively large
variance and is bimodal. This result provides theoretical justification for a proposed heuristic
method for analog-to-digital conversion based on the SSR model [23]. In this method, the
input signal is transformed so that it has a large variance and is bimodal.
As a means of verification of our theory, in Sec. III our general results are compared
to the specific capacity results contained in [11, 16, 21]. This leads us to find and justify
improvements to these previous results.
Before we proceed however, the remainder of this section outlines our notation, describes
the SSR model, and derives some important results that we utilize.
A. Information Theoretic Definitions
Recent work using the SSR model has described performance using measures other than
mutual information [24, 25, 26, 27, 28, 29]. However, in line with much theoretical neuro-
science research [14], here we use the information theoretic viewpoint where the SSR model
can be considered to be a communications channel [8].
Throughout, we denote the probability mass function (PMF) of a discrete random vari-
able, α, as Pα(·), the probability density function (PDF) of a continuous random variable,
β, as fβ(·), and the cumulative distribution function (CDF) of β as Fβ(·).
All signals are discrete-time memoryless sequences of samples drawn from the same sta-
tionary probability distribution. This differs from the detection scenario often considered in
SR research, in which the input signal is periodic. Such a signal does not convey new infor-
mation with an increasing number of samples, and cannot be considered from an information
theoretic viewpoint [7].
Consider two continuous random variables, X and Y , with PDFs fX(x) and fY (x), with
the same support, S. The relative entropy—or Kullback-Liebler divergence—between the
two distributions is defined as [30]
D(fX ||fY ) =
fX(η) log2
fX(η)
fY (η)
dη. (1)
Suppose X and Y have joint PDF, fXY (x, y). Shannon’s mutual information between X and
Y is defined as the relative entropy between the joint PDF and the product of the marginal
PDFs [30],
I(X, Y ) =
fXY (x, y) log2
fXY (x, y)
fX(x)fY (y)
= H(Y )−H(Y |X) bits per sample. (2)
where H(Y ) is the entropy of Y and H(Y |X) is the average conditional entropy of Y given
The definition of mutual information also holds for discrete random variables, and for
one variable discrete and one continuous. The entropy of a discrete random variable, Y , is
given by
H(Y ) = −
PY (n) log2 PY (n), (3)
while a continuous random variable, X , has differential entropy
H(X) = −
fX(η) log2 (fX(η))dη. (4)
In this paper we are interested in the case of X continuous with support S and Y discrete,
with N states, in which case the average conditional entropy of Y given X is
H(Y |X) = −
fX(x)
PY |X(n|x) log2 (PY |X(n|x))dx. (5)
In information theory, the term channel capacity is defined as being the maximum achiev-
able mutual information of a given channel [30]. Suppose X is the source random variable,
and Y is the random variable at the output of the channel. Usually, the channel is assumed
to be fixed and the maximization performed over all possible source PDFs, fX(x). The
channel capacity, C, can be expressed as the optimization problem,
Find: C = max
{fX(x)}
I(X, Y ). (6)
Usually there are prescribed constraints on the source distribution such as a fixed average
power, or a finite alphabet [30]. In Sec. III we will also consider the more stringent constraint
that the PDF of the source is known other than its variance. In this situation, channel
capacity is determined by finding the optimal source variance, or as is often carried out in
SR research, the optimal noise variance.
B. SSR Model
Fig. 1 shows a schematic diagram of the SSR model. The array consists of N parallel
threshold nonlinearities—or ‘devices’, each of which receive the same random input signal,
X , with PDF fX(·). The i–th device in the model is subject to continuously valued iid—
independent and identically distributed—additive random noise, ηi (i = 1, .., N), with PDF
fη(·). Each noise signal is required to also be independent of the signal, X . The output of
each device, yi, is unity if the input signal, X , plus the noise on that device’s threshold, ηi, is
greater than the threshold value, θ. The output signal is zero otherwise. The outputs from
each device, yi, are summed to give the overall output signal, y =
i=1 yi. This output is
integer valued, y ∈ [0, .., N ], and is therefore a quantization (digitization) of X [28].
The conditional PMF of the output given the input is Py|X(y = n|X = x), n ∈ [0.., N ].
We abbreviate this to Py|X(n|x). The output distribution is
Py(n) =
Py|X(n|x)fX(x)dx n ∈ 0, .., N. (7)
The mutual information between X and y is that of a semi-continuous channel [8], and
can be written as
I(X, y) = H(y)−H(y|X)
Py(n) log2 Py(n)−
fX(x)
Py|X(n|x) log2 Py|X(n|x)dx
. (8)
To progress further we use the notation introduced in [8]. Let P1|x be the probability of
the i–th threshold device giving output yi = 1 in response to input signal value, X = x. If
the noise CDF is Fη(·), then
P1|x = 1− Fη(θ − x). (9)
As noted in [8], Py|X(n|x) is given by the binomial distribution as
Py|X(n|x) =
P n1|x(1− P1|x)N−n n ∈ 0, .., N, (10)
and Eq. (8) reduces to
I(X, y) =−
Py(n) log2
Py(n)
fX(x)P1|x log2 P1|xdx+
fX(x)(1− P1|x) log2 (1− P1|x)dx. (11)
Numerically evaluating Eq. (11) as a function of input SNR for given signal and noise
distributions finds that the mutual information has a unimodal stochastic resonance curve
for N > 1, even when the signal and noise are both suprathreshold—i.e. the threshold value,
θ, is set to the signal mean [11, 24].
Further analytical simplification of Eq. (8) is possible in the case where the signal and
noise PDFs are identical with the same variance, i.e. fX(x) = fη(θ− x) ∀ x [11]. The result
I(X, y) = log2 (N + 1)−
2 ln 2
N + 1
(N + 1− 2n) log2 n. (12)
What is quite remarkable about this result is that the mutual information is independent
of the shape of the PDFs of the signal and noise, other than that fX(x) = fη(θ − x) ∀ x.
This means that both PDFs have the same shape, but may possibly have different means,
and be mutually reversed along the x-axis about their means. In Sec. IID we compare the
mutual information of Eq. (12) with our calculations of the general channel capacity.
C. Describing SSR Using a Single PDF, fQ(τ)
We now show that the mutual information in the SSR model depends solely on N , and
an auxiliary PDF, fQ(·). This PDF is shown to be that of the random variable describing
the conditional average output of the SSR model, given that the input signal is X = x.
1. fQ(τ) as the PDF of the Average Transfer Function
Although the output of the SSR model, y, is a discrete random variable, the conditional
expected value of y, given the input is X = x, is a continuous random variable, since X is. We
label this random variable as Ȳ . Since the PMF of y given X = x is the binomial PMF as in
Eq. (10), we know that Ȳ is the random variable that results from ȳ = E[y|X = x] = NP1|x.
Inverting this gives x = θ − F−1η
1− ȳ
The PDF of Ȳ can be derived from fX(·), since ȳ = NP1|x provides an invertible trans-
formation of X , with PDF fX(x), to Ȳ , with PDF fȲ (ȳ). Using the well known expression
for the resultant PDF, and provided the support of fX(x) is contained in the support of
fη(θ − x)—since otherwise dxdȳ does not necessarily exist—we have
fȲ (ȳ) =fX (x)
x=θ−F−1η (1− ȳN )
fX(x)
Nfη(θ − x)
x=θ−F
η (1− ȳN )
, ȳ ∈ [0, N ]. (13)
Our condition regarding the supports of the signal and noise ensures that fη(·) 6= 0. If we
make a further change to a new random variable, Q, via τ = ȳ
, the PDF of Q is
fQ(τ) =
fX(x)
fη(θ − x)
x=θ−F
η (1−τ)
, τ ∈ [0, 1], (14)
and the PDF of Ȳ can be written as
fȲ (ȳ) =
, (15)
which illustrates the physical significance of the auxiliary PDF, fQ(·), as the PDF of ȳN .
2. Mutual Information in Terms of fQ(τ)
Making a change of variable in Eq. (11) from x to τ , via τ = P1|x = 1− Fη(θ − x) gives
I(X, y) =−
Py(n) log2
Py(n)
∫ τ=1
fQ(τ)τ log2 τdτ+
∫ τ=1
fQ(τ)(1 − τ) log2 (1− τ)dτ, (16)
where
Py(n) =
∫ τ=1
fQ(τ)τ
n(1− τ)N−ndτ. (17)
Eqs. (16) and (17) show that the PDF fQ(τ) encapsulates the behavior of the mutual infor-
mation in the SSR model.
3. Entropy of the random variable, Q
If we make a change of variable from τ to x, and note that fX(x)dx = fQ(τ)dτ , the
entropy of Q can be written as
H(Q) = −
fQ(τ) log2 (fQ(τ))dτ
fX(x) log2
fX(x)
fη(θ − x)
= −D(fX(x)||fη(θ − x)), (18)
which is the negative of the relative entropy between the signal PDF, and the noise PDF
reversed about x = 0 and shifted by θ. In the event that the noise PDF is an even function
about its mean, and θ is equal to the signal mean, then the entropy of Q is simply the
negative of the relative entropy between the signal and noise PDFs.
4. Examples of the PDF fQ(τ)
The PDF fQ(τ) can be derived for specific signal and noise distributions. Table I lists
fQ(τ) for several cases where the signal and noise share the same distribution and a mean
of zero, but with not necessarily equal variances. The threshold value, θ, is also set to zero.
For each case considered, the standard deviation of the noise can be written as aση, where
a is a positive constant, and the standard deviation of the signal can be written aσx. We find
that fQ(τ) in each case is a function of a single parameter that we call the noise intensity,
σ = ση/σx. Given this, from Eq. (16), it is clear that the mutual information must be a
function only of the ratio, σ, so that it is invariant to a change in σx provided ση changes
by the same proportion. This fact is noted to be true for the Gaussian case in [8], and the
uniform case in [11], but here we have illustrated why.
We note however, that if θ is not equal to the signal mean, then fQ(τ) will depend on
the ratio θ
, as well as θ and σ, and therefore so will the mutual information.
Table I also lists the entropy of Q for three cases where an analytical expression could be
found.
D. Large N SSR: Literature Review and Outline of This Paper
In the absence of noise, the maximum mutual information is the maximum entropy of the
output signal, log2 (N + 1). It has been shown for very specific signal and noise distributions
that the mutual information in the SSR model scales with 0.5 log2 (N) for large N [11, 21].
This means that the channel capacity for large N under the specified conditions is about
half the maximum noiseless channel capacity. This situation is discussed in Sec. III.
The only other work to consider SSR in the large N regime finds that the optimal noise
intensity for Gaussian signal and noise occurs for σ ≃ 0.6 [16]. Unlike [21]—which uses
the exact expression of Eq. (12), and derives a large N expression by approximating the
summation with an integral—[16] begins by using a Fisher information based approximation
to the mutual information.
In Appendix A1 we re-derive the formula of [16] in a different manner, which results
in new large N approximations for the output entropy, as well as the mutual information.
These approximations provide the basis for the central result of this paper, which is a
general sufficient condition for achieving channel capacity in the SSR model, for any arbitrary
specified signal or noise distribution. This is discussed in Section II. These new general
results are compared with the specific results of [11, 16, 21] in Sec. III.
II. A GENERAL EXPRESSION FOR THE SSR CHANNEL CAPACITY FOR
LARGE N
Fisher information [30, 31] has previously been discussed in numerous papers on both
neural coding [32] and stochastic resonance [33], and both [34, 35]. However, most SR
studies using Fisher information consider only the case where the signal itself is not a
random variable. When it is a random variable, it is possible to connect Fisher information
and Shannon mutual information under special conditions, as discussed in [16, 22, 34, 36].
It is demonstrated in [16] that the Fisher information at the output of the SSR model as
a function of input signal value X = x, is given by
J(x) =
dP1|x
P1|x(1− P1|x)
. (19)
In [16], Eq. (19) is used to approximate the large N mutual information in the SSR model
via the formula
I(X, y) = H(X)− 0.5
∫ x=∞
fX(x) log2
dx. (20)
This expression—which is derived under much more general circumstance in [22, 37]—relies
on an assumption that an efficient Gaussian estimator for x can be found from the output
of the channel, in the limit of large N .
In Appendix A1 we outline an alternative derivation to Eq. (20)—from which Eq. (19)
can be inferred—that is specific to the SSR model, and provides additional justification
for its large N asymptotic validity. This alternative derivation allows us to find individual
expressions for both the output entropy and conditional output entropy. This derivation
makes use of the auxiliary PDF, fQ(τ), derived in Sec. IC. The significance of this approach
is that it leads to our demonstration of the new results that the output entropy can be
written for large N as
H(y) ≃ log2 (N)−D(fX(x)||fη(θ − x)), (21)
while the mutual information can be written as
I(X, y) ≃ 0.5 log2
−D(fX ||fS), (22)
where fS(·) is a PDF known as Jeffrey’s prior,
fS(x) =
. (23)
It is proven in Appendix A2 that for the SSR model Eq. (23) is indeed a PDF. This is
a remarkable result, as in general Jeffrey’s prior has no such simple form. Substitution of
Eq. (23) into Eq. (22) and simplifying leads to Eq. (20), which verifies this result.
By inspection of Eq. (19), fS(x) can be derived from knowledge of the noise PDF, fη(η),
since
fS(x) =
fη(θ − x)
Fη(θ − x)(1− Fη(θ − x))
. (24)
A. A Sufficient Condition for Optimality
Since relative entropy is always non-negative, from Eq. (22) a sufficient condition for
achieving the large N channel capacity is that
fX(x) = fS(x) ∀ x, (25)
with the resultant capacity as
C(X, y) = 0.5 log2
≃ 0.5 log2N − 0.3956. (26)
Eq. (26) holds provided the conditions for the approximation given by Eq. (20) hold. Other-
wise, the RHSs of Eqs. (21) and (22) give lower bounds. This means that for the situations
considered previously in [16, 21] where the signal and noise both have the same distribution
(but different variances), we can expect to find channel capacity that is less than or equal
to that of Eq. (26). This is discussed in Sec. III.
The derived sufficient condition of Eq. (25) leads to two ways in which capacity can be
achieved, (i) an optimal signal PDF for a given noise PDF, and (ii) an optimal noise PDF
for a given signal PDF.
B. Optimizing the Signal Distribution
Assuming Eq. (20) holds, the channel capacity achieving input PDF, f oX(x), can be found
for any given noise PDF from Eqs. (24) and (25) as
f oX(x) =
fη(θ − x)
Fη(θ − x)(1− Fη(θ − x))
. (27)
1. Example: Uniform Noise
Suppose the iid noise at the input to each threshold device in the SSR model is uniformly
distributed on the interval [−ση/2, ση/2] so that it has PDF
fη(ξ) =
, ξ ∈ [−ση/2, ση/2]. (28)
Substituting Eq. (28) and its associated CDF into Eq. (27), we find that the optimal signal
PDF is
f oX(x) =
− (x− θ)2
, x ∈ [θ − ση/2, θ + ση/2]. (29)
This PDF is in fact the PDF of a sine-wave with uniformly random phase, amplitude ση/2,
and mean θ. A change of variable to the interval τ ∈ [0, 1] via the substitution τ =
(x − θ)/ση + 0.5 results in the PDF of the Beta distribution with parameters 0.5 and 0.5,
also known as the arcsine distribution. As mentioned in Sec. I, this result provides some
theoretical justification for the analog-to-digital conversion method proposed in [23].
This Beta distribution is bimodal, with the most probable values of the signal those near
zero and unity. Similar results for an optimal input distribution in an information theoretic
optimization of a neural system have been found in [38]. These results were achieved numer-
ically using the Blahut-Arimoto algorithm often used in information theory to find channel
capacity achieving source distributions, or rate-distortion functions [30].
2. Gaussian Noise
Suppose the iid noise at the input to each threshold device has a zero mean Gaussian
distribution with variance σ2η, with PDF
fη(ξ) =
2πσ2η
. (30)
Substituting Eq. (30) and its associated CDF into Eq. (27), gives the optimal signal PDF.
The resultant expression for f oX(x) does not simplify much, and contains the standard error
function, erf(·) [39].
We are able to verify that the resultant PDF has the correct shape via Fig. 8 in [16],
which presents the result of numerically optimizing the signal PDF, fX(x), for unity variance
zero mean Gaussian noise, θ = 0, and N = 10000. As with the work in [38], the numerical
optimization is achieved using the Blahut-Arimoto algorithm. It is remarked in [16] that
the optimal fX(x) is close to being Gaussian. This is illustrated by plotting both fX(x) and
a Gaussian PDF with nearly the same peak value as fX(x). It is straightforward to show
that a Gaussian with the same peak value as our analytical f oX(x) has variance 0.25π
2. If
the signal was indeed Gaussian, then we would have σ = 2/π ≃ 0.6366, which is very close
to the value calculated for actual Gaussian signal and noise in Sec. III.
Our analytical f oX(x) from Eqs. (30) and (27), with θ = 0, is plotted on the interval
x ∈ [−3, 3] in Fig. 2, along with a Gaussian PDF with variance 0.25π2. Clearly the optimal
signal PDF is very close to the Gaussian PDF. Our Fig. 2 is virtually identical to Fig. 8
in [16]. It is emphasized that the results in [16] were obtained using an entirely different
method that involves numerical iterations, and therefore provides excellent validation of our
theoretical results.
C. Optimizing the Noise Distribution
We now assume that the signal distribution is known and fixed. We wish to achieve
channel capacity by finding the optimal noise distribution. It is easy to show by integrating
Eq. (24) that the CDF corresponding to the PDF, fS(·), evaluated at x, can be written in
terms of the CDF of the noise distribution as
FS(x) = 1−
arcsin
Fη(θ − x)
. (31)
If we now let fX(x) = fS(x), then FX(s) = FS(x), and rearranging Eq. (31) gives the optimal
noise CDF in terms of the signal CDF as
F oη (x) = sin
(1− FX(θ − x))
= 0.5 + 0.5 cos (πFX(θ − x)). (32)
Differentiating F oη (x) gives the optimal noise PDF as a function of the signal PDF and CDF,
f oη (x) =
sin (π(1− FX(θ − x)))fX(θ − x). (33)
Unlike optimizing the signal distribution, which is the standard way for achieving channel
capacity in information theory [30], we have assumed a signal distribution, and found the
‘best’ noise distribution, which is equivalent to optimizing the channel, rather than the
signal.
1. Example: Uniform Signal
Suppose the signal is uniformly distributed on the interval x ∈ [−σx/2, σx/2]. From
Eqs. (32) and (33), the capacity achieving noise distribution has CDF
F oη (x) = 0.5 + 0.5 sin
π(x− θ)
, x ∈ [θ − σx/2, θ + σx/2] (34)
and PDF
f oη (x) =
π(x− θ)
, x ∈ [θ − σx/2, θ + σx/2]. (35)
Substitution of F oη (x) and f
η (x) into Eq. (19) finds the interesting result that the Fisher
information is constant for all x,
J(x) = N
. (36)
This is verified in Eq. (37) below.
D. Consequences of Optimizing the Large N Channel Capacity
1. Optimal Fisher Information
Regardless of whether we optimize the signal for given noise, or optimize the noise for a
given signal, it is straightforward to show that the Fisher information can be written as a
function of the signal PDF,
J(x) = Nπ2(fX(x))
2. (37)
Therefore, the Fisher information at large N channel capacity is constant for the support
of the signal iff the signal is uniformly distributed. The optimality of constant Fisher infor-
mation in a neural coding context is studied in [32].
2. The Optimal PDF fQ(τ)
A further consequence that holds in both cases is that the ratio of the signal PDF to the
noise PDF is
fX(x)
fη(θ − x)
π sin (π(1− FX(x)))
. (38)
This is not a PDF. However, if we make a change of variable via τ = 1 − Fη(θ − x) we get
the PDF fQ(τ) discussed in Sec. IC, which for channel capacity is
f oQ(τ) =
τ(1− τ)
, τ ∈ [0, 1]. (39)
This optimal fQ(τ) is in fact the PDF of the beta distribution with parameters 0.5 and 0.5,
i.e. the arcsine distribution. It is emphasised that this result holds regardless of whether
the signal PDF is optimised for a given noise PDF or vice versa.
3. Output Entropy at Channel Capacity
From Eq. (18), the entropy of Q is equal to the negative of the relative entropy between
fX(x) and fη(θ − x). The entropy of Q when capacity is achieved can be calculated from
Eq. (39) using direct integration as
H(Q) = log2 (π)− 2. (40)
From Eqs. (21) and (18), the large N output entropy at channel capacity in the SSR model
H(y) = log2
. (41)
4. The Optimal Output PMF is Beta-Binomial
Suppose we have signal and noise such that fQ(τ) = f
Q(τ)—i.e. the signal and noise
satisfy the sufficient condition, Eq. (25)—but that N is not necessarily large. We can derive
the output PMF for this situation, by substituting Eq. (39) into Eq. (17) to get
Py(n) =
τ (n−0.5)(1− τ)(N−n−0.5)dτ
) β(n+ 0.5, N − n + 0.5)
β(0.5, 0.5)
. (42)
where β(a, b) is a Beta function. This PMF can be recognized as that of the Beta-binomial—
or negative hypergeometric—distribution with parameters N, 0.5, 0.5 [40]. It is emphasized
that Eq. (42) holds as an exact analytical result for any N .
5. Analytical Expression for the Mutual Information
The exact expression for the output PMF of Eq. (42) allows exact calculation of both the
output entropy, and the mutual information without need for numerical integration, using
Eq. (16). This is because when fQ(τ) = f
Q(τ), the integrals in Eq. (16) can be evaluated
exactly to get
Io(X, y) =−
Py(n) log2
Py(n)
+N log2
. (43)
The exact values of Io(X, y) and the corresponding output entropy, Ho(y), are plotted in
Fig. 3(a) for N = 1, .., 1000. For comparison, the exact I(X, y) of Eq. (12), which holds for
fX(x) = fη(θ−x), is also plotted, as well as the corresponding entropy, H(y) = log2 (N + 1).
It is clear that Io(X, y) is always larger than the mutual information of the fX(x) = fη(θ−x)
case, and that Ho(y) is always less than its entropy, which is the maximum output entropy.
To illustrate that the large N expressions derived are lower bounds to the exact formula
plotted in Fig. 3(a), and that the error between them decreases with N , Fig. 3(b) shows the
difference between the exact and the large N mutual information and output entropy. This
difference clearly decreases with increasing N .
E. A Note on the Output Entropy
The SSR model has been described in terms of signal quantization theory in [28], and
compared with the related process of companding in [41]. In this context quantization means
the conversion of a continuously valued signal to a discretely valued signal that has only
a finite number of possible values. Quantization in this sense occurs in analog-to-digital
converter circuits, lossy compression algorithms, and in histogram formation [42]. For a
deterministic scalar quantizer with N +1 output states, N threshold values are required. In
quantization theory, there is a concept of high resolution quantizers, in which the distribution
of N → ∞ threshold values can be described by a point density function, λ(x). For such
quantizers, it can be shown that the quantizer output, y, in response to a random variable,
X , has entropy H(y) ≃ log2N − D(fX ||λ) [42]. This is strikingly similar to our Eq. (21)
for the large N output entropy of the SSR model. In fact, since the noise that perturbs
the fixed threshold value, θ, is additive, each threshold acts as an iid random variable with
PDF fη(θ − x), and therefore for large N , fη(θ − x) acts as a density function describing
the relative frequency of threshold values as a function of x, just as λ(x) does for a high
resolution deterministic quantizer.
For deterministic quantizers, the point density function can be used to approximate
the high resolution distortion incurred by the quantization process. For the SSR model
however, since the quantization has a random aspect, the distortion has a component due
to randomness as well as lossy compression, and cannot be simply calculated from fη(·).
Instead, one can use the Fisher information to calculate the asymptotic mean square error
distortion, which is not possible for deterministic high resolution quantizers.
III. CHANNEL CAPACITY FOR LARGE N AND ‘MATCHED’ SIGNAL AND
NOISE
Unlike the previous section, we now consider channel capacity under the constraint of
‘matched’ signal and noise distributions—i.e. where both the signal and noise, while still
independent, have the same distribution, other than their variances. The mean of both
signal and noise is zero and the threshold value is also θ = 0. In this situation the mutual
information depends solely on the ratio σ = ση/σx, which is the only free variable. Finding
channel capacity is therefore equivalent to finding the optimal value of noise intensity, σ.
Such an analysis provides verification of the more general capacity expression of Eq. (26),
which cannot be exceeded.
Furthermore, inspection of Eq. (A10) shows that the largeN approximation to the mutual
information consists of a term that depends on N and a term that depends only on σ. This
shows that for large N the channel capacity occurs for the same value of σ—which we denote
as σo—for all N .
This fact is recognized in both [21] for uniform signal and noise—where σo → 1—and [16],
for Gaussian signal and noise. Here, we investigate the value of σo and the mutual infor-
mation at σo for other signal and noise distributions, and compare the channel capacity
obtained with the case where fX(x) = fS(x). This comparison finds that the results of [16]
overstates the true capacity, and that large N results in [11, 21] need to be improved to be
consistent with the central results of this paper.
From Eq. (22), channel capacity for large N occurs for the value of σ that minimizes the
relative entropy between fX and fS. If we let
f(σ) =
∫ x=∞
fX(x) ln
dx, (44)
then from Eq. (20), it is also clear that this minimization is equivalent to solving the following
problem,
σo = min
f(σ). (45)
This is exactly the formulation stated in [16]. Problem (45) can be equivalently expressed
σo = min
f(σ) = D(fX ||fη) +
∫ x=∞
fX(x) log2 (P1|x)dx
, (46)
where we have assumed that both the signal and noise PDFs are even functions. The
function f(σ) can be found for any specified signal and noise distribution by numerical
integration, and Problem (46) easily solved numerically. If an exact expression for the
relative entropy term is known, then only g(σ) =
∫ x=∞
fX(x) log2 (P1|x)dx needs to be
numerically calculated.
Table II gives the result of numerically calculating the value of σo, and the corresponding
large N channel capacity, C(X, y), for a number of distributions. In each case, C(X, y) −
0.5 log2 (N) < −0.3956, as required by Eq. (26). The difference between capacity and
0.5 log2 (N) is about 0.4 bits per sample. In the limit of large N , this shows that capacity
is almost identical, regardless of the distribution. However, the value of σo at which this
capacity occurs is different in each case.
As discussed in Sec. I B, the mutual information is identical whenever the signal and noise
PDFs are identical, i.e. σ = 1. It is shown below in Eq. (48) that for large N the mutual
information at σ = 1 is I(X, y) = 0.5 log2 (N)− 0.6444. Given that the channel capacity is
slightly larger than this, as indicated by Table II, for each case there is a constant difference
between the channel capacity and the mutual information at σ = 1. This value is also listed
in Table II.
A. Improvements to Previous Large N Approximations
We now use the results of Sec. II to show that previous large N expressions for the mutual
information in the literature for the σ = 1, Gaussian and uniform cases can be improved.
1. SSR for Large N and σ = 1
We now consider the situation where fX(x) = fη(x), so that σ = 1. It is shown in [11]
that in this case as N approaches infinity, Eq. (12) reduces to
I(X, y) ≃ 0.5 log2
N + 1
≃ 0.5 log2 (N + 1)− 0.7213. (47)
To show that this expression can be improved, we begin with the version of Eq. (20) given
by Eq. (A10). When σ = 1 we have fQ(τ) = 1 and H(Q) = 0. The integrals in Eq. (A10)
can be solved to give the large N mutual information at σ = 1 as
I(X, y) ≃ 0.5 log2
≃ 0.5 log2N − 0.6044. (48)
Although Eqs. (47) and (48) agree as N → ∞, the constant terms do not agree. It is shown
in Appendix A3 that the discrepancy can be resolved by improving on the approximation
to the average conditional entropy, H(y|X), made in [11]. The output entropy at σ = 1
can be shown to be simply H(y) = log2 (N + 1) [11]. Subtracting Eq. (A18) from H(y) and
letting N approach infinity gives
I(X, y) ≃ 0.5 log2
(N + 2)e
, (49)
which does have a constant term which agrees with Eq. (48). The explanation of the dis-
crepancy is that [11] uses the Euler-Maclaurin summation formula to implicitly calculate
log2 (N !) in the large N approximation to H(y|X). Using Stirling’s approximation for N !,
as done here, gives a more accurate approximation.
The increased accuracy of Eq. (48) can be verified by numerically comparing both Eq. (48)
and Eq. (47) with the exact expression for I(X, y) of Eq. (12), as N increases. The error
between the exact expression and Eq. (48) approaches zero as N increases, whereas the error
between Eq. (12) and Eq. (47) approaches a nonzero constant for large N of 0.5 log2
0.117 bits per sample.
2. Uniform Signal and Noise
A derivation is given in [21] of an exact expression for I(X, y) for uniform signal and
noise and σ ≤ 1. In addition, [21] finds a large N approximation to the mutual information.
Using the same arguments as for the σ = 1 case, this approximation can be improved to
I(X, y) ≃ σ
(N + 2)e
+ (1− σ)(1− log2 (1− σ))− σ log2 (σ). (50)
The accuracy of Eq. (50) can be verified by numerical comparison with the exact formula
in [21], as N increases. If one replicates Fig. 3 of [21] in this manner, it is clear that Eq. (50)
is the more accurate approximation.
Differentiating Eq. (50) with respect to σ and setting to zero obtains the optimal value
of σ as
(N + 2)
(N + 2) +
. (51)
The channel capacity at σo is
C(X, y) = 1− log2 (1− σo) = log2
(N + 2)e
. (52)
Clearly, limN→∞ σo = 1, and the capacity approaches 0.5 log2 ((N + 2)e/(2π)), which agrees
with Eq. (49). Expressions for σo and the corresponding capacity for large N are also given
in [21]. Again, these are slightly different to Eqs. (51) and (52), due to the slightly inaccurate
terms in the large N approximation to H(y|X). However the important qualitative result
remains the same, which is that the channel capacity scales with 0.5 log2 (N) and the value
of σ which achieves this asymptotically approaches unity.
3. Gaussian Signal and Noise
In [16], an analytical approximation for σo for the specific case of Gaussian signal and
noise is derived using a Taylor expansion of the Fisher information inside the integral in
Eq. (20). We give a slightly different derivation of this approach that uses the PDF fQ(τ).
We begin with Problem (46). Solving this problem requires differentiating f(σ) with
respect to σ and solving for zero. From Table I, the derivative of the relative entropy
between fX and fη is
D(fX ||fη) =
σ−1 − σ−3
. (53)
For the second term, g(σ), we take the lead from [16] and approximate ln (P1|x) by its second
order Taylor series expansion [39]. The result is that
g(σ) = −
∫ x=∞
fX(x) log2 (P1|x)dx ≃ 1 +
πσ2 ln 2
. (54)
Numerical testing finds that the approximation of Eq. (54) appears to be quite accurate for
all σ, as the relative error is no more than about 10 percent for σ > 0.2. However, as we will
see, this is inaccurate enough to cause the end result for the approximate channel capacity
to significantly overstate the true channel capacity.
Taking the derivative of Eq. (54) with respect to σ, subtracting it from Eq. (53), setting
the result to zero and solving for σ gives the optimal value of σ found in [16], σo ≃
0.6028.
An expression for the mutual information at σo can be found by back-substitution. Car-
rying this out gives the large N channel capacity for Gaussian signal and noise as
C(X, y) ≃ 0.5 log2
e(π − 2)
, (55)
which can be written as C(X, y) ≃ 0.5 log2N − 0.3169.
Although Eq. (55) is close to correct, recall from Sec. II that capacity must be less than
0.5 log2N − 0.3956 and hence Eq. (55) significantly overstates the true capacity.
Acknowledgments
This work was funded by the Australian Research Council, an Australian Academy of
Science Young Researcher’s Award, as well as EPSRC grant EP/C523334/1, and we grate-
fully acknowledge this support. The authors would also like to thank Priscilla Greenwood, of
Arizona State University, Pierre-Olivier Amblard of Laboratoire des Images et des Signaux,
France, Thinh Nguyen of Oregon State University and Simon Durrant of the University
of Plymouth, UK, for valuable discussions and generous provision of preprints, and the
anonymous referees, whose comments have led to significant improvements in this paper.
APPENDIX A: DERIVATIONS
1. Mutual Information for Large N and Arbitrary σ
This appendix contains derivations of the large N approximations to the output entropy
and mutual information discussed in Sec. II.
a. Conditional Output Entropy
An approximation to the conditional output entropy, H(y|X), can be derived by noting
that for large N the binomial distribution can be approximated by a Gaussian distribution
with the same mean and variance—i.e. NP1|x and NP1|x(1 − P1|x) respectively. Provided
0≪NP1|x≪N we have
Py|X(n|x) ≃
2πNP1|x(1− P1|x)
(n−NP1|x)2
2NP1|x(1− P1|x)
. (A1)
The average conditional output entropy is H(y|X) =
fX(x)Ĥ(y|x)dx, where
Ĥ(y|x) = −
Py|X(n|x) log2 (Py|X(n|x)). (A2)
Using the well known result for the entropy of a Gaussian random variable [30] we can write
Ĥ(y|x) ≃ 0.5 log2 (2πeNP1|x(1− P1|x)). (A3)
Multiplying both sides of Eq. (A3) by fX(x) and integrating over all x gives
H(y|X) ≃ 0.5 log2 (2πeN) + 0.5
fX(x) log2
P1|x(1− P1|x)
= 0.5 log2 (2πeN) + 0.5
∫ τ=1
fQ(τ) log2 (τ)dτ+
∫ τ=1
fQ(τ) log2 (1− τ)dτ. (A4)
Eq. (A4) can be verified for the case where fX(x) = fη(θ − x), since this means fQ(τ) =
1 and
∫ τ=1
fQ(τ) log2 (τ)dτ = − log2 (e). Consequently Eq. (A4) reduces to H(y|X) ≃
0.5 log2
which agrees precisely with Eq. (A19). This approximation breaks down
when P1|x is close to zero or unity. Furthermore, Eq. (A4) holds exactly only for values
of x for which Py|X(n|x) is exactly Gaussian. Otherwise, H(y|X) is strictly less than the
approximation given.
b. Output Distribution and entropy
For large N , since Py|X(n|x) is Gaussian, y/N approaches a delta function located at
P1|x = n/N . From Eqs. (7) and (17), this means that Py(n) can be written in terms of the
PDF of the average transfer function, fQ(·), as
Py(n) ≃
. (A5)
This result can be derived more rigorously using saddlepoint methods [22].
Consider the case where the signal and noise both have the same distribution but different
variances. When the noise intensity, σ > 1, then fQ(0) = fQ(1) = 0, whereas for σ < 1,
we have fQ(0) = fQ(1) = ∞. From Eq. (A5), this means Py(0) and Py(N) are either
zero or infinite. However, for finite N , there is some finite nonzero probability that all
output states are on or off. Indeed, at σ = 1, we know that Py(n) =
∀ n, and at
σ = 0, Py(0) = Py(N) = 0.5. Furthermore, for finite N , Eq. (A5) does not guarantee that
n=0 Py(n) = 1. To increase the accuracy of our approximation by ensuring Py(0) and
Py(N) are always finite, and that Py(n) forms a valid PMF, we define a new approximation
P ′y(n) =
fQ( nN )
for n = 1, .., N − 1
fQ(mN )
for n = 0, n = N.
Fig. 4 shows that the approximation given by P ′y(n) is highly accurate for N as small as 63,
for σ both smaller and larger than unity.
Consider the entropy of the discrete random variable y. Making use of Eq. (A5), we have
H(y) = −
Py(n) log2 (Py(n))
= − 1
log2 (N)
. (A7)
Suppose that the summations above can be approximated by integrals, without any
remainder terms. Carrying this out and then making the change of variable τ = n/N gives
H(y) ≃ log2N −
∫ τ=1
fQ(τ) log2 (fQ(τ))dτ
= log2N +H(Q), (A8)
where H(Q) is the differential entropy of the random variable Q. Performing a change of
variable in Eq. (A8) of τ = 1− Fη(θ − x) gives
H(y) ≃ log2 (N)−D(fX(x)||fη(θ − x)). (A9)
This result shows that H(y) for large N is approximately the sum of the number of output
bits and the negative of the relative entropy between fX and fη. Therefore, since relative
entropy is always non-negative, the approximation to H(y) given by Eq. (A9) is always less
than or equal to log2 (N). This agrees with the known expression for H(y) in the specific
case of σ = 1 of log2 (N + 1), which holds for any N .
c. Mutual Information
Subtracting Eq. (A4) from Eq. (A8) gives a large N approximation to the mutual infor-
mation as
I(X, y) ≃ 0.5 log2
+H(Q)
− 0.5
∫ τ=1
fQ(τ) log2 (τ(1− τ))dτ. (A10)
As discussed in the main text, the mutual information scales with 0.5 log2 (N). The im-
portance of the N -independent terms in Eq. (A10) is that they determine how the mutual
information varies from 0.5 log2
for different PDFs, fQ(τ).
Fig. 5 shows, as examples, the approximation of Eq. (A10), as well as the exact mutual
information—calculated by numerical integration—for the Gaussian and Laplacian cases,
for a range of σ and increasing N . As with the output entropy, the mutual information
approximation is quite good for σ > 0.7, but worsens for smaller σ. However, as N increases
the approximation improves.
Eq. (A10) can be rewritten via the change of variable, x = θ − F−1η (1− τ), as
I(X, y) = 0.5 log2
∫ x=∞
fX(x) log2 (P1|x(1− P1|x))dx−D(fX(x)||fη(θ − x)). (A11)
Rearranging Eq. (A11) gives Eq. (20)—with the Fisher information, J(x), given by
Eq. (19)—which is precisely the same as that derived in [16] as an asymptotic large N
expression for the mutual information. Our analysis extends [16] by finding large N ap-
proximations to both H(y) and H(y|X), as well as the output distribution, Py(n). We have
also illustrated the role of the PDF, fQ(τ), in these approximations, and justified the use of
Eq. (20) for the SSR model.
2. Proof that fS(x) is a PDF
As shown in [16], the Fisher information for the SSR model is given by Eq. (19). Consider
fS(x) as in Eq. (24). Since fη(x) is a PDF and Fη(x) is the CDF of η evaluated at x, we
have fS(x) ≥ 0 ∀ x. Letting h(x) = Fη(θ − x), Eq. (24) can be written as
fS(x) =
−h′(x)
h(x)− h(x)2
. (A12)
Suppose fη(x) has support x ∈ [−a, a]. Integrating fS(x) over all x gives
∫ x=a
fS(x)dx =
∫ x=a
−h′(x)
h(x)− h(x)2
2 arcsin
|x=ax=−a
(arcsin(0)− arcsin(1)) = 1, (A13)
which means fS(x) is a PDF.
3. H(y|X) for large N and σ = 1
Here we derive a large N approximation to H(y|X) used in Sec. IIIA 1. For σ = 1 the
output PMF is Py(n) =
∀ n [11]. Using this, it can be shown that
Py(n) log2
= log2 (N !)−
N + 1
n log2 n. (A14)
We will now see that both terms of Eq. (A14) can be simplified by approximations that hold
for large N . Firstly, for the log2 (N !) term, we can make use of Stirling’s formula [39], which
is valid for large N ,
N ! ∼
(2πN)NN exp (−N). (A15)
This approximation is particularly accurate if the log is taken of both sides, which we require.
Secondly, the sum in the second term of Eq. (A14) can be approximated by an integral and
simplified by way of the Euler-Maclaurin summation formula [39]. The result is
N + 1
n log2 n ≃ N log2 (N + 1)−
N(N + 2)
2 ln 2(N + 1)
. (A16)
Subtracting Eq. (A16) from the log of Eq. (A15) gives
Py(n) log2
≃ 0.5 log2
2 ln 2
N + 2
N + 1
+ 0.5 log2 (2π)− O
, (A17)
where we have used N log2 (1 +
) = 1
. When Eq. (A17) is substituted into an
exact expression for H(y|X) given in [11], we get
H(y|X) = N
2 ln 2
Py(n) log2
≃ 0.5 log2N + 0.5
N + 1
log2 (e)+
0.5 log2 (2π)−O
. (A18)
The final result is that for large N ,
H(y|X) ≃ 0.5 log2
. (A19)
[1] L. Gammaitoni, P. Hänggi, P. Jung, and F. Marchesoni, Reviews of Modern Physics 70, 223
(1998).
[2] F. Moss, L. M. Ward, and W. G. Sannita, Clinical Neurophysiology 115, 267 (2004).
[3] K. Wiesenfeld, D. Pierson, E. Pantazelou, C. Dames, and F. Moss, Phys. Rev. Lett. 72, 2125
(1994).
[4] A. R. Bulsara and A. Zador, Phys. Rev. E 54, R2185 (1996).
[5] F. Chapeau-Blondeau, Phys. Rev. E 53, 5469 (1996).
[6] P. E. Greenwood, L. M. Ward, and W. Wefelmeyer, Phys. Rev. E 60, 4687 (1999).
[7] M. DeWeese and W. Bialek, Nuovo Cimento 17, 733 (1995).
[8] N. G. Stocks, Phys. Rev. Lett. 84, 2310 (2000).
[9] J. J. Collins, C. C. Chow, and T. T. Imhoff, Phys. Rev. E 52, R3321 (1995).
[10] R. P. Morse and P. Roper, Phys. Rev. E 61, 5683 (2000).
[11] N. G. Stocks, Phys. Rev. E 63, 041114 (2001).
[12] M. D. McDonnell, N. G. Stocks, C. E. M. Pearce, and D. Abbott, Phys. Lett. A 352, 183
(2006).
[13] N. G. Stocks, D. Allingham, and R. P. Morse, Fluct. Noise Lett. 2, L169 (2002).
[14] W. Bialek, M. DeWeese, F. Rieke, and D. Warland, Physica A 200, 581 (1993).
[15] J. F. Lindner, M. Bennett, and K. Wiesenfeld, Phys. Rev. E 72, 051911 (2005).
[16] T. Hoch, G. Wenning, and K. Obermayer, Phys. Rev. E 68, 011911 (2003).
[17] S. Durrant and J. Feng (2007), In Preparation.
[18] A. Nikitin, N. G. Stocks, and R. P. Morse, Phys. Rev. E 75, 021121 (2007).
[19] N. G. Stocks and R. Mannella, Phys. Rev. E 64, 030902(R) (2001).
[20] R. P. Morse, P. F. Morse, T. B. Nunn, K. A. M. Archer, and P. Boyle, JARO 8, 42 (2007).
[21] N. G. Stocks, Phys. Lett. A 279, 308 (2001).
[22] N. Brunel and J. Nadal, Neural Computation 10, 1731 (1998).
[23] T. Nguyen (2007), accepted for publication in IEEE Transactions on Signal Processing.
[24] M. D. McDonnell, D. Abbott, and C. E. M. Pearce, Microelectronics Journal 33, 1079 (2002).
[25] M. D. McDonnell, D. Abbott, and C. E. M. Pearce, Fluct. Noise Lett. 2, L205 (2002).
[26] D. Rousseau, F. Duan, and F. Chapeau-Blondeau, Phys. Rev. E 68, 031107 (2003).
[27] D. Rousseau and F. Chapeau-Blondeau, Phys. Lett. A 321, 280 (2004).
[28] M. D. McDonnell, N. G. Stocks, C. E. M. Pearce, and D. Abbott, Fluct. Noise Lett. 5, L457
(2005).
[29] Y. Wang and L. Wu, Fluct. Noise Lett. 5, L435 (2005).
[30] T. M. Cover and J. A. Thomas, Elements of Information Theory (John Wiley and Sons, New
York, 1991).
[31] E. L. Lehmann and G. Casella, Theory of Point Estimation (Springer, New York, 1998).
[32] M. Bethge, D. Rotermund, and K. Pawelzik, Neural Comput. 14, 2317 (2002).
[33] P. E. Greenwood, U. U. Müller, and L. M. Ward, Phys. Rev. E 70, 051110 (2004).
[34] M. Stemmler, Network: Comput. in Neural Syst. 7, 687 (1996).
[35] P. E. Greenwood, L. M. Ward, D. F. Russell, A. Neiman, and F. Moss, Phys. Rev. Lett. 84,
4773 (2000).
[36] K. Kang and H. Sompolinsky, Phys. Rev. Lett. 86, 4958 (2001).
[37] B. S. Clarke and A. R. Barron, IEEE Transactions on Information Theory 36, 453 (1990).
[38] S. Schreiber, C. K. Machens, A. V. Herz, and S. B. Laughlin, Neural Computation 14, 1323
(2002).
[39] M. R. Spiegel and J. Liu, Mathematical Handbook of Formulas and Tables (McGraw-Hill,
1999).
[40] M. Evans, N. Hastings, and B. Peacock, Statistical Distributions (John Wiley and Sons, 2000),
3rd ed.
[41] P.-O. Amblard, S. Zozor, and O. J. J. Michel, in Proc. 2006 IEEE International Conference
on Acoustics, Speech, and Signal Processing (2006), vol. 3, pp. 716–719.
[42] A. Gersho and R. M. Gray, Vector Quantization and Signal Compression (Kluwer Academic
Publishers, 1992).
FIG. 1: The SSR model consists of N parallel threshold devices, each with the same threshold
value, θ. The common input signal is a continuously valued random signal, X, consisting of a
sequence of discrete time uncorrelated samples. Each device receives independently noisy versions
of X. The noise signals, ηi, are iid additive random signals that are independent of X. The output
from the i–th device, yi, is unity if X + ηi > θ and zero otherwise. The overall output, y, is the
sum of the individual outputs, yi.
−3 −2 −1 0 1 2 3
N(0,0.25π2)
FIG. 2: The optimal signal PDF, f oX(x), for zero mean, unity variance Gaussian noise, and thresh-
old value θ = 0, as obtained from Eq. (27). Superimposed is a Gaussian PDF with the same peak
value as f oX(x), so that it has variance 0.25π
2. This figure uses our new theoretical results to
analytically replicate Fig. 8 in [16], which was calculated numerically.
200 400 600 800 1000
(X,y)
I(X,y), σ=1
H(y), σ=1
(a) Exact Expressions
200 400 600 800 1000
(X,y)
I(X,y), σ=1
(b) Error in large N expressions
FIG. 3: (a) Exact expressions obtained using f oQ(τ), for Io(X, y), and Ho(y), as well as the exact
mutual information and output entropy when fX(x) = fη(θ− x) (denoted as σ = 1), as a function
of N . (b) The difference between the exact expressions for Io(X, y), Ho(y) and I(X, y) for fX(x) =
fη(θ − x), and the corresponding large N expressions given by Eqs. (22), (41) and (49).
0 10 20 30 40 50 60
Exact
Approx
(a) σ = 0.4, Gaussian signal and noise
0 10 20 30 40 50 60
0.005
0.015
0.025
Exact
Approx
(b) σ = 1.6, Gaussian signal and noise
FIG. 4: Approximation to the output PMF, Py(n), given by Eq. (A6), for N = 63. Circles indicate
the exact Py(n) obtained by numerical integration and the crosses show approximations.
0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6
N=511
N=255
N=127
(a) Gaussian
0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6
N=511
N=255
N=127
(b) Laplacian
FIG. 5: Large N approximation to mutual information given by Eq. (A10) and exact mutual
information calculated numerically. The exact expression is shown by thin solid lines, and the
approximation by circles, with a thicker solid line interpolating between values of σ as an aid
to the eye. The approximation can be seen to always be a lower bound on the exact mutual
information.
TABLE I: The auxiliary PDF, fQ(τ), for five different ‘matched’ signal and noise distributions
(i.e. same distribution but with different variances), as well as H(Q), the entropy of fQ(τ). The
threshold value, θ, and the signal and noise means are assumed to be zero, so that these results are
independent of θ. The noise intensity, σ = ση/σx, is the ratio of the noise standard deviation to the
signal standard deviation. For the Cauchy case, σλ is the ratio of the full-width-at-half-maximum
parameters. The label ‘NAS’ indicates that there is no analytical solution for the entropy.
Distribution fQ(τ) H(Q)
Gaussian σ exp
(1− σ2)
erf−1(2τ − 1)
− log2 (σ)− 12 ln 2
Uniform, σ ≥ 1
σ, − 1
+ 0.5 ≤ τ ≤ 1
+ 0.5,
0, otherwise.
log2 σ
Laplacian
σ(2τ)(σ−1) for 0 ≤ τ ≤ 0.5,
σ(2(1 − τ))(σ−1) for 0.5 ≤ τ ≤ 1.
− log2 (σ)− 12 ln 2
Logistic σ
(τ(1−τ))(σ−1)
(τσ+(1−τ)σ)2
Cauchy σλ
1+tan2 (π(τ−0.5))
(1+σ2
tan2 (π(τ−0.5)))
TABLE II: Large N channel capacity and optimal σ for ‘matched’ signal and noise
Distribution C(X, y)− 0.5 log2 (N) σo C(X, y)− I(X, y)|σ=1
Gaussian −0.3964 0.6563 0.208
Logistic −0.3996 0.5943 0.205
Laplacian −0.3990 0.5384 0.205
	Introduction
	Information Theoretic Definitions
	SSR Model
	Describing SSR Using a Single PDF, fQ()
	fQ() as the PDF of the Average Transfer Function
	Mutual Information in Terms of fQ()
	Entropy of the random variable, Q
	Examples of the PDF fQ()
	Large N SSR: Literature Review and Outline of This Paper
	A General Expression for the SSR Channel Capacity for Large N
	A Sufficient Condition for Optimality
	Optimizing the Signal Distribution
	Example: Uniform Noise
	Gaussian Noise
	Optimizing the Noise Distribution
	Example: Uniform Signal
	Consequences of Optimizing the Large N Channel Capacity
	Optimal Fisher Information
	The Optimal PDF fQ()
	Output Entropy at Channel Capacity
	The Optimal Output PMF is Beta-Binomial
	Analytical Expression for the Mutual Information
	A Note on the Output Entropy
	Channel Capacity for Large N and `Matched' Signal and Noise
	Improvements to Previous Large N Approximations
	SSR for Large N and =1
	Uniform Signal and Noise
	Gaussian Signal and Noise
	Acknowledgments
	Derivations
	Mutual Information for Large N and Arbitrary 
	Conditional Output Entropy
	Output Distribution and entropy
	Mutual Information
	Proof that fS(x) is a PDF
	H(y|X) for large N and =1
	References
