When the Cramer-Rao Inequality provides no information
WHEN THE CRAMÉR-RAO INEQUALITY PROVIDES NO INFORMATION
STEVEN J. MILLER
Abstract. We investigate a one-parameter family of probability densities (related to the
Pareto distribution, which describes many natural phenomena) where the Cramér-Rao inequal-
ity provides no information.
1. Cramér-Rao Inequality
One of the most important problems in statistics is estimating a population parameter from a
finite sample. As there are often many different estimators, it is desirable to be able to compare
them and say in what sense one estimator is better than another. One common approach is to
take the unbiased estimator with smaller variance. For example, if X1, . . . , Xn are independent
random variables uniformly distributed on [0, θ], Yn = maxi Xi and X = (X1 + · · ·+ Xn)/n, then
Yn and 2X are both unbiased estimators of θ but the former has smaller variance than the
latter and therefore provides a tighter estimate.
Two natural questions are (1) which estimator has the minimum variance, and (2) what bounds
are available on the variance of an unbiased estimator? The first question is very hard to solve
in general. Progress towards its solution is given by the Cramér-Rao inequality, which provides
a lower bound for the variance of an unbiased estimator (and thus if we find an estimator that
achieves this, we can conclude that we have a minimum variance unbiased estimator).
Date: February 5, 2008.
2000 Mathematics Subject Classification. 62B10 (primary), 62F12, 60E05 (secondary).
Key words and phrases. Cramér-Rao Inequality, Pareto distribution, power law.
The author would like to thank Alan Landman for many enlightening conversations and the referees for helpful
comments. The author was partly supported by NSF grant DMS0600848.
http://arXiv.org/abs/0704.0923v1
2 STEVEN J. MILLER
Cramér-Rao Inequality: Let f(x; θ) be a probability density function with continuous parameter
θ. Let X1, . . . , Xn be independent random variables with density f(x; θ), and let Θ̂(X1, . . . , Xn)
be an unbiased estimator of θ. Assume that f(x; θ) satisfies two conditions:
(1) we have
· · ·
Θ̂(x1, . . . , xn)
f(xi; θ)dxi
· · ·
Θ̂(x1, . . . , xn)
i=1 f(xi; θ)
dx1 · · ·dxn;
(1.1)
(2) for each θ, the variance of Θ̂(X1, . . . , Xn) is finite.
var(Θ̂) ≥
∂ log f(x;θ)
)2], (1.2)
where E denotes the expected value with respect to the probability density function f(x; θ).
For a proof, see for example [CaBe]. The expected value in (1.2) is called the information
number or the Fisher information of the sample.
As variances are non-negative, the Cramér-Rao inequality (equation (1.2)) provides no useful
bounds on the variance of an unbiased estimator if the information is infinite, as in this case we
obtain the trivial bound that the variance is greater than or equal to zero. We find a simple
one-parameter family of probability density functions (related to the Pareto distribution) that
satisfy the conditions of the Cramér-Rao inequality, but the expectation (i.e., the information) is
infinite. Explicitly, our main result is
Theorem: Let
f(x; θ) =
−θ log−3 x if x ≥ e
0 otherwise,
(1.3)
WHEN THE CRAMÉR-RAO INEQUALITY PROVIDES NO INFORMATION 3
where aθ is chosen so that f(x; θ) is a probability density function. The information is infinite
when θ = 1. Equivalently, the Cramér-Rao inequality yields the trivial (and useless) bound that
Var(Θ̂) ≥ 0 for any unbiased estimator Θ̂ of θ when θ = 1.
In §2 we analyze the density in our theorem in great detail, deriving needed results about aθ
and its derivatives as well as discussing how f(x; θ) is related to important distributions used to
model many natural phenomena. We show the information is infinite when θ = 1 in §3, which
proves our theorem. We also discuss there properties of estimators for θ. While it is not clear
whether or not this distribution has an unbiased estimator, there is (at least for θ close to 1) an
asymptotically unbiased estimator rapidly converging to θ as the sample size tends to infinity. By
examining the proof of the Cramér-Rao inequality we see that we may weaken the assumption of
an unbiased estimator. While typically there is a cost in such a generalization, as our information
is infinite there is no cost in our case. We may therefore conclude that arguments such as those
used to prove the Cramér-Rao inequality cannot provide any information for estimators of θ from
this distribution.
2. An Almost Pareto Density
Consider
f(x; θ) =
aθ/(x
θ log3 x) if x ≥ e
0 otherwise,
(2.1)
where aθ is chosen so that f(x; θ) is a probability density function. Thus
xθ log3 x
= 1. (2.2)
We chose to have log3 x in the denominator to ensure that the above integral converges, as does
log x times the integrand; however, the expected value (in the expectation in (1.2)) will not
converge.
4 STEVEN J. MILLER
For example, 1/x logx diverges (its integral looks like log log x) but 1/x log2 x converges (its
integral looks like 1/ logx); see pages 62–63 of [Rud] for more on close sequences where one
converges but the other does not. This distribution is close to the Pareto distribution (or a power
law). Pareto distributions are very useful in describing many natural phenomena; see for example
[DM, Ne, NM]. The inclusion of the factor of log−3 x allows us to have the exponent of x in the
density function equal 1 and have the density function defined for arbitrarily large x; it is also
needed in order to apply the Dominated Convergence Theorem to justify some of the arguments
below. If we remove the logarithmic factors then we obtain a probability distribution only if the
density vanishes for large x. As log
x is a very slowly varying function, our distribution f(x; θ)
may be of use in modeling data from an unbounded distribution where one wants to allow a
power law with exponent 1, but cannot as the resulting probability integral would diverge. Such
a situation occurs frequently in the Benford Law literature; see [Hi, Rai] for more details.
We study the variance bounds for unbiased estimators Θ̂ of θ, and in particular we show that
when θ = 1 then the Cramér-Rao inequality yields a useless bound.
Note that it is not uncommon for the variance of an unbiased estimator to depend on the
value of the parameter being estimated. For example, consider again the uniform distribution on
[0, θ]. Let X denote the sample mean of n independent observations, and Yn = max1≤i≤n Xi be
the largest observation. The expected value of 2X and n+1
Yn are both θ (implying each is an
unbiased estimator for θ); however, Var(2X) = θ2/3n and Var(n+1
Yn) = θ
2/n(n+1) both depend
on θ, the parameter being estimated (see, for example, page 324 of [MM] for these calculations).
Lemma 2.1. As a function of θ ∈ [1,∞), aθ is a strictly increasing function and a1 = 2. It has
a one-sided derivative at θ = 1, and daθ
∈ (0,∞).
Proof. We have
xθ log3 x
= 1. (2.3)
WHEN THE CRAMÉR-RAO INEQUALITY PROVIDES NO INFORMATION 5
When θ = 1 we have
x log3 x
, (2.4)
which is clearly positive and finite. In fact, a1 = 2 because the integral is
x log3 x
log−3 x
d log x
2 log2 x
∣∣∣∣∣
; (2.5)
though all we need below is that a1 is finite and non-zero, we have chosen to start integrating at
e to make a1 easy to compute.
It is clear that aθ is strictly increasing with θ, as the integral in (2.4) is strictly decreasing with
increasing θ (because the integrand is decreasing with increasing θ).
We are left with determining the one-sided derivative of aθ at θ = 1, as the derivative at any
other point is handled similarly (but with easier convergence arguments). It is technically easier
to study the derivative of 1/aθ, as
(2.6)
xθ log
. (2.7)
The reason we consider the derivative of 1/aθ is that this avoids having to take the derivative of
the reciprocals of integrals. As a1 is finite and non-zero, it is easy to pass to
|θ=1. Thus we
= lim
x1+h log3 x
x log3 x
= lim
1 − xh
x log3 x
. (2.8)
We want to interchange the integration with respect to x and the limit with respect to h above.
This interchange is permissible by the Dominated Convergence Theorem (see Appendix A for
details of the justification). Note
1 − xh
= − log x; (2.9)
6 STEVEN J. MILLER
one way to see this is to use the limit of a product is the product of the limits, and then use
L’Hospital’s rule, writing xh as eh log x. Therefore
x log2 x
; (2.10)
as this is finite and non-zero, this completes the proof and shows daθ
|θ=1 ∈ (0,∞). �
Remark 2.2. We see now why we chose f(x; θ) = aθ/x
θ log3 x instead of f(x; θ) = aθ/x
θ log2 x.
If we only had two factors of log x in the denominator, then the one-sided derivative of aθ at θ = 1
would be infinite.
Remark 2.3. Though the actual value of daθ
|θ=1 does not matter, we can compute it quite
easily. By (2.10) we have
x log
d log x
log x
= −1. (2.11)
Thus by (2.6), and the fact that a1 = 2 (Lemma 2.1), we have
= −a21 ·
= 4. (2.12)
3. Computing the Information
We now compute the expected value, E
∂ log f(x;θ)
; showing it is infinite when θ = 1
completes the proof of our main result. Note
log f(x; θ) = log aθ − θ log x + log log
∂ log f(x; θ)
− log x. (3.1)
WHEN THE CRAMÉR-RAO INEQUALITY PROVIDES NO INFORMATION 7
By Lemma 2.1 we know that daθ
is finite for each θ ≥ 1. Thus
∂ log f(x; θ)
− log x
− log x
xθ log3 x
. (3.2)
If θ > 1 then the expectation is finite and non-zero. We are left with the interesting case when
θ = 1. As daθ
|θ=1 is finite and non-zero, for x sufficiently large (say x ≥ x1 for some x1, though
by Remark 2.3 we see that we may take any x1 ≥ e
4) we have
∣∣∣∣ ≤
log x
. (3.3)
As a1 = 2, we have
∂ log f(x; θ)
)2] ∣∣∣∣∣
log x
x log
2x logx
log−1 x
d log x
log log x
= ∞. (3.4)
Thus the expectation is infinite. Let Θ̂ be any unbiased estimator of θ. If θ = 1 then the
Cramér-Rao inequality gives
var(Θ̂) ≥ 0, (3.5)
which provides no information as variances are always non-negative. This completes the proof of
our theorem. �
We now discuss estimators for θ for our distribution f(x; θ). If X1, . . . , Xn are n independent
random variables with common distribution f(x; θ), then as n → ∞ the sample median converges
to the population median µ̃θ (if n = 2m + 1 then the sample median converges to being normally
distributed with median µ̃θ and variance 1/8mf(µ̃θ; θ)
2; see for example Theorem 8.17 of [MM]).
8 STEVEN J. MILLER
1.1 1.2 1.3 1.4
Figure 1. Plot of the median µ̃θ of f(x; θ) as a function of θ (µ̃1 = e
For θ close to 1 we see in Figure 1 that the median µ̃θ of f(x; θ) is strictly decreasing with
increasing θ, which implies that there is an inverse function g such that g(µ̃θ) = θ. We obtain an
estimator to θ by applying g to the sample median. This estimator is a consistent estimator (as
the sample size tends to infinity it will tend to θ) and should be asymptotically unbiased.
The proof of the Cramér-Rao inequality starts with
0 = E
· · ·
Θ̂(x1, . . . , xn) − θ
h(x1; θ) · · ·h(xn; θ)dx1 · · ·dxn
, (3.6)
where Θ̂(x1, . . . , xn) is an unbiased estimator of θ depending only on the sample values x1, . . . , xn.
In our case (when each h(x; θ) = f(x; θ)) we may not have an unbiased estimator. If we denote
this expectation by F(θ), for our investigations all that we require is that dF(θ)/dθ is finite (which
is easy to show). Going through the proof of the Cramér-Rao inequality shows that the effect
of this is to replace the factor of 1 in (1.2) with (1 + dF(θ)/dθ)2; thus the generalization of the
Cramér-Rao inequality for our estimator is
var(Θ̂) ≥
dF(θ)
∂ log f(x; θ)
. (3.7)
As our variance is infinite for θ = 1 we see that, no matter what ‘nice’ estimator we use, we will
not obtain any useful information from such arguments.
WHEN THE CRAMÉR-RAO INEQUALITY PROVIDES NO INFORMATION 9
Appendix A. Applying the Dominated Convergence Theorem
We justify applying the Dominated Convergence Theorem in the proof of Lemma 2.1. See, for
example, [SS] for the conditions and a proof of the Dominated Convergence Theorem.
Lemma A.1. For each fixed h > 0 and any x ≥ e, we have
1 − xh
∣∣∣∣ ≤ e log x, (A.1)
and e log x
x log3 x
is positive and integrable, and dominates each 1−x
x log3 x
Proof. We first prove (A.1). As x ≥ e and h > 0, note xh ≥ 1. Consider the case of 1/h ≤ log x.
Since |1 − xh| < 1 + xh ≤ 2xh, we have
|1 − xh|
≤ 2 log x. (A.2)
We are left with the case of 1/h > log x, or h logx < 1. We have
|1 − xh| = |1 − eh log x|
∣∣∣∣∣1 −
(h log x)n
∣∣∣∣∣
= h log x
(h log x)n−1
< h log x
(h log x)n−1
(n − 1)!
= h logx · eh log x. (A.3)
This, combined with h log x < 1 and xh ≥ 1 yields
|1 − xh|
eh log x
= e log x. (A.4)
It is clear that log x
x log3 x
is positive and integrable, and by L’Hospital’s rule (see (2.9)) we have that
1 − xh
x log3 x
x log2 x
. (A.5)
Thus the Dominated Convergence Theorem implies that
1 − xh
x log3 x
x log2 x
= −1 (A.6)
10 STEVEN J. MILLER
(the last equality is derived in Remark 2.3). �
References
[CaBe] G. Casella and R. Berger, Statistical Inference, 2nd edition, Duxbury Advanced Series, Pacific Grove,
CA, 2002.
[DM] D. Devoto and S. Martinez, Truncated Pareto Law and oresize distribution of ground rocks, Mathematical
Geology 30 (1998), no. 6, 661–673.
[Hi] T. Hill, A statistical derivation of the significant-digit law, Statistical Science 10 (1996), 354–363.
[MM] I. Miller and M. Miller, John E. Freund’s Mathematical Statistics with Applications, seventh edition,
Prentice Hall, 2004.
[Ne] M. E. J. Newman, Power laws, Pareto distributions and Zipfs law, Contemporary Physics 46 (2005),
no. 5, 323-351.
[NM] M. Nigrini and S. J. Miller, Benford’s Law applied to hydrology data – results and relevance to other
geophysical data, preprint.
[Rai] R. A. Raimi, The first digit problem, Amer. Math. Monthly 83 (1976), no. 7, 521–538.
[Rud] W. Rudin, Principles of Mathematical Analysis, third edition, International Series in Pure and Applied
Mathematics, McGraw-Hill Inc., New York, 1976.
[SS] E. Stein and R. Shakarchi, Real Analysis: Measure Theory, Integration, and Hilbert Spaces, Princeton
University Press, Princeton, NJ, 2005.
Department of Mathematics, Brown University, 151 Thayer Street, Providence, RI 02912
E-mail address: sjmiller@math.brown.edu
	1. Cramér-Rao Inequality
	2. An Almost Pareto Density
	3. Computing the Information
	Appendix A. Applying the Dominated Convergence Theorem
	References
