Estimation of experimental data redundancy and related statistics
EPJ manuscript No.
(will be inserted by the editor)
Estimation of experimental data redundancy and related
statistics
Igor Grabec
Faculty of Mechanical Engineering, University of Ljubljana,
Aškerčeva 6, PP 394, 1001 Ljubljana, Slovenia,
Tel: +386 01 4771 605, Fax: +386 01 4253 135, E-mail: igor.grabec@fs.uni-lj.si
Received: date / Revised version: date
Abstract. Redundancy of experimental data is the basic statistic from which the complexity of a natural
phenomenon and the proper number of experiments needed for its exploration can be estimated. The
redundancy is expressed by the entropy of information pertaining to the probability density function of
experimental variables. Since the calculation of entropy is inconvenient due to integration over a range
of variables, an approximate expression for redundancy is derived that includes only a sum over the set
of experimental data about these variables. The approximation makes feasible an efficient estimation of
the redundancy of data along with the related experimental information and information cost function.
From the experimental information the complexity of the phenomenon can be simply estimated, while
the proper number of experiments needed for its exploration can be determined from the minimum of
the cost function. The performance of the approximate estimation of these statistics is demonstrated on
two–dimensional normally distributed random data.
PACS. 06.20.DK Measurement and error theory – 02.50.+s Probability theory, stochastic processes, and
statistics – 89.70.+c Information science
1 Introduction
The basic task of experimental physical exploration of nat-
ural phenomena is to provide quantitative data on mea-
sured variables and, from them extract physical laws [1].
Related to this task, experimenters must decide how many
experiments to perform in order to provide proper exper-
http://arxiv.org/abs/0704.0162v1
2 Igor Grabec: Estimation of experimental data redundancy and related statistics
imental data. We know that it is reasonable to repeat ex-
periments as long as they yield essentially new data, and
to stop repetition when the data become redundant. In
order to describe this concept objectively, we have intro-
duced in previous articles [2,3] two statistics called exper-
imental information I and redundancy R of experimental
data based on the entropy of information [4]. Their differ-
ence C = R−I can be interpreted as the information cost
function of the experimental exploration. From the cost
function minimum, the proper number N◦ of experiments
can be determined in an objective way. The entropy of in-
formation is defined by the integral of a nonlinear function
of the probability density function of experimental data,
and consequently its calculation is numerically demand-
ing. This property represents a serious obstacle, especially
when treating multivariate data. Therefore, our aim is to
show how this obstacle can be effectively avoided by es-
timating data redundancy without integration. For this
purpose we first briefly repeat the route to the definition
of redundancy [2,3] and subsequently show how the inte-
gral in the corresponding expression can be approximated.
The performance of the derived approximate method of
calculation is demonstrated using two–dimensional nor-
mally distributed random data.
2 Redundancy of experimental data
Let us consider a phenomenon characterized by N mea-
surements of a variable x using an instrument with span
Sx = (−L,L). Properties of the instrument are specified
by calibration on a unit u. The probability density func-
tion (PDF) of the instrument’s output scattering during
calibration is described by the scattering function ψ(x, u).
When the scattering is caused by mutually independent
disturbances in the experimental system, the scattering
function is Gaussian [1,4] :
ψ(x, u) = g(x− xi, σ) =
(x− u)2
. (1)
We apply this function in our further treatment. The mean
value u and standard deviation σ can be estimated statis-
tically by repetition of calibration.
Let xi denote the most probable instrument output
in the i–th experiment. Using ψ(x, xi) we describe the
properties of the explored phenomenon during the i–th
experiment. Similarly, the properties in a series of N re-
peated experiments, which yield the basic data set {xi; i =
1, . . . , N}, are described by the experimentally estimated
fN (x) =
ψ(x, xi). (2)
In addition, we introduce a uniform reference PDF ρ(x) =
1/(2L) indicating that all outcomes of the experiment are
hypothetically equally probable before executing the ex-
periments.
Based upon functions fN (x) and ρ(x) we describe the
indeterminacy of variable x by the negative value of the
relative entropy [5,6,7]:
Hx = −
f(x) log
(fN (x)
dx. (3)
Similarly, we describe the uncertainty Hu of calibration
performed on a unit u by:
Hu = −
ψ(x, u) log
(ψ(x, u)
dx. (4)
Igor Grabec: Estimation of experimental data redundancy and related statistics 3
Using the difference of these statistics we define the ex-
perimental information:
I = Hx −Hu
f(x) log(fN (x)) dx
ψ(x, u) log(ψ(x, u)) dx. (5)
Using Eq. 2 in this expression we get:
I = log(N) − 1
ψ(x, xi) log
ψ(x, xj)
ψ(x, u) log
ψ(x, u)
dx. (6)
If we express the logarithm in the second term as:
ψ(x, xj)
= logψ(x, xi) + log
ψ(x, xj)
ψ(x, xi)
we obtain:
I = log(N) +
ψ(x, xi) log
ψ(x, xi)
ψ(x, xi) log
ψ(x, xj)
ψ(x, xi)
ψ(x, u) log
ψ(x, u)
dx. (8)
The second and the fourth term on the right side of this
equation yield 0 and we get:
I = log(N)− 1
ψ(x, xi) log
ψ(x, xj)
ψ(x, xi)
With the last term we introduce the statistic called redun-
dancy of data:
ψ(x, xi) log
ψ(x, xj)
ψ(x, xi)
dx (10)
with which we get the basic relation:
I = log(N)−R (11)
If |xi − xj | ≫ σ for all pairs i#j, there is no overlapping
of functions ψ(x, xi), ψ(x, xj); therefore, the sum in the
logarithm is ∼ 0, and consequently the redundancy is R ∼
0. In the opposite case, when |xi − xj | ≪ σ, it follows
that ψ(x, xi) ∼ ψ(x, xj). Due to good overlapping in this
case, the corresponding term in the expression of R yields
log(2)/N and R > 0.
This property indicates that experimental information
is increasing with increasing N as I ∼ log(N) if the ac-
quired data are well separated with respect to σ. However,
with an increasing number of data, they are ever more
densely distributed, which results in an increasing overlap-
ping of distributions that causes increasing redundancy of
measurements. Although the expression in Eq. 10 for re-
dundancy R is rather cumbersome due to the included
integral, we expect that R could be estimated without
integration by the simpler function of distances between
data points. For this purpose we next consider the prop-
erties of the scattering function ψ(x, xi).
If the Gaussian function ψ(x, xi) = g(x−xi, σ) is con-
sidered as an approximation of the delta function δ(x−xi),
and the logarithm as a slowly changing function, the inte-
gration in Eq. 10 can be carried out, which yields for the
redundancy the first order approximate expression with-
out the integral:
ψ(xi, xj)
ψ(xi, xi)
If we take into account Eq. 1, we get for the redundancy
the following approximate expression that depends only
4 Igor Grabec: Estimation of experimental data redundancy and related statistics
on standard functions of distances between data points:
− (xi − xj)
However, this first order approximation is rather rough
because the distribution ψ(xi, xj) has the width σ > 0 and
the logarithm in Eq. 10 includes the fraction of functions
ψ(x, xj)/ψ(x, xi). To proceed to a better approximation,
we have examined the case of just two data points, since
it mainly determines the property of the redundancy. In
this case the integration of the first three terms in a Taylor
series expansion of the logarithm yields the second approx-
imation:
− (xi − xj)
, (14)
which is obtained from the previous one by merely chang-
ing 2σ2 → 4σ2. This property indicates that a still better
approximation could be obtained by properly adapting
2σ2 in Eq. 13. For this purpose we have proceeded with
numerical investigations which have shown that a nearly
optimal approximation is obtained if 2σ2 in Eq. 13 is re-
placed by ∼ 5.1σ2:
− (xi − xj)
5.1σ2
. (15)
Numerical investigations have further shown that this for-
mula also yields good results in cases with many data
points.
Since the integral is excluded from Eq. 15, the redun-
dancy R can be estimated from Eq. 15 with essentially less
computational effort than from Eq. 10. This advantage is
especially outstanding in a multivariate case where the
redundancy is defined by multiple integrals, while in the
approximate formula in Eq. 15 only the term (xi − xj) in
the exponential function has to be replaced by the norm
of corresponding vectors. Due to this advantage, it is also
reasonable to estimate approximately the experimental in-
formation using the basic formula I = log(N) − R. The
experimental information I converges with the increasing
number of data N to a certain limit value from which the
complexity of the phenomenon under investigation can be
estimated using the formula K ≈ exp(IN→∞) introduced
previously [2,3]. The complexity K indicates how many
non–overlapping scattering distributions are needed in the
estimator Eq. 2 to describe the PDF of the observed phe-
nomenon.
The information cost function is the difference of the
redundancy and experimental information: C = R − I.
During minimization of this cost, the experimental infor-
mation provides for a proper adaptation of the PDF es-
timator to the experimental data, while the redundancy
prevents excessive growth of the number of data points. By
the position of the cost function minimum we introduce
the proper number No of the data and the corresponding
experiments that are needed to judiciously represent the
phenomenon under exploration. By inserting the expres-
sion I = log(N) − R into C = R − I, we obtain for the
information cost function the formula:
C = 2R− log(N). (16)
Therefore the proper number No can also be determined
from the approximately estimated redundancy Ro. This
number roughly corresponds to the ratio between the mag-
nitude of the characteristic region where experimental data
Igor Grabec: Estimation of experimental data redundancy and related statistics 5
appear and the magnitude of the characteristic region cov-
ered by the scattering function [2,3].
3 Numerical examples
To demonstrate the properties of the approximations R1,
R2, Ro let us first consider the case of just two data
points separated by a distance x1 − x2. Fig. 1 shows the
dependence of redundancy R on relative distance d =
(x1−x2)/σ as determined by the integral in Eq. 10 and ap-
proximations in Eqs. 13,14,15. Improvement achieved by
subsequent steps of approximation and a fairly good agree-
ment between approximation Ro and R calculated by the
integral is evident. However, in a case with more data
points we can generally expect slightly worse agreement
due to overlapping of more than two scattering functions
in the sum of the approximation formula in Eq. 15. The
performance in such a case is demonstrated in the next
example.
In order to provide for reproduction of the demon-
strated example, we consider a two–dimensional Gaussian
random phenomenon with zero mean value. The stan-
dard deviation of both components is equal to s = 2.5,
while their covariance is zero. The data generated by a
standard Gaussian generator are represented in the two-
dimensional span (−10,+10)⊗ (−10,+10) using the scat-
tering width σ = 0.5. In such a case we can theoretically
predict that the proper number of data samples should be
No ≈ (s/σ)2 = 25.
For the demonstration, a set of Nmax = 100 two-
dimensional data samples {(xi, yi); i = 1 . . .Nmax} was
0 1 2 3 4 5 6
Fig. 1. Dependence of redundancy R on relative distance d =
(x1−x2)/σ between data points as determined by the integral
in Eq. 10, and approximations in Eqs. 13,14,15.
0.005
0.015
0.025
0.035
Fig. 2. PDF determined by 100 data points xi, yi.
generated. The corresponding probability density function
was estimated using Eq. 2 adapted to the two–dimensional
case with statistically independent components:
fN (x, y) =
ψ(x, xi)ψ(y, yi). (17)
The resulting PDF with N = 100 is graphically repre-
sented in Fig. 2.
6 Igor Grabec: Estimation of experimental data redundancy and related statistics
0 10 20 30 40 50 60 70 80 90 100
REDUNDANCY VERSUS NUMBER OF DATA
Fig. 3. Dependence of redundancy R on number N of data
points as determined by the integral in Eq. 10 – (R), and ap-
proximation in Eq. 15 – (Ro) adapted to the two–dimensional
case.
From the generated data the redundancy was calcu-
lated using Eqs. 10 and 15 adapted to the two–dimensional
case. The dependence of redundancy R on the number N
of accounted data points is shown in Fig. 3. Fairly good
agreement between both statistics is again evident.
Approximately estimated redundancy was further uti-
lized in the calculation of statistics I and C. They are
shown as functions of the number of data points N in
Fig. 4 together with R(N) and log(N). Agreement with
the same statistics calculated more exactly by integration
can be established by comparing this figure with Fig. 4.
In both cases we obtain for the proper number the value
No = 28. This value depends on the statistical properties
of the data set used in its calculation; a statistical esti-
mation from 100 different data sets yields the estimate
No ≈ 25±13 which agrees well with the theoretically pre-
0 10 20 30 40 50 60 70 80 90 100
INFORMATION STATISTICS VERSUS NUMBER OF DATA
log( N)
Fig. 4. Dependence of information statistics on the number N
of data points as approximately determined from Eq. 15. The
minimum of the cost function occurs at N = 28.
0 10 20 30 40 50 60 70 80 90 100
log( N)
Fig. 5. Dependence of information statistics on the number N
of data points as determined based on integration.
dicted value No = 25. Similarly as in the one–dimensional
case [2], it turns out that the function fNo(x, y) is only a
rough estimator of the hypothetical PDF. This property is
a consequence of the fact that experimental information I
and redundancy R have equal weights in the cost function
C = R− I.
Igor Grabec: Estimation of experimental data redundancy and related statistics 7
Figs. 4 and 5 indicate that experimental information I
converges with increasing N to a certain limit value from
which the complexity of the phenomenon under investiga-
tion can be approximately estimated as K ≈ exp INmax .
In our case we get the estimate K ≈ 21. The number
of non–overlapping scattering distributions that represent
the PDF of the observed phenomenon is thus slightly smaller
than the proper number No of experiments needed for its
exploration.
4 Conclusions
From the statistics introduced in the previous articles [2,3]
based on information entropy, we have here derived an ap-
proximate formula for the calculation of redundancy R of
experimental data. It is important that this formula does
not include the integral by which the information entropy
is defined. This makes feasible a simplified and fairly good
estimation of redundancy and, with it, the related exper-
imental information and cost function. The advantage of
the approximate calculation becomes outstanding in mul-
tivariate cases because multiple integration is not needed
there. A serious obstacle for the application of the con-
cept of experimental information and redundancy of data
can thus be avoided. Efficient estimation of the experi-
mental information and cost function, and with them the
determined complexity of the phenomenon and the proper
number of experiments needed for its exploration, could
be considered valuable in planning experimental work. In
addition, the complexityK or the proper numberNo could
be applied in the field of neural networks [1,8] to deter-
mine the appropriate number of cells needed to deal with
a certain phenomenon.
Acknowledgment
This work was supported by the Ministry of Higher Edu-
cation, Science and Technology of the Republic of Slovenia
and EU – COST.
References
1. I. Grabec and W. Sachse, Synergetics of Measurement, Pre-
diction and Control (Springer-Verlag, Berlin, 1997).
2. I. Grabec, Experimental modeling of physical laws, Eur.
Phys. J., B, 22 129-135 (2001)
3. I. Grabec, Extraction of physical laws from joint exper-
imental data, Eur. Phys. J., B, 48 279-289 (2005) (DOI:
10.1140/epjb/e2005-00391-0)
4. J. C. G. Lesurf, Information and Measurement (Institute of
Physics Publishing, Bristol, 2002)
5. T. M. Cover and J. A. Thomas, Elements of Information
Theory (John Wiley & Sons, New York, 1991).
6. A. N. Kolmogorov, IEEE Trans. Inf. Theory, IT-2 102-108
(1956).
7. D. J. C. MacKay, Information Theory, Inference, and Learn-
ing Algorithms (Cambridge University Press, Cambridge,
UK, 2003)
8. S. Haykin, Neural Networks, (Prentice Hall International,
Inc., Upper Saddle River, New Jersey, 1999)
	Introduction
	Redundancy of experimental data
	Numerical examples
	Conclusions
