Invariant estimator

In statistics, the concept of being an invariant estimator is a criterion that can be used to compare the properties of different estimators for the same quantity. It is a way of formalising the idea that an estimator should have certain intuitively appealing qualities. Strictly speaking, "invariant" would mean that the estimates themselves are unchanged when both the measurements and the parameters are transformed in a compatible way, but the meaning has been extended to allow the estimates to change in appropriate ways with such transformations. The term equivariant estimator is used in formal mathematical contexts that include a precise description of the relation of the way the estimator changes in response to changes to the dataset and parameterisation: this corresponds to the use of "equivariance" in more general mathematics.
General setting.
Background.
In statistical inference, there are several approaches to estimation theory that can be used to decide immediately what estimators should be used according to those approaches. For example, ideas from Bayesian inference would lead directly to Bayesian estimators. Similarly, the theory of classical statistical inference can sometimes lead to strong conclusions about what estimator should be used. However, the usefulness of these theories depends on having a fully prescribed statistical model and may also depend on having a relevant loss function to determine the estimator. Thus a Bayesian analysis might be undertaken, leading to a posterior distribution for relevant parameters, but the use of a specific utility or loss function may be unclear. Ideas of invariance can then be applied to the task of summarising the posterior distribution. In other cases, statistical analyses are undertaken without a fully defined statistical model or the classical theory of statistical inference cannot be readily applied because the family of models being considered are not amenable to such treatment. In addition to these cases where general theory does not prescribe an estimator, the concept of invariance of an estimator can be applied when seeking estimators of alternative forms, either for the sake of simplicity of application of the estimator or so that the estimator is robust.
The concept of invariance is sometimes used on its own as a way of choosing between estimators, but this is not necessarily definitive. For example, a requirement of invariance may be incompatible with the requirement that the estimator be mean-unbiased; on the other hand, the criterion of median-unbiasedness is defined in terms of the estimator's sampling distribution and so is invariant under many transformations.
One use of the concept of invariance is where a class or family of estimators is proposed and a particular formulation must be selected amongst these. One procedure is to impose relevant invariance properties and then to find the formulation within this class that has the best properties, leading to what is called the optimal invariant estimator.
Some classes of invariant estimators.
There are several types of transformations that are usefully considered when dealing with invariant estimators. Each gives rise to a class of estimators which are invariant to those particular types of transformation.
The combination of permutation invariance and location invariance for estimating a location parameter from an independent and identically distributed dataset using a weighted average implies that the weights should be identical and sum to one. Of course, estimators other than a weighted average may be preferable.
Optimal invariant estimators.
Under this setting, we are given a set of measurements formula_1 which contains information about an unknown parameter formula_2. The measurements formula_1 are modelled as a vector random variable having a probability density function formula_4 which depends on a parameter vector formula_2.
The problem is to estimate formula_2 given formula_1. The estimate, denoted by formula_8, is a function of the measurements and belongs to a set formula_9. The quality of the result is defined by a loss function formula_10 which determines a risk function formula_11. The sets of possible values of formula_1, formula_2, and formula_8 are denoted by formula_15, formula_16, and formula_9, respectively.
In classification.
In statistical classification, the rule which assigns a class to a new data-item can be considered to be a special type of estimator. A number of invariance-type considerations can be brought to bear in formulating prior knowledge for pattern recognition.
Mathematical setting.
Definition.
An invariant estimator is an estimator which obeys the following two rules:
To define an invariant or equivariant estimator formally, some definitions related to groups of transformations are needed first. Let formula_15 denote the set of possible data-samples. A group of transformations of formula_15, to be denoted by formula_24, is a set of (measurable) 1:1 and onto transformations of formula_15 into itself, which satisfies the following conditions:
Datasets formula_34 and formula_35 in formula_15 are equivalent if formula_37 for some formula_29. All the equivalent points form an equivalence class.
Such an equivalence class is called an orbit (in formula_15). The formula_40 orbit, formula_41, is the set formula_42.
If formula_15 consists of a single orbit then formula_44 is said to be transitive.
A family of densities formula_45 is said to be invariant under the group formula_24 if, for every formula_29 and formula_48 there exists a unique formula_49 such that formula_50 has density formula_51. formula_52 will be denoted formula_53.
If formula_45 is invariant under the group formula_24 then the loss function formula_56 is said to be invariant under formula_24 if for every formula_29 and formula_59 there exists an formula_60 such that formula_61 for all formula_62. The transformed value formula_63 will be denoted by formula_64.
In the above, formula_65 is a group of transformations from formula_16 to itself and formula_67 is a group of transformations from formula_9 to itself.
An estimation problem is invariant(equivariant) under formula_24 if there exist three groups formula_70 as defined above.
For an estimation problem that is invariant under formula_24, estimator formula_72 is an invariant estimator under formula_24 if, for all formula_74 and formula_29,
Properties.
For a given problem, the invariant estimator with the lowest risk is termed the "best invariant estimator". Best invariant estimator cannot always be achieved. A special case for which it can be achieved is the case when formula_82 is transitive.
Example: Location parameter.
Suppose formula_2 is a location parameter if the density of formula_15 is of the form formula_86. For formula_87 and formula_88, the problem is invariant under formula_89. The invariant estimator in this case must satisfy
thus it is of the form formula_91 (formula_92). formula_82 is transitive on formula_16 so the risk does not vary with formula_2: that is, formula_96. The best invariant estimator is the one that brings the risk formula_97 to minimum.
In the case that L is the squared error formula_98
Pitman estimator.
The estimation problem is that formula_99 has density formula_100, where "θ" is a parameter to be estimated, and where the loss function is formula_101. This problem is invariant with the following (additive) transformation groups:
The best invariant estimator formula_72 is the one that minimizes
and this is Pitman's estimator (1939).
For the squared error loss case, the result is
If formula_108 (i.e. a multivariate normal distribution with independent, unit-variance components) then
If formula_110 (independent components having a Cauchy distribution with scale parameter "σ") then
formula_111. However the result is
with