Orthogonality principle

In statistics and signal processing, the orthogonality principle is a necessary and sufficient condition for the optimality of a Bayesian estimator. Loosely stated, the orthogonality principle says that the error vector of the optimal estimator (in a mean square error sense) is orthogonal to any possible estimator. The orthogonality principle is most commonly stated for linear estimators, but more general formulations are possible. Since the principle is a necessary and sufficient condition for optimality, it can be used to find the minimum mean square error estimator.
Orthogonality principle for linear estimators.
The orthogonality principle is most commonly used in the setting of linear estimation. In this context, let "x" be an unknown random vector which is to be estimated based on the observation vector "y". One wishes to construct a linear estimator formula_1 for some matrix "H" and vector "c". Then, the orthogonality principle states that an estimator formula_2 achieves minimum mean square error if and only if
If "x" and "y" have zero mean, then it suffices to require the first condition.
Example.
Suppose "x" is a Gaussian random variable with mean "m" and variance formula_5 Also suppose we observe a value formula_6 where "w" is Gaussian noise which is independent of "x" and has mean 0 and variance formula_7 We wish to find a linear estimator formula_8 minimizing the MSE. Substituting the expression formula_9 into the two requirements of the orthogonality principle, we obtain
and
Solving these two linear equations for "h" and "c" results in
so that the linear minimum mean square error estimator is given by
This estimator can be interpreted as a weighted average between the noisy measurements "y" and the prior expected value "m". If the noise variance formula_18 is low compared with the variance of the prior formula_19 (corresponding to a high SNR), then most of the weight is given to the measurements "y", which are deemed more reliable than the prior information. Conversely, if the noise variance is relatively higher, then the estimate will be close to "m", as the measurements are not reliable enough to outweigh the prior information.
Finally, note that because the variables "x" and "y" are jointly Gaussian, the minimum MSE estimator is linear. Therefore, in this case, the estimator above minimizes the MSE among all estimators, not only linear estimators.
General formulation.
Let formula_20 be a Hilbert space of random variables with an inner product defined by formula_21. Suppose formula_22 is a closed subspace of formula_20, representing the space of all possible estimators. One wishes to find a vector formula_24 which will approximate a vector formula_25. More accurately, one would like to minimize the mean squared error (MSE) formula_26 between formula_2 and formula_28.
In the special case of linear estimators described above, the space formula_20 is the set of all functions of formula_28 and formula_31, while formula_22 is the set of linear estimators, i.e., linear functions of formula_31 only. Other settings which can be formulated in this way include the subspace of causal linear filters and the subspace of all (possibly nonlinear) estimators.
Geometrically, we can see this problem by the following simple case where formula_22 is a one-dimensional subspace:
We want to find the closest approximation to the vector formula_28 by a vector formula_2 in the space formula_22. From the geometric interpretation, it is intuitive that the best approximation, or smallest error, occurs when the error vector, formula_38, is orthogonal to vectors in the space formula_22. 
More accurately, the general orthogonality principle states the following: Given a closed subspace formula_22 of estimators within a Hilbert space formula_20 and an element formula_28 in formula_20, an element formula_24 achieves minimum MSE among all elements in formula_22 if and only if formula_46 for all formula_47
Stated in such a manner, this principle is simply a statement of the Hilbert projection theorem. Nevertheless, the extensive use of this result in signal processing has resulted in the name "orthogonality principle."
A solution to error minimization problems.
The following is one way to find the minimum mean square error estimator by using the orthogonality principle.
We want to be able to approximate a vector formula_28 by 
where
is the approximation of formula_28 as a linear combination of vectors in the subspace formula_22 spanned by formula_53 Therefore, we want to be able to solve for the coefficients, formula_54, so that we may write our approximation in known terms.
By the orthogonality theorem, the square norm of the error vector, formula_55, is minimized when, for all "j",
Developing this equation, we obtain
If there is a finite number formula_58 of vectors formula_59, one can write this equation in matrix form as
Assuming the formula_59 are linearly independent, the Gramian matrix can be inverted to obtain
thus providing an expression for the coefficients formula_63 of the minimum mean square error estimator.