Rayleigh quotient

In mathematics, the Rayleigh quotient () for a given complex Hermitian matrix "M" and nonzero vector "x" is defined as:
For real matrices and vectors, the condition of being Hermitian reduces to that of being symmetric, and the conjugate transpose formula_2 to the usual transpose formula_3. Note that formula_4 for any non-zero scalar "c". Recall that a Hermitian (or real symmetric) matrix is diagonalizable with only real eigenvalues. It can be shown that, for a given matrix, the Rayleigh quotient reaches its minimum value formula_5 (the smallest eigenvalue of "M") when "x" is formula_6 (the corresponding eigenvector). Similarly, formula_7 and formula_8.
The Rayleigh quotient is used in the min-max theorem to get exact values of all eigenvalues. It is also used in eigenvalue algorithms (such as Rayleigh quotient iteration) to obtain an eigenvalue approximation from an eigenvector approximation.
The range of the Rayleigh quotient (for any matrix, not necessarily Hermitian) is called a numerical range and contains its spectrum. When the matrix is Hermitian, the numerical range is equal to the spectral norm. Still in functional analysis, formula_9 is known as the spectral radius. In the context of C*-algebras or algebraic quantum mechanics, the function that to "M" associates the Rayleigh–Ritz quotient "R"("M","x") for a fixed "x" and "M" varying through the algebra would be referred to as "vector state" of the algebra.
In quantum mechanics, the Rayleigh quotient gives the expectation value of the observable corresponding to the operator "M" for a system whose state is given by "x".
If we fix the complex matrix "M", then the resulting Rayleigh quotient map (considered as a function of "x") completely determines "M" via the polarization identity; indeed, this remains true even if we allow "M" to be non-Hermitian. (However, if we restrict the field of scalars to the real numbers, then the Rayleigh quotient only determines the symmetric part of "M".)
Bounds for Hermitian formula_10.
As stated in the introduction, for any vector "x", one has formula_11, where formula_12 are respectively the smallest and largest eigenvalues of formula_10. This is immediate after observing that the Rayleigh quotient is a weighted average of eigenvalues of "M":
where formula_15 is the formula_16th eigenpair after orthonormalization and formula_17 is the formula_16th coordinate of "x" in the eigenbasis. It is then easy to verify that the bounds are attained at the corresponding eigenvectors formula_19.
The fact that the quotient is a weighted average of the eigenvalues can be used to identify the second, the third, ... largest eigenvalues. Let formula_20 
be the eigenvalues in decreasing order. If formula_21 and formula_22 is constrained to be orthogonal to formula_23, in which case formula_24, then formula_25 has maximum value formula_26, which is achieved when formula_27.
Special case of covariance matrices.
An empirical covariance matrix formula_10 can be represented as the product formula_29 of the data matrix formula_30 pre-multiplied by its transpose formula_31. Being a positive semi-definite matrix, formula_10 has non-negative eigenvalues, and orthogonal (or orthogonalisable) eigenvectors, which can be demonstrated as follows.
Firstly, that the eigenvalues formula_33 are non-negative:
Secondly, that the eigenvectors formula_38 are orthogonal to one another:
if the eigenvalues are different – in the case of multiplicity, the basis can be orthogonalized.
To now establish that the Rayleigh quotient is maximized by the eigenvector with the largest eigenvalue, consider decomposing an arbitrary vector formula_22 on the basis of the eigenvectors formula_38:
where
is the coordinate of formula_22 orthogonally projected onto formula_38. Therefore, we have:
which, by orthonormality of the eigenvectors, becomes:
The last representation establishes that the Rayleigh quotient is the sum of the squared cosines of the angles formed by the vector formula_22 and each eigenvector formula_38, weighted by corresponding eigenvalues.
If a vector formula_22 maximizes formula_25, then any non-zero scalar multiple formula_52 also maximizes formula_53, so the problem can be reduced to the Lagrange problem of maximizing formula_54 under the constraint that formula_55.
Define: formula_56. This then becomes a linear program, which always attains its maximum at one of the corners of the domain. A maximum point will have formula_57 and formula_58 for all formula_59 (when the eigenvalues are ordered by decreasing magnitude).
Thus, the Rayleigh quotient is maximized by the eigenvector with the largest eigenvalue.
Formulation using Lagrange multipliers.
Alternatively, this result can be arrived at by the method of Lagrange multipliers. The first part is to show that the quotient is constant under scaling formula_60, where formula_61 is a scalar
Because of this invariance, it is sufficient to study the special case formula_63. The problem is then to find the critical points of the function
subject to the constraint formula_65 In other words, it is to find the critical points of 
where formula_67 is a Lagrange multiplier. The stationary points of formula_68 occur at
and 
Therefore, the eigenvectors formula_71 of formula_10 are the critical points of the Rayleigh quotient and their corresponding eigenvalues formula_73 are the stationary values of formula_74. This property is the basis for principal components analysis and canonical correlation.
Use in Sturm–Liouville theory.
Sturm–Liouville theory concerns the action of the linear operator
on the inner product space defined by
of functions satisfying some specified boundary conditions at "a" and "b". In this case the Rayleigh quotient is
This is sometimes presented in an equivalent form, obtained by separating the integral in the numerator and using integration by parts: