Projection (linear algebra)

In linear algebra and functional analysis, a projection is a linear transformation formula_1 from a vector space to itself such that formula_2. That is, whenever formula_1 is applied twice to any value, it gives the same result as if it were applied once (idempotent). It leaves its image unchanged. Though abstract, this definition of "projection" formalizes and generalizes the idea of graphical projection. One can also consider the effect of a projection on a geometrical object by examining the effect of the projection on points in the object.
Definitions.
A projection on a vector space formula_4 is a linear operator formula_5 such that formula_6.
When formula_4 has an inner product and is complete (i.e. when formula_4 is a Hilbert space) the concept of orthogonality can be used. A projection formula_1 on a Hilbert space formula_4 is called an orthogonal projection if it satisfies formula_11 for all formula_12. A projection on a Hilbert space that is not orthogonal is called an oblique projection.
Projection matrix.
The eigenvalues of a projection matrix must be 0 or 1.
Examples.
Orthogonal projection.
For example, the function which maps the point formula_22 in three-dimensional space formula_23 to the point formula_24 is an orthogonal projection onto the "xy"-plane. This function is represented by the matrix
The action of this matrix on an arbitrary vector is
To see that formula_1 is indeed a projection, i.e., formula_28, we compute
Observing that formula_30 shows that the projection is an orthogonal projection.
Oblique projection.
A simple example of a non-orthogonal (oblique) projection (for definition see below) is
Via matrix multiplication, one sees that
proving that formula_1 is indeed a projection.
The projection formula_1 is orthogonal if and only if formula_35 because only then formula_30.
Properties and classification.
Idempotence.
By definition, a projection formula_1 is idempotent (i.e. formula_6).
Complementarity of range and kernel.
Let formula_39 be a finite dimensional vector space and formula_1 be a projection on formula_39. Suppose the subspaces formula_42 and formula_4 are the range and kernel of formula_1 respectively. Then formula_1 has the following properties:
The range and kernel of a projection are "complementary", as are formula_1 and formula_57. The operator formula_58 is also a projection as the range and kernel of formula_1 become the kernel and range of formula_58 and vice versa. We say formula_1 is a projection along formula_4 onto formula_42 (kernel/range) and formula_58 is a projection along formula_42 onto formula_4.
Spectrum.
In infinite dimensional vector spaces, the spectrum of a projection is contained in formula_67 as
Only 0 or 1 can be an eigenvalue of a projection. This implies that an orthogonal projection formula_1 is always a positive semi-definite matrix. In general, the corresponding eigenspaces are (respectively) the kernel and range of the projection. Decomposition of a vector space into direct sums is not unique. Therefore, given a subspace formula_4, there may be many projections whose range (or kernel) is formula_4.
If a projection is nontrivial it has minimal polynomial formula_72, which factors into distinct roots, and thus formula_1 is diagonalizable.
Product of projections.
The product of projections is not in general a projection, even if they are orthogonal. If two projections commute then their product is a projection, but the converse is false: the product of two non-commuting projections may be a projection . 
If two orthogonal projections commute then their product is an orthogonal projection. If the product of two orthogonal projections is an orthogonal projection, then the two orthogonal projections commute (more generally: two self-adjoint endomorphisms commute if and only if their product is self-adjoint).
Orthogonal projections.
When the vector space formula_39 has an inner product and is complete (is a Hilbert space) the concept of orthogonality can be used. An orthogonal projection is a projection for which the range formula_42 and the null space formula_4 are orthogonal subspaces. Thus, for every formula_77 and formula_78 in formula_39, formula_80. Equivalently:
A projection is orthogonal if and only if it is self-adjoint. Using the self-adjoint and idempotent properties of formula_1, for any formula_77 and formula_78 in formula_39 we have formula_86, formula_87, and
where formula_89 is the inner product associated with formula_39. Therefore, formula_91 and formula_92 are orthogonal projections. The other direction, namely that if formula_1 is orthogonal then it is self-adjoint, follows from
for every formula_95 and formula_96 in formula_39; thus formula_98.
!Proof of existence
Let formula_99 be a complete metric space with an inner product, and let formula_42 be a closed linear subspace of formula_99 (and hence complete as well).
For every formula_77 the following set of non-negative norm-values formula_103 has an infimum, and due to the completeness of formula_42 it is a minimum. We define formula_105 as the point in formula_42 where this minimum is obtained.
Obviously formula_105 is in formula_42. It remains to show that formula_105 satisfies formula_110 and that it is linear.
Let us define formula_111. For every non-zero formula_112 in formula_42, the following holds:
In the general case, we can have an arbitrary positive definite matrix formula_117 defining an inner product formula_118, and the projection formula_119 is given by formula_120. Then
When the range space of the projection is generated by a frame (i.e. the number of generators is greater than its dimension), the formula for the projection takes the form: formula_122. Here formula_123 stands for the Moore–Penrose pseudoinverse. This is just one of many ways to construct the projection operator.
If formula_124 is a non-singular matrix and formula_125 (i.e., formula_126 is the null space matrix of formula_127), the following holds: 
If the orthogonal condition is enhanced to formula_129 with formula_39 non-singular, the following holds:
All these formulas also hold for complex inner product spaces, provided that the conjugate transpose is used instead of the transpose. Further details on sums of projectors can be found in Banerjee and Roy (2014). Also see Banerjee (2004) for application of sums of projectors in basic spherical trigonometry.
Oblique projections.
The term "oblique projections" is sometimes used to refer to non-orthogonal projections. These projections are also used to represent spatial figures in two-dimensional drawings (see oblique projection), though not as frequently as orthogonal projections. Whereas calculating the fitted value of an ordinary least squares regression requires an orthogonal projection, calculating the fitted value of an instrumental variables regression requires an oblique projection.
Projections are defined by their null space and the basis vectors used to characterize their range (which is the complement of the null space). When these basis vectors are orthogonal to the null space, then the projection is an orthogonal projection. When these basis vectors are not orthogonal to the null space, the projection is an oblique projection. Let the vectors formula_132 form a basis for the range of the projection, and assemble these vectors in the formula_133 matrix formula_127. The range and the null space are complementary spaces, so the null space has dimension formula_135. It follows that the orthogonal complement of the null space has dimension formula_136. Let formula_137 form a basis for the orthogonal complement of the null space of the projection, and assemble these vectors in the matrix formula_126. Then the projection is defined by
This expression generalizes the formula for orthogonal projections given above.
Finding projection with an inner product.
Let formula_4 be a vector space (in this case a plane) spanned by orthogonal vectors formula_141. Let formula_96 be a vector. One can define a projection of formula_78 onto formula_4 as
where repeated indices are summed over (Einstein sum notation). The vector formula_78 can be written as an orthogonal sum such that formula_147. formula_148 is sometimes denoted as formula_149. There is a theorem in linear algebra that states that this formula_150 is the shortest distance from formula_78 to formula_4 and is commonly used in areas such as machine learning.
Canonical forms.
Any projection formula_28 on a vector space of dimension formula_154 over a field is a diagonalizable matrix, since its minimal polynomial divides formula_155, which splits into distinct linear factors. Thus there exists a basis in which formula_1 has the form
where formula_158 is the rank of formula_1. Here formula_160 is the identity matrix of size formula_158, and formula_162 is the zero matrix of size formula_163. If the vector space is complex and equipped with an inner product, then there is an "orthonormal" basis in which the matrix of "P" is
where formula_165. The integers formula_166 and the real numbers formula_167 are uniquely determined. Note that formula_168. The factor formula_169 corresponds to the maximal invariant subspace on which formula_1 acts as an "orthogonal" projection (so that "P" itself is orthogonal if and only if formula_171) and the formula_167-blocks correspond to the "oblique" components.
Projections on normed vector spaces.
When the underlying vector space formula_173 is a (not necessarily finite-dimensional) normed vector space, analytic questions, irrelevant in the finite-dimensional case, need to be considered. Assume now formula_173 is a Banach space.
Many of the algebraic results discussed above survive the passage to this context. A given direct sum decomposition of formula_173 into complementary subspaces still specifies a projection, and vice versa. If formula_173 is the direct sum formula_177, then the operator defined by formula_178 is still a projection with range formula_42 and kernel formula_4. It is also clear that formula_6. Conversely, if formula_1 is projection on formula_173, i.e. formula_6, then it is easily verified that formula_185. In other words, formula_186 is also a projection. The relation formula_6 implies formula_188 and formula_173 is the direct sum formula_190.
However, in contrast to the finite-dimensional case, projections need not be continuous in general. If a subspace formula_42 of formula_173 is not closed in the norm topology, then projection onto formula_42 is not continuous. In other words, the range of a continuous projection formula_1 must be a closed subspace. Furthermore, the kernel of a continuous projection (in fact, a continuous linear operator in general) is closed. Thus a "continuous" projection formula_1 gives a decomposition of formula_173 into two complementary "closed" subspaces: formula_197.
The converse holds also, with an additional assumption. Suppose formula_42 is a closed subspace of formula_173. If there exists a closed subspace formula_4 such that , then the projection formula_1 with range formula_42 and kernel formula_4 is continuous. This follows from the closed graph theorem. Suppose and . One needs to show that formula_204. Since formula_42 is closed and , "y" lies in formula_42, i.e. . Also, . Because formula_4 is closed and , we have formula_208, i.e. formula_209, which proves the claim.
The above argument makes use of the assumption that both formula_42 and formula_4 are closed. In general, given a closed subspace formula_42, there need not exist a complementary closed subspace formula_4, although for Hilbert spaces this can always be done by taking the orthogonal complement. For Banach spaces, a one-dimensional subspace always has a closed complementary subspace. This is an immediate consequence of Hahn–Banach theorem. Let formula_42 be the linear span of formula_116. By Hahn–Banach, there exists a bounded linear functional formula_216 such that . The operator formula_217 satisfies formula_2, i.e. it is a projection. Boundedness of formula_216 implies continuity of formula_1 and therefore formula_221 is a closed complementary subspace of formula_42.
Applications and further considerations.
Projections (orthogonal and otherwise) play a major role in algorithms for certain linear algebra problems:
As stated above, projections are a special case of idempotents. Analytically, orthogonal projections are non-commutative generalizations of characteristic functions. Idempotents are used in classifying, for instance, semisimple algebras, while measure theory begins with considering characteristic functions of measurable sets. Therefore, as one can imagine, projections are very often encountered in the context of operator algebras. In particular, a von Neumann algebra is generated by its complete lattice of projections.
Generalizations.
More generally, given a map between normed vector spaces formula_223 one can analogously ask for this map to be an isometry on the orthogonal complement of the kernel: that formula_224 be an isometry (compare Partial isometry); in particular it must be onto. The case of an orthogonal projection is when "W" is a subspace of "V." In Riemannian geometry, this is used in the definition of a Riemannian submersion.