Projection matrix

In statistics, the projection matrix formula_1, sometimes also called the influence matrix or hat matrix formula_2, maps the vector of response values (dependent variable values) to the vector of fitted values (or predicted values). It describes the influence each response value has on each fitted value. The diagonal elements of the projection matrix are the leverages, which describe the influence each response value has on the fitted value for that same observation.
Overview.
If the vector of response values is denoted by formula_3 and the vector of fitted values by formula_4,
As formula_4 is usually pronounced "y-hat", the projection matrix formula_7 is also named "hat matrix" as it "puts a hat on formula_3". The formula for the vector of residuals formula_9 can also be expressed compactly using the projection matrix:
where formula_11 is the identity matrix. The matrix formula_12 is sometimes referred to as the residual maker matrix. Moreover, the element in the "i"th row and "j"th column of formula_7 is equal to the covariance between the "j"th response value and the "i"th fitted value, divided by the variance of the former:
Therefore, the covariance matrix of the residuals formula_9, by error propagation, equals
where formula_17 is the covariance matrix of the error vector (and by extension, the response vector as well). For the case of linear models with independent and identically distributed errors in which formula_18, this reduces to:
Intuition.
From the figure, it is clear that the closest point from the vector formula_20 onto the column space of formula_21, is formula_22, and is one where we can draw a line orthogonal to the column space of formula_21. A vector that is orthogonal to the column space of a matrix is in the nullspace of the matrix transpose, so 
From there, one rearranges, so 
Therefore, since formula_26 is on the column space of formula_21, the projection matrix, which maps formula_20 onto formula_26 is just formula_22, or formula_31
Linear model.
Suppose that we wish to estimate a linear model using linear least squares. The model can be written as
where formula_33 is a matrix of explanatory variables (the design matrix), β is a vector of unknown parameters to be estimated, and ε is the error vector.
Many types of models and techniques are subject to this formulation. A few examples are linear least squares, smoothing splines, regression splines, local regression, kernel regression, and linear filtering.
Ordinary least squares.
When the weights for each observation are identical and the errors are uncorrelated, the estimated parameters are
so the fitted values are
Therefore, the projection matrix (and hat matrix) is given by
Weighted and generalized least squares.
The above may be generalized to the cases where the weights are not identical and/or the errors are correlated. Suppose that the covariance matrix of the errors is Ψ. Then since
the hat matrix is thus
and again it may be seen that formula_39, though now it is no longer symmetric.
Properties.
The projection matrix has a number of useful algebraic properties. In the language of linear algebra, the projection matrix is the orthogonal projection onto the column space of the design matrix formula_33. (Note that formula_41 is the pseudoinverse of X.) Some facts of the projection matrix in this setting are summarized as follows:
The projection matrix corresponding to a linear model is symmetric and idempotent, that is, formula_47. However, this is not always the case; in locally weighted scatterplot smoothing (LOESS), for example, the hat matrix is in general neither symmetric nor idempotent.
For linear models, the trace of the projection matrix is equal to the rank of formula_33, which is the number of independent parameters of the linear model. For other models such as LOESS that are still linear in the observations formula_3, the projection matrix can be used to define the effective degrees of freedom of the model.
Practical applications of the projection matrix in regression analysis include leverage and Cook's distance, which are concerned with identifying influential observations, i.e. observations which have a large effect on the results of a regression.
Blockwise formula.
Suppose the design matrix formula_63 can be decomposed by columns as formula_64.
Define the hat or projection operator as formula_65. Similarly, define the residual operator as formula_66.
Then the projection matrix can be decomposed as follows:
where, e.g., formula_68 and formula_69.
There are a number of applications of such a decomposition. In the classical application formula_70 is a column of all ones, which allows one to analyze the effects of adding an intercept term to a regression. Another use is in the fixed effects model, where formula_70 is a large sparse matrix of the dummy variables for the fixed effect terms. One can use this partition to compute the hat matrix of formula_72 without explicitly forming the matrix formula_63, which might be too large to fit into computer memory.