Cross Validated
Cross Validated Meta
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
2 features and 2 principal components
Whats the difference between 2 features and 2 principal components? I know what a PCA is, I just have the following problem:
If my data has 2 features, the PCA will produce 2 components. So why does the predictive power of 2 features equal the predicive power of 2 principal components?
machine-learning pca dimensionality-reduction
Robomatix
5811 silver badge1010 bronze badges
$\begingroup$ This question seems clear enough to me (even if the OP is unclear on the issues, but then, that's why they're asking). $\endgroup$ – gung - Reinstate Monica May 25 '16 at 10:59
Generally for linear models, changing the basis doesn't matter:
For linear models, representing your features in your original two-dimensional basis or a new two-dimensional basis from PCA won't change the predictive power. For the purposes of this question, there's nothing special about the PCA basis.
Let $\mathbf{x} = \left[ \begin{array}{c} x_1 \\ x_2 \end{array} \right]$ denote an observation from a two-dimensional feature space.
We can construct a new basis for this two-dimensional space using any set of two linearly independent vectors. These vectors could be from Principal Component Analysis (PCA), but they could also come from elsewhere (as long as they're linearly independent). Let $U$ be a matrix whose columns are vectors of the new basis. Coordinates of our observation $\mathbf{x}$ in terms of the new basis is given by multiplying by the inverse of the matrix $U$.
$$\mathbf{z} = U^{-1} \mathbf{x}$$
In many machine learning contexts, you use linear transformations of your observations to do stuff. You use some $A\mathbf{x}$ where matrix $A$ represents some linear map.
Clearly:
$$ A \mathbf{x} = AUU^{-1}\mathbf{x} = AU \mathbf{z}$$
That is, it's entirely equivalent to use:
linear transformation $A$ and vector $\mathbf{x}$ written in the original basis
linear transformation $AU$ and vector $\mathbf{z}$ written in the new basis
You can see though that this wouldn't hold for non-linear transformations.
Simple example: OLS regression
Model is: $$y_i = \mathbf{x}_i \cdot \boldsymbol{\beta} + \epsilon_i$$
OLS estimate: $$ \mathbf{b}_x = (X'X)^{-1}(X'\mathbf{y})$$ With data in new basis: $Z = X{U'}^{-1}$
$$ \begin{align*} \mathbf{b}_z &= (Z'Z)^{-1}(Z'\mathbf{y}) \\ &= \left( U^{-1}X'X {U'}^{-1} \right) ^{-1} {U^{-1}X' \mathbf{y} }\\ &= U' \left( X'X \right) ^{-1} U U^{-1}X' \mathbf{y} \\ &= U' \left( X'X \right) ^{-1} X' \mathbf{y} \\ &= U' \mathbf{b}_x \\ \end{align*} $$
Keep going and the residuals will be the same etc... You have the same predictive power and the estimated coefficients are related by the change in basis.
Matthew GunnMatthew Gunn
In your question, I believe features refers to the size of the space. For example, measurements for population variation were taken on two features: hair color and shoe size.
Principle component analysis is often can often be associated with dimension reduction, but it need not be. In it's most general formulation, it is simply a rotation of the space to the most coordinate system where the variances along the axes are uncorrelated. In such a coordinate system, the components are linear combinations of the features, and in some cases, they may be the features themselves.
The questions is asking you why using all of the features and using all of the principle components provides the same predictive power. I'll let you work that part out yourself.
BillyziegeBillyziege
Without any context, if you have two noncollinear variables, you have two features and two PCs. The PC reflects the structure of the data. Multiple collinear variables lead to fewer PCs than variables, the PCs simply provide an orthonormal basis for a matrix of features.
What is perhaps interesting is sparse representation. LASSO is a supervised learning approach where a small number of features can be selected for predictive value. Alternately, choosing a small number of PCs and their loadings provides sparse representation of a matrix of features in an unsupervised approach. In that sense, the implications would be vastly different.
AdamOAdamO
Thanks for contributing an answer to Cross Validated!
Not the answer you're looking for? Browse other questions tagged machine-learning pca dimensionality-reduction or ask your own question.
How can top principal components retain the predictive power on a dependent variable (or even lead to better predictions)?
A problem with interpretation of principal components as linear combinations of features
What happens if I do principal components of the principal components?
Models using more (almost all) principal components of the data are unexpectedly worse in cross validation
Principal component analysis and DBScan
Understanding the determination of principal components
Cross Validated works best with JavaScript enabled