I'm rather new to PCA and was hoping to have some confusion cleared up. Lets say for example we have a feature matrix that's nx100 and I want to get it down to something a bit smaller, p-dimensions, without losing too much variance. After applying PCA and receiving and new feature matrix nxp, I would use x_reduced to predict some target variable y.My question is, after the transformation, the new reduced feature matrix has been rotated by the eigenvectors and is sitting on a new basis. Yet, our y has not changed relative to X_reduced. I'm unsure about how y_original and x_reduced can be used for training since y has not changed with respect to x_reduced. Is there a way to correct for this or am I not thinking about it correctly?
PCA and maintaining relationship with target variable
statistics;feature scaling
The short answer is that the y_original and x_reduced are still connected to each other, so it is safe to train your data using y_original and x_reduced.  While x_reduced is on a different scale, as you mentioned via eigenvectors, it still is representative of the data that was attached to that observation, just in a different format.  You lose a lot of interpretability as far as what the actual numbers mean which is why it may seem confusing, but it's just a transformed representation of the x_original that (hopefully) contains enough of the x_original variability to make it useful.