Ive a model with 14 dependent variables (all of them are significant) and 678 observations. I used best subset regression and validation set (33% of data for the validation) to find which statistical model has the lowest MSE (for my curiosity). I got the following graph which surprisingly the MSE for validation data set is always lower than training data set for all the models (from 1 to 14 dependent variables).Here is the code that I used,library(MASS)set.seed(1)train=sample(seq(678),452,replace=FALSE)trainregfit.exh=regsubsets(HPV~. -Model.Types..code.-Year..code.,data=Mydata, nvmax=NULL,force.in = NULL, force.out = NULL, method=exhaustive)val.errors=rep(NA,14)x.test=model.matrix(HPV~.-Model.Types..code.-Year..code.,data=Mydata[-train,])     for(i in 1:14){  coefi=coef(regfit.exh,id=i)  pred=x.test[,names(coefi)]%*%coefi  val.errors[i]=mean((Mydata$HPV[-train]-pred)^2)}plot(sqrt(val.errors),ylab=Root MSE,ylim=c(3,12), pch=11, type=b)points(sqrt(regfit.exh$rss[-1]/452),col=blue,pch=11,type=b)legend(topright,legend=c(Training,Validation),col=c(blue,black),pch=11)How come the validation root MSE could always beat the training?. Any feedback would be appreciated.
Comparing training and validation data set Root MSE for a best subset regression?
r;regression
Im a little late, but better late than never.  It looks like your line where you find the coefficients: regfit.exh=regsubsets(HPV~. -Model.Types..code.-Year..code.,data=Mydata, nvmax=NULL,force.in = NULL, force.out = NULL, method=exhaustive)should have data=Mydata[train,] instead of data=Mydata.Your model had already seen your test samples, so the validation error is not an accurate assessment.