I am wondering, if there are any heuristics on number of features versus number of observations. Obviously, if a number of features is equal to the number of observations, the model will overfit. By using sparse methods (LASSO, elastic net) we can remove several features to reduce the model. My question is (theoretically): before we use metrics to assess the model selection are there any empirical observations which relate the optimal number of features to the number of observations? For example: for a binary classification problem with 20 instances in each class, is there any upper limit on the number of features to use?
Any rules of thumb on number of features versus number of instances? (small data sets)
feature selection;model selection
Multiple papers have opined that only in rare cases is there a known distribution of the error as a function of the number of features and sample size.The error surface for a given set of instances, and features, is a function of the correlation (or lack of) between features.This paper suggests the following:For uncorrelated features, the optimal feature size is $ N-1 $ (where $ N $ is sample size)As feature correlation increases, and the optimal feature size becomes proportional to $ \sqrt N $ for highly correlated features.Another (empirical) approach that could be taken, is to draw the learning curves for different sample sizes from the same dataset, and use that to predict classifier performance at different sample sizes. Here's the link to the paper.