I'm using the scikit-learn gradient boosting classifier found here.If I run the classifier on the same data without seeding the random number generator, I get different feature importances, and notably, different ones are 0 each time. If I just look at the features where the feature importance is < .005, most of the same ones appear, but there is a little variation there too.My question is, does this have an impact other than computation time? If it were to hurt accuracy, I imagine that would be due to over-fitting. However, if the classifier already identifies the relatively unimportant features, then doesn't that take care of the problem?If it does hurt accuracy, how should I select which features to get rid of?
Does using unimportant features hurt accuracy?
python;scikit learn
I used the gradient boosting classifier in a project a while ago. We had about 130 features and since the system had to be implemented on a microcontroller I tried to reduce the number of features as far as possible.Here is a plot showing the performance (F1-score) of GBC models trained with the top n most important features:From that image it looks like good-enough performance is already reached with only 30 out of 130 features, while the other 100 (unimportant) features don't seem to have much positive nor negative influence.Only when you zoom in a bit on the Y-axis you see this:This shows that the performance reaches its peak (0.93) at 90 out of 130 features. So 40 features are unimportant if you want the best model.To answer your question does this have an impact other than computation time?: it does have an impact on performance, since the performance decreases again with more than 90 features, but only slightly.To answer your second question If it does hurt accuracy, how should I select which features to get rid of?:I did the following:Select the top n featuresUse a cross-validated grid search to train a GBC model on the top n featuresRepeat with the top n+1,... featuresSelect the model / features with the best scoreHow to select the top n features in step 1):I actually didn't use GBC for feature selection, but random forests. I don't know why, but my experiments have shown that selecting features with RF and then training a GBC model on them works better than selecting features with GBC and then training a GBC model on them.