I'm currently working with Python and Scikit learn for classification purposes, and doing some reading around GridSearch I thought this was a great way for optimising my estimator parameters to get the best results.My methodology is this:Split my data into training/test.Use GridSearch with 5Fold Cross validation to train and test my estimators(Random Forest, Gradient Boost, SVC amongst others) to get the best estimators with the optimal combination of hyper parameters.I then calculate metrics on each of my estimators such as Precision, Recall, FMeasure and Matthews Correlation Coefficient, using my test set to predict the classifications and compare them to actual class labels.It is at this stage that I see strange behaviour and I'm unsure how to proceed. Do I take the .best_estimator_ from the GridSearch and use this as the 'optimal' output from the grid search, and perform prediction using this estimator? If I do this I find that the stage 3 metrics are usually much lower than if I simply train on all training data and test on the test set. Or, do I simply take the output GridSearchCV object as the new estimator? If I do this I get better scores for my stage 3 metrics, but it seems odd using a GridSearchCV object instead of the intended classifier (E.g. a random Forest) ...EDIT:So my question is what is the difference between the returned GridSearchCV object and the .best_estimator_ attribute? Which one of these should I use for calculating further metrics? Can I use this output like a regular classifier (e.g. using predict), or else how should I use it?
How to use the output of GridSearch?
machine learning;cross validation
Decided to go away and find the answers that would satisfy my question, and write them up here for anyone else wondering.The .best_estimator_ attribute is an instance of the specified model type, which has the 'best' combination of given parameters from the param_grid. Whether or not this instance is useful depends on whether the refit parameter is set to True (it is by default). For example:clf = GridSearchCV(estimator=RandomForestClassifier(),                     param_grid=parameter_candidates,                    cv=5,                    refit=True,                    error_score=0,                    n_jobs=-1)clf.fit(training_set, training_classifications)optimised_random_forest = clf.best_estimator_return optimised_random_forestWill return a RandomForestClassifier. This is all pretty clear from the documentation. What isn't clear from the documentation is why most examples don't specifically use the .best_estimator_ and instead do this:clf = GridSearchCV(estimator=RandomForestClassifier(),                     param_grid=parameter_candidates,                    cv=5,                    refit=True,                    error_score=0,                    n_jobs=-1)clf.fit(training_set, training_classifications)return clfThis second approach returns a GridSearchCV instance, with all the bells and whistles of the GridSearchCV such as .best_estimator_, .best_params, etc, which itself can be used like a trained classifier because:Optimised Random Forest Accuracy:  0.916970802919708[[139  47] [ 44 866]]GridSearchCV Accuracy:  0.916970802919708[[139  47] [ 44 866]]It just uses the same best estimator instance when making predictions. So in practise there's no difference between these two unless you specifically only want the estimator instance itself. As a side note, my differences in metrics were unrelated and down to a buggy class weighting function.