If I understand correctly, nested-CV can help me evaluate what model and hyperparameter tuning process is best. The inner loop (GridSearchCV) finds the best hyperparameters, and the outter loop (cross_val_score) evaluates the hyperparameter tuning algorithm. I then choose which tuning/model combo from the outer loop that minimizes mse (I'm looking at regression classifier) for my final model test. I've read the questions/answers on nested-cross-validation, but haven't seen an example of a full pipeline that utilizes this. So, does my code below (please ignore the actual hyperparameter ranges - this is just for example) and thought process make sense?from sklearn.cross_validation import cross_val_score, train_test_splitfrom sklearn.grid_search import GridSearchCVfrom sklearn.metrics import mean_squared_errorfrom sklearn.ensemble import RandomForestRegressorfrom sklearn.svm import SVRfrom sklearn.datasets import make_regression# create some regression dataX, y = make_regression(n_samples=1000, n_features=10)params = [{'C':[0.01,0.05,0.1,1]},{'n_estimators':[10,100,1000]}]# setup models, variablesmean_score = []models = [SVR(), RandomForestRegressor()]# split into train and test setsX_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.3)# estimate performance of hyperparameter tuning and model algorithm pipelinefor idx, model in enumerate(models):    clf = GridSearchCV(model, params[idx], scoring='mean_squared_error')    # this performs a nested CV in SKLearn    score = cross_val_score(clf, X_train, y_train, scoring='mean_squared_error')    # get the mean MSE across each fold    mean_score.append(np.mean(score))    print('Model:', model, 'MSE:', mean_score[-1])# estimate generalization performance of the best model selection techniquebest_idx = mean_score.index(max(mean_score)) # because SKLearn flips MSE signs, max works OK herebest_model = models[best_idx]clf_final = GridSearchCV(best_model, params[best_idx])clf_final.fit(X_train, y_train)y_pred = clf_final.predict(X_test)rmse = np.sqrt(mean_squared_error(y_test, y_pred))print('Final Model': best_model, 'Final model RMSE:', rmse)
Nested cross-validation and selecting the best regression model - is this the right SKLearn process?
python;scikit learn;cross validation;model selection
Yours is not an example of nested cross-validation.Nested cross-validation is useful to figure out whether, say, a random forest or a SVM is better suited for your problem. Nested CV only outputs a score, it does not output a model like in your code.This would be an example of nested cross validation:from sklearn.datasets import load_bostonfrom sklearn.cross_validation import KFoldfrom sklearn.metrics import mean_squared_errorfrom sklearn.grid_search import GridSearchCVfrom sklearn.ensemble import RandomForestRegressorfrom sklearn.svm import SVRimport numpy as npparams = [{'C': [0.01, 0.05, 0.1, 1]}, {'n_estimators': [10, 100, 1000]}]models = [SVR(), RandomForestRegressor()]df = load_boston()X = df['data']y = df['target']cv = [[] for _ in range(len(models))]for tr, ts in KFold(len(X)):    for i, (model, param) in enumerate(zip(models, params)):        best_m = GridSearchCV(model, param)        best_m.fit(X[tr], y[tr])        s = mean_squared_error(y[ts], best_m.predict(X[ts]))        cv[i].append(s)print(np.mean(cv, 1))By the way, a couple of thoughts:I see no purpose to grid search for n_estimators for your random forest. Obviously, the more, the merrier. Things like max_depth is the kind of regularization that you want to optimize. The error for the nested CV of RandomForest was much higher because you did not optimize for the right hyperparameters, not necessarily because it is a worse model.You might also want to try gradient boosting trees.