Q:

How to pass two estimator objects to sklearn's GridSearchCV so that they have the same parameters in each step?

I'm trying to use GridSearchCV from SKlearn to tune hyperparameters for my estimator. 

In the first step, the estimator is used to for is SequentialFeatureSelection, which is a custom library that performs wrapper based feature selection. This means iteratively adding new features and identifying the ones where the estimator performs best with. Hence, the SequentialFeatureSelection method requires my estimator. This library is programmed so that it is perfectly fine to use with SKlearn, so I integrate it in the first step of the GridSearchCV pipeline to transform the features to the ones selected.
In the second step, I would like to use exactly the same classifier with exactly the same parameters to be fitted and predict the outcome. However with the parameter grid, I can only either set the parameters to the classifier that I pass to SequentialFeatureSelector OR the ones in 'clf' and I cannot assure that they are always the same.
Finally, with the selected features and selected parameters I want to predict on a previously held out test-set.

On the bottom of the page of the SFS library, they show how to use SFS with GridSearchCV, but there the KNN algorithm used to select features and the one used to predict are also using different parameters. And when I check for myself after traininf SFS and GridSearchCV, the parameters are never the same, even when I use the clf.clone() as proposed. Here is my code:
import sklearn.pipeline
import sklearn.tree
import sklearn.model_selection
import mlxtend.feature_selection

def sfs(x, y):
    x_train, x_test, y_train, y_test = sklearn.model_selection.train_test_split(x, y, test_size=0.2, random_state=0)

    clf = sklearn.tree.DecisionTreeClassifier()

    param_grid = {
        "sfs__estimator__max_depth": [5]
    }

    sfs = mlxtend.feature_selection.SequentialFeatureSelector(clone_estimator=True,  # Clone like in Tutorial
                                                              estimator=clf,
                                                              k_features=10,
                                                              forward=True,
                                                              floating=False,
                                                              scoring='accuracy',
                                                              cv=3,
                                                              n_jobs=1)

    pipe = sklearn.pipeline.Pipeline([('sfs', sfs), ("clf", clf)])

    gs = sklearn.model_selection.GridSearchCV(estimator=pipe,
                                              param_grid=param_grid,
                                              scoring='accuracy',
                                              n_jobs=1,
                                              cv=3,
                                              refit=True)

    gs = gs.fit(x_train, y_train)

    # Both estimators should have depth 5!
    print("SFS Final Estimator Depth: " + str(gs.best_estimator_.named_steps.sfs.estimator.max_depth))
    print("CLF Final Estimator Depth: " + str(gs.best_estimator_._final_estimator.max_depth))

    # Evaluate...
    y_test_pred = gs.predict(x_test)
    # Accuracy etc...

The question would be, how do I assure that they always have the same parameters set within the same pipeline?
Thanks!

A:

I found a solution, where I overwrite some methods of the SequentialFeatureSelector (SFS) class to also use its estimator for predicting after transformation. This is done by introducing a Custom SFS class 'CSequentialFeatureSelector' that overwrites the following methods from SFS:

In the fit(self, X, y) method, not only the normal fit is performed, but also the self.estimator is the fitted on the transformed data, so that it is possible to implement predict and predict_proba methods for the SFS class.
I implemented predict and predict_probba methods for the SFS class, that call the predict and predict_probba methods of the fitted self.estimator.

Hence, I only have one estimator left that is used for SFS and predicting.
Here is some of the code:
import sklearn.pipeline
import sklearn.tree
import sklearn.model_selection
import mlxtend.feature_selection

class CSequentialFeatureSelector(mlxtend.feature_selection.SequentialFeatureSelector):
    def predict(self, X):
        X = self.transform(X)
        return self.estimator.predict(X)

    def predict_proba(self, X):
        X = self.transform(X)
        return self.estimator.predict_proba(X)

    def fit(self, X, y):
        self.fit_helper(X, y) # fit helper is the 'old' fit method, which I copied and renamed to fit_helper
        self.estimator.fit(self.transform(X), y)
        return self

def sfs(x, y):
    x_train, x_test, y_train, y_test = sklearn.model_selection.train_test_split(x, y, test_size=0.2, random_state=0)

    clf = sklearn.tree.DecisionTreeClassifier()

    param_grid = {
        "sfs__estimator__max_depth": [3, 4, 5]
    }

    sfs = mlxtend.feature_selection.SequentialFeatureSelector(clone_estimator=True,
                                                              estimator=clf,
                                                              k_features=10,
                                                              forward=True,
                                                              floating=False,
                                                              scoring='accuracy',
                                                              cv=3,
                                                              n_jobs=1)

    # Now only one object in the pipeline (in fact this is not even needed anymore)
    pipe = sklearn.pipeline.Pipeline([('sfs', sfs)])

    gs = sklearn.model_selection.GridSearchCV(estimator=pipe,
                                              param_grid=param_grid,
                                              scoring='accuracy',
                                              n_jobs=1,
                                              cv=3,
                                              refit=True)

    gs = gs.fit(x_train, y_train)

    print("SFS Final Estimator Depth: " + str(gs.best_estimator_.named_steps.sfs.estimator.max_depth))

    y_test_pred = gs.predict(x_test)
    # Evaluate performance of y_test_pred

