<jupyter_start><jupyter_text>In k-fold cross-validation, the original sample is randomly partitioned into k equal sized subsamples. Of the k subsamples, a single subsample is retained as the validation data for testing the model, and the remaining k − 1 subsamples are used as training data. The cross-validation process is then repeated k times, with each of the k subsamples used exactly once as the validation data. The k results can then be averaged to produce a single estimation. The advantage of this method over repeated random sub-sampling (see below) is that all observations are used for both training and validation, and each observation is used for validation exactly once. 10-fold cross-validation is commonly used, but in general k remains an unfixed parameter.Source: https://en.wikipedia.org/wiki/Cross-validation_(statistics)k-fold_cross-validation<jupyter_code>lb_make = LabelEncoder()

credit_data2 = credit_data.copy()
credit_data2['AgeCat'] = pd.cut(credit_data2['Age (years)'], 4)
credit_data2['AgeCat'] = lb_make.fit_transform(credit_data2["AgeCat"].astype(str))

credit_data2['CredAmtCat'] = pd.cut(credit_data2['Credit Amount'],3)
credit_data2['CredAmtCat'] = lb_make.fit_transform(credit_data2["CredAmtCat"].astype(str))

credit_data2['CredDurCat'] = pd.cut(credit_data2['Duration of Credit (month)'],4)
credit_data2['CredDurCat'] = lb_make.fit_transform(credit_data2["CredDurCat"].astype(str))
credit_data2 = credit_data2.drop(["Age (years)", "Credit Amount", "Duration of Credit (month)"], axis=1)
credit_data2.describe()

X = credit_data2.loc[:, credit_data.columns != 'Creditability']
y = credit_data2["Creditability"]
X_train, X_validation, y_train, y_validation = train_test_split(X, y, test_size=1/3., random_state=42)

## function to select the columns

class ColumnSelector(BaseEstimator, TransformerMixin):
    def __init__(self, columns):
        self.columns = columns

    def fit(self, X, y=None):
        return self

    def transform(self, X):
        assert isinstance(X, pd.DataFrame)

        try:
            return X[self.columns]
        except KeyError:
            cols_error = list(set(self.columns) - set(X.columns))
            raise KeyError("The DataFrame does not include the columns: %s" % cols_error)


pipeline = Pipeline(steps = [
        ("features", make_union(
                ColumnSelector(list(X)),
                )),
                ("model",RandomForestClassifier(random_state=42))
])

pipeline.fit(X_train, y_train)

pipeline.score(X_validation, y_validation)

print("RF Score before CV: %s" % pipeline.score(X_validation, y_validation))

# get list of hyperparameters

hyperparameters = { 'model__max_depth': [50, 70,90],
                    'model__min_samples_leaf': [1,2,3]
                  }

clf = GridSearchCV(pipeline, hyperparameters, cv=10)

clf.fit(X_train, y_train)

print("RF Score after CV: %s" % clf.score(X_validation, y_validation))<jupyter_output>RF Score before CV: 0.739520958084
RF Score after CV: 0.793413173653
{'fraction_non_alphanumeric': 0.10878243512974052, 'fraction_numerical': 0.02262142381902861, 'mean_word_length': 4.568518518518519, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 4, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '56536', 'license': 'MIT', 'n_tokens_mistral': 980, 'n_tokens_neox': 914, 'n_words': 259, 'path': 'Section 7/7.3.ipynb', 'repo_name': 'PacktPublishing/Scikit-learn-Recipes', 'size': '3006'}