<jupyter_start><jupyter_text>PdPipeline + Sklearn Model This notebook is meant to provide a brief tutorial on how to combine a `PdPipeline` and a scikit-learn model (any `sklearn` estimator, in fact) to create a combined sklearn-compliant estimator object. This object inherits any `predict`, `predict_proba`, etc. methods from the composing estimator, and naturally applies the inner pipeline to any input dataframe before passing the processed dataframe (or actually, its cast to a `numpy.ndarray`) to that estimator.The main new functionality introduced here is a `pdpipe` class that can be extended to create such custom pipline+estimator classes with construtor parameters encapsulating both pipeline hyperparameters and model hyperparameters. Thus, the created pipeline-model object can be thought of as a stronger model with additional hyperparameters, and this extended model can have its hyperparameters optimized using sklearn functionalities such as the `model_selection.GridSearchCV` class and the `model_selection.cross_val_score` function.  The Data Let's start by creating a mock dataset that can demonstrate basic `pdpipe` functionalities.<jupyter_code>import pandas as pd
df = pd.DataFrame(
    data=[
        [23, 'Jo', 'M', True, 0.07, 'USA', 'Living life to its fullest'],
        [52, 'Regina', 'F', False, 0.26, 'Germany', 'I hate cats'],
        [23, 'Dana', 'F', True, 0.3, 'USA', 'the pen is mightier then the sword'],
        [25, 'Bo', 'M', False, 2.3, 'Greece', 'all for one and one for all'],
        [80, 'Richy', 'M', False, 100.2, 'Finland', 'I gots the dollarz'],
        [60, 'Paul', 'M', True, 1.87, 'Denmark', 'blah'],
        [44, 'Derek', 'M', True, 1.1, 'Denmark', 'every life is precious'],
        [72, 'Regina', 'F', True, 7.1, 'Greece', 'all of you get off my porch'],
        [50, 'Jim', 'M', False, 0.2, 'Germany', 'boy do I love dogs and cats'],
        [80, 'Wealthus', 'F', False, 123.2, 'Finland', 'me likey them moniez'],
    ],
    columns=['Age', 'Name', 'Gender', 'Smoking', 'Savings', 'Country', 'Quote'],
)
df<jupyter_output><empty_output><jupyter_text>Defining a combined object In this section we will define a custom class which will capture all the pipeline logic and the model concatenated to it.In this case we hard-code the use of the `LogisticRegression` model. With a bit more complex code, the model type/family itself to use can be set up as a hyperparameter.<jupyter_code>from typing import Optional
from sklearn.linear_model import LogisticRegression
import pdpipe as pdp
from pdpipe.skintegrate import PdPipelineAndSklearnEstimator
class MyPipelineAndModel(PdPipelineAndSklearnEstimator):
    
    def __init__(
        self,
        drop_gender: Optional[bool] = False,
        scale_numeric: Optional[bool] = False,
        ohencode_country: Optional[bool] = True,
        savings_bin_val: Optional[int] = None,
        fit_intercept: Optional[bool] = True,
    ):
        self.drop_gender = drop_gender
        self.scale_numeric = scale_numeric
        self.ohencode_country = ohencode_country
        self.savings_bin_val = savings_bin_val
        self.fit_intercept = fit_intercept
        cols_to_drop = []
        stages = [
            pdp.ColDrop(['Name', 'Quote'], errors='ignore'),
        ]
        if savings_bin_val:
            stages.append(pdp.Bin({'Savings': [savings_bin_val]}, drop=False))
            stages.append(pdp.Encode('Savings_bin'))
        if scale_numeric:
            stages.append(pdp.Scale('MinMaxScaler'))
        if drop_gender:
            cols_to_drop.append('Gender')
        else:
            stages.append(pdp.Encode('Gender'))
        if ohencode_country:
            stages.append(pdp.OneHotEncode('Country'))
        else:
            cols_to_drop.append('Country')
        stages.append(pdp.ColDrop(cols_to_drop, errors='ignore'))
        pline = pdp.PdPipeline(stages)
        model = LogisticRegression(fit_intercept=fit_intercept)
        super().__init__(pipeline=pline, estimator=model)
mp = MyPipelineAndModel(
    drop_gender=True,
    scale_numeric=True,
    ohencode_country=True,
    savings_bin_val=1,
    fit_intercept=True,
)
mp
mp.pipeline
mp.estimator
mp.score
mp.score?
mp.pipeline(df)
mp.pipeline[0:4](df)<jupyter_output><empty_output><jupyter_text>Test our custom estimator checks out<jupyter_code>from sklearn.utils.estimator_checks import check_estimator<jupyter_output><empty_output><jupyter_text>Actually, it does not. :(<jupyter_code># check_estimator(mp)<jupyter_output><empty_output><jupyter_text>But it is going to work anyway!  X-y subsets<jupyter_code>x_lbls = ['Age', 'Gender', 'Savings', 'Country']
all_x = df[x_lbls]
all_y = df['Smoking']
all_x.shape
all_y.shape
all_x<jupyter_output><empty_output><jupyter_text>Check inheritence of predict() This section demonstrates that the composed object inherits all the cool `sklearn` methods from the composing estimator!<jupyter_code>mp.fit(all_x, all_y)
mp.predict(all_x)
mp.predict(all_x).dtype == bool
mp.predict_proba(all_x)
mp.predict_proba(all_x).dtype == float
mp.predict_log_proba(all_x)
mp.decision_function(all_x)
mp.score(all_x, all_y)
mp.classes_<jupyter_output><empty_output><jupyter_text>Note that if we use `LinearRegression`, for example, which only has `predict` and `score`, our derivative pipeline+model object will only inherit these two methods:<jupyter_code>from sklearn.linear_model import LinearRegression
pipe_and_linreg = PdPipelineAndSklearnEstimator(
    pipeline=mp.pipeline,
    estimator=LinearRegression()
)
pipe_and_linreg.fit(all_x, all_y)
pipe_and_linreg.predict(all_x)
assert not hasattr(pipe_and_linreg, 'predict_proba')
assert not hasattr(pipe_and_linreg, 'predict_log_proba')
assert not hasattr(pipe_and_linreg, 'decision_function')<jupyter_output><empty_output><jupyter_text>GridSearchCV Here we can see how this object works flawlessly with sklearn's `GridSearchCV`!By default, the scoring function used to select the best model is the `score` method the inner estimator has. If the estimator has no such method we will have to provide a scorer function to the scorer constructor parameter of the `GridSearchCV` class. We will take a look at this later.<jupyter_code>from sklearn.model_selection import GridSearchCV
gcv = GridSearchCV(
    estimator=mp,
    param_grid={
        'savings_bin_val': [1, 2],
        'scale_numeric': [True, False],
        'drop_gender': [True, False],
        'ohencode_country': [True, False],
    },
    cv=3,
)
gcv
from sklearn.utils.validation import check_is_fitted
from sklearn.exceptions import NotFittedError
try:
    check_is_fitted(gcv)
except NotFittedError:
    print("Not fitted - as expected")
gcv.fit(all_x, all_y)
assert check_is_fitted(gcv) is None
gcv
# gcv.cv_results_
gcv.best_estimator_
gcv.best_score_
gcv.best_params_<jupyter_output><empty_output><jupyter_text>Working with custom scoring functions We can also use a custom scoring function with `GridSearchCV`. As usual, we will have to convert any metric / scoring function (a function with a signature of the form `score(y_true_, y_pred, ...)`) to a scorer function - with a signature of the form `scorer(estimator, X, y, ...)`; this is done with the `sklearn.metrics.make_scorer` function.Here, however, we will also have to convert the resulting `sklearn` scorer into a scorer that can handle our combined pipeline+model object. This can be done by importing the aptly-named `pdpipe_scorer_from_sklearn_scorer` from `pdpipe.skintegrate`, and applying it to the scorer to get a pdpipe-compliant one.<jupyter_code>from sklearn.metrics import fbeta_score, make_scorer
ftwo_scorer = make_scorer(fbeta_score, beta=2)
from pdpipe.skintegrate import pdpipe_scorer_from_sklearn_scorer
my_scorer = pdpipe_scorer_from_sklearn_scorer(ftwo_scorer)
my_scorer
gcv = GridSearchCV(
    estimator=mp,
    param_grid={
        'savings_bin_val': [1, 2],
        'scale_numeric': [True, False],
        'drop_gender': [True, False],
        'ohencode_country': [True, False],
    },
    cv=3,
    scoring=my_scorer,
)
gcv.fit(all_x, all_y)
gcv.best_score_
gcv.best_params_
# gcv.cv_results_<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10046671579464506, 'fraction_numerical': 0.007000736919675755, 'mean_word_length': 4.554570259208731, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 37, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '572672', 'license': 'MIT', 'n_tokens_mistral': 2685, 'n_tokens_neox': 2612, 'n_words': 774, 'path': 'notebooks/pipeline_and_model.ipynb', 'repo_name': 'rohankumardubey/pdpipe', 'size': '8142'}