<jupyter_start><jupyter_text>Creating a DataFrame friendly scikit-learn pre-processing pipelineThe titanic data set includes categorical features such as class and sex, and also has missing values for numeric features (i.e., age) and categorical features (i.e., embarked). The aim is to predict whether or not a passenger survived. A standard sklearn example for this dataset can be found [here](https://scikit-learn.org/stable/auto_examples/compose/plot_column_transformer_mixed_types.htmlsphx-glr-auto-examples-compose-plot-column-transformer-mixed-types-py)We will build a preprocessing pipeline which:- for categorical variables will fill missing values with the string *Unknown* and then one-hot encode- for numerical values will fill missing values with median valuesThe strength of `sklearndf` is to maintain the scikit-learn conventions and expressivity, while also preserving dataframes, and hence feature names. We can see this after using fit_transform on our preprocessing pipeline.<jupyter_code>import numpy as np
from sklearn.datasets import fetch_openml
from sklearn.model_selection import train_test_split

# Relevant sklearndf imports
from sklearndf.transformation import (
    ColumnTransformerDF,
    OneHotEncoderDF,
    SimpleImputerDF,
)
from sklearndf.pipeline import (
    PipelineDF,
    ClassifierPipelineDF
)
from sklearndf.classification import RandomForestClassifierDF

# Load titanic data
titanic_X, titanic_y = fetch_openml("titanic", version=1, as_frame=True, return_X_y=True)

# Select features
numerical_features = ['age', 'fare']
categorical_features = ['embarked', 'sex', 'pclass']

# Create a pre-processing pipeline
preprocessing_numeric_df = SimpleImputerDF(strategy="median")

preprocessing_categorical_df = PipelineDF(
    steps=[
        ('imputer', SimpleImputerDF(strategy='constant', fill_value='Unknown')),
        ('one-hot', OneHotEncoderDF(sparse=False, handle_unknown="ignore"))
    ]
)

preprocessing_df = ColumnTransformerDF(
    transformers=[
        ('categorical', preprocessing_categorical_df, categorical_features),
        ('numeric', preprocessing_numeric_df, numerical_features),
    ]
)

# Run pre-processing
transformed_df = preprocessing_df.fit_transform(X=titanic_X, y=titanic_y)
transformed_df.head()<jupyter_output><empty_output><jupyter_text>Tracing features from post-transform to originalThe `sklearndf` pipeline has a `feature_names_original_` attribute which returns a series mapping the output columns (the series index) to the input columns (the series values). We can therefore easily select all output features generated from a given input feature, such as in this case for embarked.<jupyter_code>embarked_type_derivatives = preprocessing_df.feature_names_original_ == "embarked"
transformed_df.loc[:, embarked_type_derivatives].head()<jupyter_output><empty_output><jupyter_text>Completing the pipeline with a classifierScikit-learn regressors and classifiers have a `sklearndf` sibling obtained by appending DF to the class name; the API remains the same. The result of any predict and decision function will be returned as a pandas series (single output) or data frame (class probabilities or multi-output).We can combine the preprocessing pipeline above with a classifier to create a full predictive pipeline. sklearndf provides two useful, specialised pipeline objects for this, `RegressorPipelineDF` and `ClassifierPipelineDF`. Both implement a special two-step pipeline with one pre-processing step and one prediction step, while staying compatible with the general sklearn pipeline idiom.Using `ClassifierPipelineDF` we can combine the preprocessing pipeline with `RandomForestClassifierDF()` to fit a model to a selected training set and then score and a test set.<jupyter_code># create full pipeline
pipeline_df = ClassifierPipelineDF(
    preprocessing=preprocessing_df,
    classifier=RandomForestClassifierDF(
        n_estimators=1000,
        max_features=2/3,
        max_depth=7,
        random_state=42,
        n_jobs=-3
    )
)

# split data and then fit and score random forest classifier
df_train, df_test, y_train, y_test = train_test_split(titanic_X, titanic_y, random_state=42)
pipeline_df.fit(df_train, y_train)
print(f"model score: {pipeline_df.score(df_test, y_test).round(2)}")<jupyter_output>model score: 0.79
{'fraction_non_alphanumeric': 0.0702364394993046, 'fraction_numerical': 0.003940658321743162, 'mean_word_length': 5.8710191082802545, 'pattern_counts': {'":': 0, '<': 12, '<?xml version=': 0, '>': 12, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '640775', 'license': 'Apache-2.0', 'n_tokens_mistral': 1245, 'n_tokens_neox': 1192, 'n_words': 408, 'path': 'sphinx/auxiliary/Titanic_getting_started_example.ipynb', 'repo_name': 'mtsokol/sklearndf', 'size': '4314'}