<jupyter_start><jupyter_text>Using Pipelines on Melbourn Housing Dataset<jupyter_code># Importing the required libraries
import pandas as pd
import numpy as np
# Reading the data into dataframe
df = pd.read_csv("melb_data.csv")
df.head()
# Separating the features and the target variables
X = df.drop(["Price"], axis=1)
y = df["Price"]
# Splitting the data in train test datasets for validation
from sklearn.model_selection import train_test_split
X_train_full, X_test_full, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=1)
print("X_train_full: ", X_train_full.shape)
print("X_test_full: ", X_test_full.shape)
print("y_train: ", y_train.shape)
print("X_test: ", y_test.shape)
# Since we dont want to use all the columns. For this purpose, we'll split the columns into numeric and categorial.

# For numerical columns we can get the columns in a list using by looping through all the columns and checking their dtype
num_cols = [col for col in X_train_full.columns if X_train_full[col].dtypes in ["int64", "float64"]]

# For categorical columns, we select the columns that have cardinality < 10
obj_cols = [col for col in X_train_full.columns if X_train_full[col].dtypes=="object" and X_train_full[col].nunique() < 10]

# Combining the required num and obj cols
cols = num_cols + obj_cols
X_train = X_train_full[cols].copy()
X_test = X_test_full[cols].copy()
# Checking to verify all the required columns are added
X_train.head()<jupyter_output><empty_output><jupyter_text>Preprocessing Next step is to preprocess the data. In our case, we have to perform different preprocessing, e.g., for numeric cols we have to deal with the null values and in the case of obj cols we have to deal with the null values and also perform encoding.For this purpose, ColumnTransformer will be used to for numeric columns to perform operations on the numeric columns and Pipeline will be used to perform operations on obj colums since there are multiple operations for obj columns, e.g., handling null values and encoding<jupyter_code>from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer

# First step is to prepare a SimpleImputer for numeric columns
num_imputer = SimpleImputer(strategy="constant")

# Next step is to prepare a pipeline for obj columns
obj_pipeline = Pipeline(steps=[("obj_imputer", SimpleImputer(strategy="most_frequent")),
                               ("one_hot_encoding", OneHotEncoder(handle_unknown="ignore"))
                              ])

preprocessing = ColumnTransformer(transformers=[("num", num_imputer, num_cols),
                                                    ("cat", obj_pipeline, obj_cols)
                                                    ])

# Next step would be to define a model
from sklearn.ensemble import RandomForestRegressor
model = RandomForestRegressor(n_estimators=100, random_state=1)<jupyter_output><empty_output><jupyter_text>Using Pipeline The final step in the process would be to define a pipeline and provide the preprocessing and model to it.<jupyter_code>my_pipeline = Pipeline(steps=[("preprocessing", preprocessing),
                              ("model", model)])

my_pipeline.fit(X_train, y_train)
y_pred = my_pipeline.predict(X_test)
# Finally, we use Mean Absolute Error to check the accuracy
from sklearn.metrics import mean_absolute_error

print("MAE: ", mean_absolute_error(y_test, y_pred))<jupyter_output>MAE:  157178.0246242724
{'fraction_non_alphanumeric': 0.07677053824362606, 'fraction_numerical': 0.009065155807365438, 'mean_word_length': 4.482919254658385, 'pattern_counts': {'":': 0, '<': 14, '<?xml version=': 0, '>': 12, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '193573', 'license': 'MIT', 'n_tokens_mistral': 1049, 'n_tokens_neox': 1018, 'n_words': 366, 'path': 'Pipelines_Melbourne_Housing.ipynb', 'repo_name': 'raziiq/python-pipelines', 'size': '3530'}