<jupyter_start><jupyter_text>Concrete Feature Engineering--- Reference> [What Is Feature Engineering](https://www.kaggle.com/ryanholbrook/what-is-feature-engineering)> [Data Source](https://www.kaggle.com/ryanholbrook/fe-course-data)--- Dependencies<jupyter_code>import pandas as pd
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import cross_val_score

%load_ext autotime<jupyter_output>time: 160 µs (started: 2021-08-15 15:29:08 -07:00)<jupyter_text>--- Import Dataset To illustrate these ideas we'll see how adding a few synthetic features to a dataset can improve the predictive performance of a random forest model.The [Concrete](https://www.kaggle.com/sinamhd9/concrete-comprehensive-strength) dataset contains a variety of concrete formulations and the resulting product's *compressive strength*, which is a measure of how much load that kind of concrete can bear. The task for this dataset is to predict a concrete's compressive strength given its formulation.<jupyter_code>df = pd.read_csv("data/concrete.csv")
df.head()<jupyter_output><empty_output><jupyter_text>--- Baseline You can see here the various ingredients going into each variety of concrete. We'll see in a moment how adding some additional synthetic features derived from these can help a model to learn important relationships among them.We'll first establish a baseline by training the model on the un-augmented dataset. This will help us determine whether our new features are actually useful.Establishing baselines like this is good practice at the start of the feature engineering process. A baseline score can help you decide whether your new features are worth keeping, or whether you should discard them and possibly try something else.<jupyter_code>X = df.copy()
y = X.pop("CompressiveStrength")

# Train and score baseline model
baseline = RandomForestRegressor(criterion="mae", random_state=0)
baseline_score = cross_val_score(
    baseline, X, y, cv=5, scoring="neg_mean_absolute_error"
)
baseline_score = -1 * baseline_score.mean()

print(f"MAE Baseline Score: {baseline_score:.4}")<jupyter_output>MAE Baseline Score: 8.232
time: 8.81 s (started: 2021-08-15 15:29:08 -07:00)<jupyter_text>If you ever cook at home, you might know that the ratio of ingredients in a recipe is usually a better predictor of how the recipe turns out than their absolute amounts. We might reason then that ratios of the features above would be a good predictor of `CompressiveStrength`.The cell below adds three new ratio features to the dataset.<jupyter_code>X = df.copy()
y = X.pop("CompressiveStrength")

# Create synthetic features
X["FCRatio"] = X["FineAggregate"] / X["CoarseAggregate"]
X["AggCmtRatio"] = (X["CoarseAggregate"] + X["FineAggregate"]) / X["Cement"]
X["WtrCmtRatio"] = X["Water"] / X["Cement"]

# Train and score model on dataset with additional ratio features
model = RandomForestRegressor(criterion="mae", random_state=0)
score = cross_val_score(
    model, X, y, cv=5, scoring="neg_mean_absolute_error"
)
score = -1 * score.mean()

print(f"MAE Score with Ratio Features: {score:.4}")<jupyter_output>MAE Score with Ratio Features: 7.948
time: 12.9 s (started: 2021-08-15 15:29:17 -07:00)<jupyter_text>And sure enough, performance improved! This is evidence that these new ratio features exposed important information to the model that it wasn't detecting before.<jupyter_code>X.head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09088260996213225, 'fraction_numerical': 0.023303233323623652, 'mean_word_length': 5.881763527054108, 'pattern_counts': {'":': 0, '<': 18, '<?xml version=': 0, '>': 20, 'https://': 3, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '55907', 'license': 'MIT', 'n_tokens_mistral': 1079, 'n_tokens_neox': 982, 'n_words': 376, 'path': '18-Day-Kaggle-Competition-Automobile/concrete-baseline.ipynb', 'repo_name': 'Ivanbh214/30-Days-of-ML-Kaggle', 'size': '3433'}