<jupyter_start><jupyter_text>Lambda School Data Science, Unit 2: Predictive Modeling Applied Modeling, Module 4You will use your portfolio project dataset for all assignments this sprint. AssignmentComplete these tasks for your project, and document your work.- [ ] Continue to iterate on your project: data cleaning, exploratory visualization, feature engineering, modeling.- [ ] Make a Shapley force plot to explain at least 1 individual prediction.- [ ] Share at least 1 visualization on Slack.(If you haven't completed an initial model yet for your portfolio project, then do today's assignment using your Tanzania Waterpumps model.) Stretch Goals- [ ] Make Shapley force plots to explain at least 4 individual predictions.    - If your project is Binary Classification, you can do a True Positive, True Negative, False Positive, False Negative.    - If your project is Regression, you can do a high prediction with low error, a low prediction with low error, a high prediction with high error, and a low prediction with high error.- [ ] Use Shapley values to display verbal explanations of individual predictions.- [ ] Use the SHAP library for other visualization types.The [SHAP repo](https://github.com/slundberg/shap) has examples for many visualization types, including:- Force Plot, individual predictions- Force Plot, multiple predictions- Dependence Plot- Summary Plot- Summary Plot, Bar- Interaction Values- Decision PlotsWe just did the first type during the lesson. The [Kaggle microcourse](https://www.kaggle.com/dansbecker/advanced-uses-of-shap-values) shows two more. Experiment and see what you can learn! Links- [Kaggle / Dan Becker: Machine Learning Explainability — SHAP Values](https://www.kaggle.com/learn/machine-learning-explainability)- [<NAME>: Interpretable Machine Learning — Shapley Values](https://christophm.github.io/interpretable-ml-book/shapley.html)- [SHAP repo](https://github.com/slundberg/shap) & [docs](https://shap.readthedocs.io/en/latest/)<jupyter_code>%cd /Users/ezb/Jupyter\ notebooks/datasets
import pandas as pd 

df = pd.read_csv('DC_Properties.csv', low_memory=False)
def datawrangle(x):
    # make copy
    x = x.copy()

    # drop X and Y because they're the same as LONGITUDE and LATITUDE
    x = x.drop(columns=['X', 'Y'])

    # drop redundant or useless columns
    x = x.drop(columns=['Unnamed: 0', 'STATE', 'CITY', 'SOURCE', 'FULLADDRESS', 'NATIONALGRID'])

    # drop null values in price since that's our target
    x = x.dropna(subset=['PRICE'], how='any')

    return x

df = datawrangle(df)
from sklearn.metrics import mean_absolute_error, r2_score, mean_squared_error
from sklearn.impute import SimpleImputer
from sklearn.ensemble import RandomForestRegressor
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
import category_encoders as ce

trainval, test = train_test_split(
    df, test_size=20000, random_state=0
)

train, val = train_test_split(
    trainval, test_size=20000, 
    random_state=0
)

target = 'PRICE'

x_train = train.drop(columns=target)
y_trian = train[target]
x_val = val.drop(columns=target)
y_val = val[target]
x_test = test.drop(columns=target)
y_test = test[target]

x_train.shape, y_train.shape , x_val.shape, y_val.shape, x_test.shape, y_test.shape
processor = make_pipeline(
    ce.OrdinalEncoder(),
    SimpleImputer(strategy='median')
)

model = RandomForestRegressor(
    n_estimators=100,
    random_state=0,
    n_jobs=-1
)

x_train_processed = processor.fit_transform(x_train)
x_val_processed = processor.fit_transform(x_val)

model.fit(x_train_processed, y_train)
import numpy as np

y_pred_train = model.predict(x_train_processed)
y_pred_val = model.predict(x_val_processed)
    
print('Train Root Mean Squared Error:', np.sqrt(mean_squared_error(y_train, y_pred_train)))
print('Validation Root Mean Square Error:', np.sqrt(mean_squared_error(y_val, y_pred_val)))
print('Train Mean Absolute Error:', mean_absolute_error(y_train, y_pred_train))
print('Validation Mean Absolute Error:', mean_absolute_error(y_val, y_pred_val))
print('Train R^2 Score:', r2_score(y_train, y_pred_train))
print('Validation R^2 Score:', r2_score(y_val, y_pred_val))
val.sort_values('PRICE', ascending=False).head()
import shap

row = x_val.loc[[129078]]

explainer = shap.TreeExplainer(model)
row_processed = processor.transform(row)
shap_values = explainer.shap_values(row_processed)

shap.initjs()
shap.force_plot(
    base_value=explainer.expected_value, 
    shap_values=shap_values, 
    features=row
)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09711684370257967, 'fraction_numerical': 0.007370474745285064, 'mean_word_length': 5.435146443514644, 'pattern_counts': {'":': 0, '<': 6, '<?xml version=': 0, '>': 6, 'https://': 6, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '320773', 'license': 'MIT', 'n_tokens_mistral': 1503, 'n_tokens_neox': 1409, 'n_words': 434, 'path': 'module4/assignment_applied_modeling_4.ipynb', 'repo_name': 'ezorigo/DS-Unit-2-Applied-Modeling', 'size': '4623'}