<jupyter_start><jupyter_text>Shapash in Jupyter - OverviewWith this tutorial you:Understand how Shapash works in Jupyter Notebookwith a simple use caseContents:- Build a Regressor- Compile Shapash SmartExplainer- Display global and local explanability- Export local summarized explainability with to_pandas method- Save Shapash object in pickle fileData from Kaggle [House Prices](https://www.kaggle.com/c/house-prices-advanced-regression-techniques/data)<jupyter_code>import pandas as pd
from category_encoders import OrdinalEncoder
from lightgbm import LGBMRegressor
from sklearn.model_selection import train_test_split<jupyter_output><empty_output><jupyter_text>Building Supervized Model<jupyter_code>from shapash.data.data_loader import data_loading
house_df, house_dict = data_loading('house_prices')
y_df=house_df['SalePrice'].to_frame()
X_df=house_df[house_df.columns.difference(['SalePrice'])]
house_df.head()<jupyter_output><empty_output><jupyter_text>Encoding Categorical Features<jupyter_code>from category_encoders import OrdinalEncoder

categorical_features = [col for col in X_df.columns if X_df[col].dtype == 'object']

encoder = OrdinalEncoder(
    cols=categorical_features,
    handle_unknown='ignore',
    return_df=True).fit(X_df)

X_df=encoder.transform(X_df)<jupyter_output><empty_output><jupyter_text>Train / Test Split<jupyter_code>Xtrain, Xtest, ytrain, ytest = train_test_split(X_df, y_df, train_size=0.75, random_state=1)<jupyter_output><empty_output><jupyter_text>Model Fitting<jupyter_code>regressor = LGBMRegressor(n_estimators=200).fit(Xtrain,ytrain)
y_pred = pd.DataFrame(regressor.predict(Xtest),columns=['pred'],index=Xtest.index)<jupyter_output><empty_output><jupyter_text>Understand my model with shapash  Declare and Compile SmartExplainer<jupyter_code>from shapash.explainer.smart_explainer import SmartExplainer
xpl = SmartExplainer(features_dict=house_dict) # Optional parameter, dict specifies label for features name 
xpl.compile(
    x=Xtest,
    model=regressor,
    preprocessing=encoder, # Optional: compile step can use inverse_transform method
    y_pred=y_pred # Optional
)<jupyter_output>Backend: Shap TreeExplainer<jupyter_text>Display features importance<jupyter_code>xpl.plot.features_importance()<jupyter_output><empty_output><jupyter_text>Focus on a specific subsetYou can use the features_importance method to compare the contribution of features of a subset to the global features importance<jupyter_code>subset = [ 168, 54, 995, 799, 310, 322, 1374,
          1106, 232, 645, 1170, 1229, 703, 66,  
          886, 160, 191, 1183, 1037, 991, 482,  
          725, 410, 59, 28, 719, 337, 36]
xpl.plot.features_importance(selection=subset)<jupyter_output><empty_output><jupyter_text>Understand how a feature contributes- The contribution_plot allows to analyse how one feature affects prediction- Type of plot depends on the type of features- You can use feature name, feature label or feature number to specify which feature you want to analyze<jupyter_code>xpl.plot.contribution_plot("OverallQual")
xpl.plot.contribution_plot("Second floor square feet")<jupyter_output><empty_output><jupyter_text>Display a Summarized but Explicit local explainability  Filter methodUse the filter method to specify how to summarize local explainability There are 4 parameters to customize the summary: - max_contrib : maximum number of criteria to display- threshold : minimum value of the contribution (in absolute value) necessary to display a criterion- positive : display only positive contribution? Negative?(default None)- features_to_hide : list of features you don't want to display<jupyter_code>xpl.filter(max_contrib=8,threshold=100)<jupyter_output><empty_output><jupyter_text>Display local plot, applying your filteryou can use row_num, index or query parameter to specify which prediction you want to explain<jupyter_code>xpl.plot.local_plot(index=560)<jupyter_output><empty_output><jupyter_text>Save your Explainer & Export results  Export your local explanation to pandas DataFrame:to_pandas method has the same parameters as the filter method<jupyter_code>summary_df= xpl.to_pandas(
    max_contrib=3, # Number Max of features to show in summary
    threshold=5000,
)
summary_df.head()<jupyter_output><empty_output><jupyter_text>Save your explainer in Pickle FileYou can save the SmartExplainer Object in a pickle file to make new plots later or launch the WebApp again<jupyter_code>xpl.save('./xpl.pkl')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07839643652561247, 'fraction_numerical': 0.02338530066815145, 'mean_word_length': 7.316666666666666, 'pattern_counts': {'":': 0, '<': 52, '<?xml version=': 0, '>': 52, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '167602', 'license': 'Apache-2.0', 'n_tokens_mistral': 1513, 'n_tokens_neox': 1387, 'n_words': 379, 'path': 'tutorial/tutorial02-Shapash-overview-in-Jupyter.ipynb', 'repo_name': 'amnaabbassi/shapash', 'size': '4490'}