<jupyter_start><jupyter_text>Using Bedrock ModelAnalyzer<jupyter_code>import pickle

import numpy as np
import pandas as pd
import lightgbm as lgb
from bedrock_client.bedrock.analyzer.model_analyzer import ModelAnalyzer
from bedrock_client.bedrock.analyzer import ModelTypes

from preprocess.constants import FEATURES, TARGET, PROTECTED_FEATURES<jupyter_output><empty_output><jupyter_text>Load model and test datasetHere we use our model artefact trained elsewhere.<jupyter_code>model = pickle.load(open("./data/lgb_model.pkl", "rb"))
test_df = pd.read_parquet("./data/test.gz.parquet")
x_test = test_df[FEATURES]
y_test = test_df[TARGET].values
# Predict
y_pred = model.predict(x_test)<jupyter_output><empty_output><jupyter_text>Explainability and Fairness using ModelAnalyzerWe shall use `bedrock_client.bedrock.analyzer.model_analyzer.ModelAnalyzer` to generate model explainability analysis and fairness metrics locally. You can use it to generate and log them in Bedrock.Refer to [documentation](https://docs.basis-ai.com/guides/explainability-and-fairness) and [API references](https://docs.basis-ai.com/guides/explainability-and-fairness/api-references) for more details.<jupyter_code>analyzer = ModelAnalyzer(model, "credit_risk", model_type=ModelTypes.TREE)

# To set test data used for calculating explainability
analyzer.test_features(x_test)
analyzer.stored_sample_limit(sample_limit=None)  # To output all SHAP values for the test data

# To set fairness configuration, groundtruth labels and model inference results for generating fairness metrics
analyzer.fairness_config(PROTECTED_FEATURES).test_labels(y_test).test_inference(y_pred)

(shap_values, base_values, global_xai_data, fairness_metrics) = analyzer.analyze()<jupyter_output>Setting feature_perturbation = "tree_path_dependent" because no background data was given.
LightGBM binary classifier with TreeExplainer shap values output has changed to a list of ndarray<jupyter_text>Analyzer outputs SHAP valuesArray of SHAP values calculated on provided `test_features`<jupyter_code>shap_values<jupyter_output><empty_output><jupyter_text>Base valueBase value is the average model output calculated on background data<jupyter_code>base_values<jupyter_output><empty_output><jupyter_text>Global XAI dataGlobal XAI returns the mean absolute shap values of each feature, together with its correlation with said feature<jupyter_code>global_xai_data.keys()
global_xai_data["class 1"]["EXT_SOURCE_1"]<jupyter_output><empty_output><jupyter_text>Fairness metricsSince fairness config is passed, the model analyzer also returns the fairness metrics.<jupyter_code>fairness_metrics.keys()
fairness_metrics["CODE_GENDER"]["class 1"]<jupyter_output><empty_output><jupyter_text>Optional: Plotting explainabilityUsing the compute SHAP values and `shap.summary_plot`, we can obtain a preliminary view of the effects of the top features.<jupyter_code>import shap

# We use the SHAP values that correspond to `TARGET=1`, ie shap_values[1]
shap.summary_plot(shap_values[1], x_test, plot_type="bar", max_display=10)
shap.summary_plot(shap_values[1], x_test, max_display=10)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08, 'fraction_numerical': 0.003492063492063492, 'mean_word_length': 8.51963746223565, 'pattern_counts': {'":': 0, '<': 32, '<?xml version=': 0, '>': 32, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '41057', 'license': 'Apache-2.0', 'n_tokens_mistral': 1007, 'n_tokens_neox': 980, 'n_words': 249, 'path': 'credit_risk/doc_modelanalyzer.ipynb', 'repo_name': 'victor-basisai/examples', 'size': '3150'}