<jupyter_start><jupyter_text>OverviewFeel free to skip to the section labeled **'Use Scenarion to interpret the model'** if you don't care about the data prep and building the model (things that don't use Scenarion)<jupyter_code>import scenarion
import pandas as pd
from sklearn.ensemble import ExtraTreesClassifier

import copy<jupyter_output><empty_output><jupyter_text>Load and Prep the data<jupyter_code>! wget https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data
# Load the data
columns = ['age',
          'work_class',
          'final_weight',
          'educ',
          'educ_num',
          'marital_status',
          'occupation',
          'relationship',
          'race',
          'sex',
          'capital_gain',
          'capital_loss',
          'work_hours',
          'native_country',
          'income']

df = pd.read_csv('adult.data', names=columns)

# view a sample of the DataFrame
df.sample(4)<jupyter_output><empty_output><jupyter_text>Do one hot encoding on categorical variables<jupyter_code># we'll only use a few for demonstration
categorical_variables = ['work_class',
                         'educ',
                         'occupation',
                         'sex']
dummy_cols_by_var = {}

new_df = copy.copy(df)

for col in  categorical_variables:
    dummy_data = pd.get_dummies(df[col], prefix=col)
    dummy_cols_by_var[col] = list(dummy_data.columns)
    
    new_df = pd.concat([new_df, dummy_data], axis=1)<jupyter_output><empty_output><jupyter_text>Train a model<jupyter_code>y_col = 'income'
x_cols = ['age','capital_gain','capital_loss','work_hours']
for var in categorical_variables:
    x_cols.extend(dummy_cols_by_var[var])

train_x = new_df[x_cols]
train_y = new_df[y_col]

model = ExtraTreesClassifier(n_estimators=100, min_samples_leaf=10)
model.fit(train_x, train_y);<jupyter_output><empty_output><jupyter_text>Use Scenarion to interpret the model As seen in other examples, we can look at **continuous variables**:<jupyter_code>col_to_test = 'age'
scenarios_df = scenarion.test_scenarios(model, new_df, x_cols, 
                        feat_to_test=col_to_test, classes_to_show=[' >50K'])<jupyter_output>Testing Scenarios: 100%|██████████| 100/100 [00:03<00:00, 25.37it/s]<jupyter_text>But we can also use Scenarion to investigate **categorical variables**:<jupyter_code>category_columns = dummy_cols_by_var['educ']

scenarios_df = scenarion.test_scenarios(model, new_df, 
            x_cols, category_columns, classes_to_show=[' >50K'], n_samples=200)<jupyter_output>Testing Scenarios: 100%|██████████| 200/200 [00:07<00:00, 25.77it/s]<jupyter_text>**Note:** You can specify the order, or a subset of categorical values, by simply passing in only a subset of the values in the order you want. See below.<jupyter_code>category_columns = ['educ_ HS-grad', 'educ_ Bachelors', 'educ_ Doctorate']
#print(category_columns)
scenarios_df = scenarion.test_scenarios(model, new_df, x_cols, category_columns, 
                                        classes_to_show=[' >50K'],n_samples=200)<jupyter_output>Testing Scenarios: 100%|██████████| 200/200 [00:07<00:00, 26.52it/s]
{'fraction_non_alphanumeric': 0.12205975842339478, 'fraction_numerical': 0.026064844246662427, 'mean_word_length': 4.142156862745098, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '515089', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 1177, 'n_tokens_neox': 1070, 'n_words': 229, 'path': 'example_income.ipynb', 'repo_name': 'Bayer-Group/scenarion', 'size': '3146'}