<jupyter_start><jupyter_text>Census Income Dataset AnalysisLink: https://archive.ics.uci.edu/ml/datasets/adultPredict whether income exceeds $50K/yr based on census data. Also known as "Census Income" dataset.<jupyter_code>import pandas as pd
from sklearn.datasets import load_boston
from sklearn.model_selection import train_test_split
from xtoy import Toy 

df = pd.read_csv(
    "http://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data",
    header=None)
df.columns = [
    "Age", "WorkClass", "fnlwgt", "Education", "EducationNum",
    "MaritalStatus", "Occupation", "Relationship", "Race", "Gender",
    "CapitalGain", "CapitalLoss", "HoursPerWeek", "NativeCountry", "Income"
]
# df = df.sample(frac=0.01, random_state=1)
train_cols = df.columns[0:-1]
label = df.columns[-1]
X = df[train_cols]
y = df[label].apply(lambda x: 0 if x == " <=50K" else 1)
toy = Toy()
toy.fit(X,y)<jupyter_output><empty_output><jupyter_text>Model Performance<jupyter_code>from interpret import show
from interpret.perf import ROC

blackbox_perf = ROC(toy.predict_proba).explain_perf(X,y, name='toy')
show(blackbox_perf)<jupyter_output><empty_output><jupyter_text>Which variable is most important?Using MorrisSensitivity to answer this questionhttps://www.sciencedirect.com/science/article/pii/S0022169412008918<jupyter_code>from interpret.blackbox import MorrisSensitivity
trans_df = pd.DataFrame(data=toy.featurizer.transform(X).A, columns=toy.feature_names_)
sensitivity = MorrisSensitivity(predict_fn=toy.best_evo.predict_proba, data=trans_df)
sensitivity_global = sensitivity.explain_global(name="Global Sensitivity")
show(sensitivity_global)
print('Why Does this person displayed below earn less than 50k dollars?')
display(X.head(1))
print('Let\'s use shap value to explain our prediction')
from interpret.blackbox import ShapKernel
import numpy as np

background_val = np.median(toy.featurizer.transform(X).A, axis=0).reshape(1, -1)
shap = ShapKernel(predict_fn=toy.best_evo.predict_proba, data=background_val, feature_names=toy.feature_names_)
from ipywidgets import IntProgress
shap_local = shap.explain_local(toy.featurizer.transform(X).A[0:1], y[0:1], name='SHAP')
show(shap_local)<jupyter_output><empty_output><jupyter_text>The Above Analysis suggest that while Education Number is 13, Education has the word 'bachelors' and his age is 39, his capital gain is just 2174<jupyter_code>df[df['CapitalGain'] <= 2200]['Income'].value_counts()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12298387096774194, 'fraction_numerical': 0.02056451612903226, 'mean_word_length': 7.353535353535354, 'pattern_counts': {'":': 0, '<': 19, '<?xml version=': 0, '>': 17, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '50978', 'license': 'MIT', 'n_tokens_mistral': 909, 'n_tokens_neox': 848, 'n_words': 180, 'path': 'Explain_Census_Income/Interpret.ipynb', 'repo_name': 'sethuiyer/mlhub', 'size': '2480'}