<jupyter_start><jupyter_text>Bayesian classifiers on the Adult dataset<jupyter_code>%load_ext autoreload
%autoreload 2

from collections import defaultdict

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import networkx as nx
from pgmpy.models import NaiveBayes, BayesianModel
from pgmpy.inference import VariableElimination

from estimators import TreeAugmentedNaiveBayesSearch, BNAugmentedNaiveBayesSearch, ForestAugmentedNaiveBayesSearch<jupyter_output>The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload<jupyter_text>Data reading<jupyter_code>df = pd.read_csv('dataset/adult.data', na_values=[' ?'])
df.head()
df.describe()<jupyter_output><empty_output><jupyter_text>Data pre-processing  Strip leading and trailing spaces from column names and cell values<jupyter_code>df.columns = df.columns.str.strip()
for column in df.columns.values:
    if df[column].dtype == 'object':
        df[column] = df[column].str.strip()
df.columns.values<jupyter_output><empty_output><jupyter_text>Removing redundant or useless features<jupyter_code>useless_features = ['fnlwgt', 'education-num']
df.drop(columns=useless_features, inplace=True)
df.columns.values<jupyter_output><empty_output><jupyter_text>Record variable names and the target variable (`high-income`)<jupyter_code>variables_names = df.columns.values
target_variable = variables_names[-1]
feature_variables = variables_names[:-1]<jupyter_output><empty_output><jupyter_text>Checking for NA values<jupyter_code>np.sum(pd.isnull(df), 0)<jupyter_output><empty_output><jupyter_text>We can drop NA values, since they are only a few, w.r.t. the size of the dataset.<jupyter_code>df = df.dropna()
np.sum(pd.isnull(df), 0)<jupyter_output><empty_output><jupyter_text>Binning continuos variables<jupyter_code>def binning(column, bins_edges, categories=None, right=False):
    '''
    Bin the given pandas Series values into specific categories names associated 
    with numerical bins. If the categories list is not given, natural numbers 
    will be used as categories.
    '''
    if column.dtype == 'object':
        return column
    bins = [(bins_edges[i], bins_edges[i + 1]) for i in range(len(bins_edges) - 1)]
    if categories is None:
        categories = list(range(len(bins)))
    if len(categories) != len(bins):
        raise Exception(
            'The lenght of the bin edges list should be one element more than the lenght of the categories list.'
        )
    _categories = dict(zip(bins, categories))
    indices = np.digitize(column, bins=bins_edges, right=right)
    values = []
    for i in range(column.size):
        values.append(
            _categories[bins[indices[i] - 1]]
        )
    return pd.Series(values, index=column.index)<jupyter_output><empty_output><jupyter_text>The `age` variable will be divided as follows:* `age` < 18: `child`* 18 <= `age` < 30: `young adult`* 30 <= `age` < 60: `adult`* `age` >= 60: `senior`<jupyter_code>_age_bins_edges = [0, 18, 30, 60, 100]
_age_categories = ['child', 'young adult', 'adult', 'senior']
df['age'] = binning(column=df['age'].copy(), bins_edges=_age_bins_edges, categories=_age_categories)
df['age'].describe()<jupyter_output><empty_output><jupyter_text>The `hours-per-week` variable will be divided as follows:* `hours-per-week` < 25: `part time`* 25 <= `hours-per-week` < 40: `full time`* 40 <= `hours-per-week` < 60: `over time`* `hours-per-week` >= 60: `too much time`<jupyter_code>_hours_bins_edges = [0, 25, 40, 60, 100]
_hours_categories = ['part time', 'full time', 'over time', 'too much time']
df['hours-per-week'] = binning(column=df['hours-per-week'].copy(), bins_edges=_hours_bins_edges, categories=_hours_categories)
df['hours-per-week'].describe()<jupyter_output><empty_output><jupyter_text>Discretizing other continuos variables The `capital-gain` and `capital-loss` variables will be quantile-based discretized.<jupyter_code>df['capital-gain'].plot(kind='hist')
df['capital-gain'], capital_gain_bins_edges = pd.qcut(df['capital-gain'], q=100, duplicates='drop', retbins=True)
df['capital-gain'].describe()
df['capital-loss'].plot(kind='hist')
df['capital-loss'], capital_loss_bins_edges = pd.qcut(df['capital-loss'], q=100, duplicates='drop', retbins=True)
df['capital-loss'].describe()<jupyter_output><empty_output><jupyter_text>Train/test split & utilities<jupyter_code>mask = np.random.rand(len(df)) < 0.8
train = df[mask]
test = df[~mask]
test_no_target = test.copy().drop(target_variable, axis=1).reset_index()
print(f"Training set shape: {train.shape}")
print(f"Test set shape: {test.shape}")<jupyter_output>Training set shape: (24168, 13)
Test set shape: (5994, 13)<jupyter_text>Store every possible state for each variable:<jupyter_code>state_names = {name: column.unique() for name, column in df.iteritems()}<jupyter_output><empty_output><jupyter_text>Prediction functions<jupyter_code>def predict(data, inf, target_variable):
    '''
    Given a Dataframe, an inference object and a target variable,
    perform prediction and return the obtained results
    '''
    results = defaultdict(list)
    for _, data_point in data.iterrows():
        if 'index' in data_point:
            del data_point['index']
        result = inf.query(
            variables=[target_variable],
            evidence=data_point.to_dict(),
            show_progress=False,
        )
        values = result.state_names[target_variable]
        for i, val in enumerate(values):
            results[val].append(result.values[i])
    return results
def perf_measure(test_data, predicted_data, positive_class, negative_class):
    '''
    Return the number of true positives, false positives, true negatives and
    false negatives
    '''
    tp, fp, tn, fn = 0, 0, 0, 0
    predicted_column = []
    keys = list(predicted_data.keys())
    for val in zip(*predicted_data.values()):
        max_index = np.argmax(val)
        predicted_column.append(keys[max_index])
    predicted_column = pd.Series(predicted_column, index=test_data.index)
    for actual, pred in zip(test_data, predicted_column): 
        if actual == pred == positive_class:
           tp += 1
        if pred == positive_class and actual != pred:
           fp += 1
        if actual == pred == negative_class:
           tn += 1
        if pred == negative_class and actual != pred:
           fn += 1

    return (tp, fp, tn, fn)
def accuracy(test_data, predicted_data):
    '''
    Return a percentage representing the classification accuracy
    over the test set
    '''
    predicted_column = []
    keys = list(predicted_data.keys())
    for val in zip(*predicted_data.values()):
        max_index = np.argmax(val)
        predicted_column.append(keys[max_index])
    predicted_column = pd.Series(predicted_column, index=test_data.index)
    equality = predicted_column.eq(test_data)
    true_equality = equality[equality == True]
    return (len(true_equality) / len(test_data)) * 100
def precision(test_data, predicted_data, positive_class, negative_class):
    '''
    Return a percentage representing the classification precision
    over the test set
    '''
    tp, fp, _, _ = perf_measure(test_data, predicted_data, positive_class, negative_class)
    return (tp / (tp + fp)) * 100
def recall(test_data, predicted_data, positive_class, negative_class):
    '''
    Return a percentage representing the classification recall
    over the test set
    '''
    tp, _, _, fn = perf_measure(test_data, predicted_data, positive_class, negative_class)
    return (tp / (tp + fn)) * 100
def f_measure(test_data, predicted_data, positive_class, negative_class):
    '''
    Return a percentage representing the classification F-score
    over the test set
    '''
    p = precision(test_data, predicted_data, positive_class, negative_class)
    r = recall(test_data, predicted_data, positive_class, negative_class)
    return 2 * ((p * r) / (p + r))<jupyter_output><empty_output><jupyter_text>Naive Bayes model<jupyter_code>%%time
nb_model = NaiveBayes(
    feature_vars=feature_variables,
    dependent_var=target_variable
)
nx.draw_circular(nb_model, with_labels=True, arrowsize=20, node_size=1000, font_size=10, alpha=0.8)
plt.show()<jupyter_output><empty_output><jupyter_text>Estimate CPDs from data<jupyter_code>nb_model.fit(train)
nb_cpds = nb_model.get_cpds()
if not nb_model.check_model():
    print('The NB model has errors.')
else:
    print('The NB model has no errors.')<jupyter_output>The NB model has no errors.<jupyter_text>Prediction on test data<jupyter_code>%%time
nb_inference = VariableElimination(nb_model)
nb_results = predict(test_no_target, nb_inference, target_variable)
nb_mean_results = {}
for k, v in nb_results.items():
    nb_mean_results[k] = np.nanmean(v)
print(nb_mean_results)
nb_accuracy = accuracy(test[target_variable], nb_results)
print(f'The Naive Bayes model has {round(nb_accuracy, 3)} % accuracy over the test set.')
nb_precision = precision(test[target_variable], nb_results, positive_class='>50K', negative_class='<=50K')
print(f'The Naive Bayes model has {round(nb_precision, 3)} % precision over the test set.')
nb_recall = recall(test[target_variable], nb_results, positive_class='>50K', negative_class='<=50K')
print(f'The Naive Bayes model has {round(nb_recall, 3)} % recall over the test set.')
nb_f_score = f_measure(test[target_variable], nb_results, positive_class='>50K', negative_class='<=50K')
print(f'The Naive Bayes model has {round(nb_f_score, 3)} % F-score over the test set.')<jupyter_output>The Naive Bayes model has 82.799 % accuracy over the test set.
The Naive Bayes model has 63.297 % precision over the test set.
The Naive Bayes model has 79.08 % recall over the test set.
The Naive Bayes model has 70.314 % F-score over the test set.<jupyter_text>TAN (Tree-Augmented Naive Bayes)<jupyter_code>%%time
tan_estimator = TreeAugmentedNaiveBayesSearch(data=df, class_node=target_variable)
tan_structure = tan_estimator.estimate()
tan_model = BayesianModel(tan_structure.edges)
nx.draw_circular(tan_model, with_labels=True, arrowsize=20, node_size=1000, font_size=10, alpha=0.8)
plt.show()
tree_check = tan_structure.copy()
tree_check.remove_node(target_variable)
if nx.is_tree(tree_check):
    print('The TAN model, without the target variable, is a tree.')
else:
    print('The TAN model, without the target variable, is not a tree.')
nx.draw_circular(tree_check, with_labels=True, arrowsize=20, node_size=1000, font_size=10, alpha=0.8)<jupyter_output>The TAN model, without the target variable, is a tree.<jupyter_text>Estimate CPDs from data<jupyter_code>tan_model.fit(train, state_names=state_names)
tan_cpds = tan_model.get_cpds()
if not tan_model.check_model():
    print('The TAN model has errors.')
else:
    print('The TAN model has no errors.')<jupyter_output>The TAN model has no errors.<jupyter_text>Prediction on test data<jupyter_code>%%time
tan_inference = VariableElimination(tan_model)
tan_results = predict(test_no_target, tan_inference, target_variable)
tan_mean_results = {}
for k, v in tan_results.items():
    tan_mean_results[k] = np.nanmean(v)
print(tan_mean_results)
tan_accuracy = accuracy(test[target_variable], tan_results)
print(f'The TAN model has {round(tan_accuracy, 3)} % accuracy over the test set.')
tan_precision = precision(test[target_variable], tan_results, positive_class='>50K', negative_class='<=50K')
print(f'The TAN model has {round(tan_precision, 3)} % precision over the test set.')
tan_recall = recall(test[target_variable], tan_results, positive_class='>50K', negative_class='<=50K')
print(f'The TAN model has {round(tan_recall, 3)} % recall over the test set.')
tan_f_score = f_measure(test[target_variable], tan_results, positive_class='>50K', negative_class='<=50K')
print(f'The TAN model has {round(tan_f_score, 3)} % F-score over the test set.')<jupyter_output>The TAN model has 85.385 % accuracy over the test set.
The TAN model has 76.012 % precision over the test set.
The TAN model has 63.212 % recall over the test set.
The TAN model has 69.024 % F-score over the test set.<jupyter_text>BAN (BN-Augmented Naive Bayes) Since the current implementation of the BAN model estimator is slow and buggy, we report the results obtained in a single run.The estimated structure is given by the following edges:('hours-per-week', 'capital-gain'), ('relationship', 'hours-per-week'), ('age', 'marital-status'), ('marital-status', 'relationship'), ('relationship', 'occupation'), ('occupation', 'sex'), ('occupation', 'workclass'), ('workclass', 'education'), ('education', 'native-country'), ('native-country', 'race'), ('high-income', 'capital-gain'), ('high-income', 'hours-per-week'), ('high-income', 'sex'), ('high-income', 'race'), ('high-income', 'native-country'), ('high-income', 'education'), ('high-income', 'workclass'), ('high-income', 'occupation'), ('high-income', 'relationship'), ('high-income', 'marital-status'), ('high-income', 'age'), ('high-income', 'capital-loss')<jupyter_code>%%time
ban_estimator = BNAugmentedNaiveBayesSearch(data=df, class_node=target_variable)
ban_structure = ban_estimator.estimate()
ban_model = BayesianModel(ban_structure.edges)
nx.draw_circular(ban_model, with_labels=True, arrowsize=20, node_size=1000, font_size=10, alpha=0.8)
plt.show()
tree_check = ban_structure.copy()
tree_check.remove_node(target_variable)
if nx.is_tree(tree_check):
    print('The BAN model, without the target variable, is a tree.')
else:
    print('The BAN model, without the target variable, is not a tree.')
nx.draw_circular(tree_check, with_labels=True, arrowsize=20, node_size=1000, font_size=10, alpha=0.8)<jupyter_output>The BAN model, without the target variable, is not a tree.<jupyter_text>Estimate CPDs from data<jupyter_code>ban_model.fit(train, state_names=state_names)
ban_cpds = ban_model.get_cpds()
if not ban_model.check_model():
    print('The BAN model has errors.')
else:
    print('The BAN model has no errors.')<jupyter_output>The BAN model has no errors.<jupyter_text>Prediction on test data<jupyter_code>%%time
ban_inference = VariableElimination(ban_model)
ban_results = predict(test_no_target, ban_inference, target_variable)
ban_mean_results = {}
for k, v in ban_results.items():
    ban_mean_results[k] = np.nanmean(v)
print(ban_mean_results)
ban_accuracy = accuracy(test[target_variable], ban_results)
print(f'The BAN model has {round(ban_accuracy, 3)} % accuracy over the test set.')
ban_precision = precision(test[target_variable], ban_results, positive_class='>50K', negative_class='<=50K')
print(f'The BAN model has {round(ban_precision, 3)} % precision over the test set.')
ban_recall = recall(test[target_variable], ban_results, positive_class='>50K', negative_class='<=50K')
print(f'The BAN model has {round(ban_recall, 3)} % recall over the test set.')
ban_f_score = f_measure(test[target_variable], ban_results, positive_class='>50K', negative_class='<=50K')
print(f'The BAN model has {round(ban_f_score, 3)} % F-score over the test set.')<jupyter_output>The BAN model has 84.381 % accuracy over the test set.
The BAN model has 69.685 % precision over the test set.
The BAN model has 67.511 % recall over the test set.
The BAN model has 68.581 % F-score over the test set.<jupyter_text>FAN (Forest-Augmented Naive Bayes)<jupyter_code>%%time
fan_estimator = ForestAugmentedNaiveBayesSearch(data=df, class_node=target_variable)
fan_structure = fan_estimator.estimate()
fan_model = BayesianModel(fan_structure.edges)
nx.draw_circular(fan_model, with_labels=True, arrowsize=20, node_size=1000, font_size=10, alpha=0.8)
plt.show()
forest_check = fan_structure.copy()
forest_check.remove_node(target_variable)
if nx.is_forest(forest_check):
    print('The FAN model, without the target variable, is a forest.')
else:
    print('The FAN model, without the target variable, is not a forest.')
nx.draw_circular(forest_check, with_labels=True, arrowsize=20, node_size=1000, font_size=10, alpha=0.8)<jupyter_output>The FAN model, without the target variable, is a forest.<jupyter_text>Estimate CPDs from data<jupyter_code>fan_model.fit(train, state_names=state_names)
fan_cpds = fan_model.get_cpds()
if not fan_model.check_model():
    print('The FAN model has errors.')
else:
    print('The FAN model has no errors.')<jupyter_output>The FAN model has no errors.<jupyter_text>Prediction on test data<jupyter_code>%%time
fan_inference = VariableElimination(fan_model)
fan_results = predict(test_no_target, fan_inference, target_variable)
fan_mean_results = {}
for k, v in fan_results.items():
    fan_mean_results[k] = np.nanmean(v)
print(fan_mean_results)
fan_accuracy = accuracy(test[target_variable], fan_results)
print(f'The FAN model has {round(fan_accuracy, 3)} % accuracy over the test set.')
fan_precision = precision(test[target_variable], fan_results, positive_class='>50K', negative_class='<=50K')
print(f'The FAN model has {round(fan_precision, 3)} % precision over the test set.')
fan_recall = recall(test[target_variable], fan_results, positive_class='>50K', negative_class='<=50K')
print(f'The FAN model has {round(fan_recall, 3)} % recall over the test set.')
fan_f_score = f_measure(test[target_variable], fan_results, positive_class='>50K', negative_class='<=50K')
print(f'The FAN model has {round(fan_f_score, 3)} % F-score over the test set.')<jupyter_output>The FAN model has 84.685 % accuracy over the test set.
The FAN model has 73.358 % precision over the test set.
The FAN model has 63.666 % recall over the test set.
The FAN model has 68.169 % F-score over the test set.<jupyter_text>Results summary<jupyter_code>indexes = ['NB', 'TAN', 'BAN', 'FAN']
accuracies = [nb_accuracy, tan_accuracy, ban_accuracy, fan_accuracy]
accuracy_df = pd.Series(data=accuracies, index=indexes, name='accuracy')
accuracy_df
precisions = [nb_precision, tan_precision, ban_precision, fan_precision]
precision_df = pd.Series(data=precisions, index=indexes, name='precision')
precision_df
recalls = [nb_recall, tan_recall, ban_recall, fan_recall]
recall_df = pd.Series(data=recalls, index=indexes, name='recall')
recall_df
f_scores = [nb_f_score, tan_f_score, ban_f_score, fan_f_score]
f_score_df = pd.Series(data=f_scores, index=indexes, name='f_score')
f_score_df<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.124053134262817, 'fraction_numerical': 0.016741684048743002, 'mean_word_length': 5.882886286361919, 'pattern_counts': {'":': 0, '<': 119, '<?xml version=': 0, '>': 110, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '27', 'dir': 'jupyter-structured-clean-dedup', 'id': '530973', 'license': 'MIT', 'n_tokens_mistral': 6407, 'n_tokens_neox': 6026, 'n_words': 1519, 'path': 'classify.ipynb', 'repo_name': 'Wadaboa/bayesian-net-classifier', 'size': '18218'}