<jupyter_start><jupyter_text>File OverviewThis file analyses the binary classifiers that were generated by the [5_country_classifier.ipynb](./5_country_classifier.ipynb) notebook. Outputs / Assets* proportion of nopes bar chart* top classifier by TPR, PPV, and ACC<jupyter_code>pickle_file = 'full_country_row_sample_rate_1_country_min_sample_cutoff_1000_cutoffs_5000-2500-1000-500-250-100_results.pkl'
import pandas as pd
import pickle
import matplotlib.pyplot as plt
import seaborn as sns

pd.options.display.max_rows = 2000
pd.options.display.max_columns = 1000
pd.options.display.max_colwidth = 255

sns.set(rc={"figure.dpi":300, 'savefig.dpi':300})
sns.set_context('notebook')
sns.set_style("ticks")
with open('./pickles/%s' % pickle_file, 'rb') as f:
    results_df = pickle.load(f)<jupyter_output><empty_output><jupyter_text>Data Evaluation* many of the models degraded into naive classifiers that did not return any positives* there is an inverse relationship between the proportion of respondents from a country and the proportion of models that were naive* there were still some models that produced non-useless - although not impressive - results when measure by TPR and PRR<jupyter_code>from visualise_binary_classifier_results.generate_country_aggregates import generate_country_aggregates

country_aggregates = results_df \
    .reset_index() \
    .groupby('country') \
    .apply(generate_country_aggregates) \
    .sort_values(by='country_proportion', ascending=False)

country_aggregates
(country_aggregates[['proportion_of_naive_classifier', 'country_proportion']]).sort_values(by='country_proportion', ascending=False).plot(kind="bar");
plt.title("Predictive model failure count by country");
# for the blog change naive to nope
nope_data = country_aggregates[['proportion_of_naive_classifier', 'country_proportion']] \
    .rename(columns={ \
        'proportion_of_naive_classifier' : 'Nope Proportion', \
        'country_proportion' : 'Country Proportion' \
    })

(nope_data[['Nope Proportion', 'Country Proportion']]).sort_values(by='Country Proportion', ascending=False).plot(kind="bar");
plt.title("Predictive model failure count by country");
sns.heatmap(country_aggregates[['proportion_of_naive_classifier', 'country_proportion']].corr(), annot=True, fmt=".2f");<jupyter_output><empty_output><jupyter_text>**Most Models are naive due to countryies being moderate minority classes in the data*** most of the models were "naive" in that they always chose NO (not from country X)    * this is a common issue when the [imbalanced data sets](https://developers.google.com/machine-learning/data-prep/construct/sampling-splitting/imbalanced-data). This is however beyond the scope of this introductory work* We can see a clear inverse correlation between `proportion_of_naive_classifier` and `country_proportion`.    * However the plot above shows it is not a linear correlation but more of a step function. India and the USA have no naive, but the rest have high naive proportion with a fair bit of variance in the actual proportion. **Model Performance**Summary: None of these could be deployed in their current current state* The top classifier measured by `True Positive Rate` (TPR) was an SVM classifier for Canada. It had a TPR of 98.9% but at the cost of a low accuracy (11.3%) and a terrible `Positive Predictive Rate` (PPV) of 3.2%.* The top classifier measured by `Positive Predictive Rate` (PPV) was a random forest classifier for India. It had a PPV of 95% and an accuracy of 87% but an abysmal 0.8% true positive rate.* The top classifier measure by `Accuracy` was a classifier that had sub 1% TPR and PPV, showing that for this dataset Accuracy is a bad measure of model performance<jupyter_code>non_useless_classifiers = results_df \
    .query('TP > 0') \
    .drop(columns=['X_columns', 'coefficients'])
print("model count %s" % results_df.shape[0])
print("non useless classifiers (i.e., P > 0): %s" % non_useless_classifiers.shape[0])
non_useless_classifiers.sort_values(by='TPR', ascending=False) \
    .rename(columns = {}) \
    [['model_type', 'column_count', 'P', 'N', 'TPR', 'TNR', 'PPV', 'NPV', 'ACC']] \
    .head(1)
non_useless_classifiers.sort_values(by='PPV', ascending=False).head(10) \
    .rename(columns = {}) \
    [['model_type', 'column_count', 'P', 'N', 'TPR', 'TNR', 'PPV', 'NPV', 'ACC']] \
    .head(1)
non_useless_classifiers.sort_values(by='ACC', ascending=False).head(10) \
    .rename(columns = {}) \
    [['model_type', 'column_count', 'P', 'N', 'TPR', 'TNR', 'PPV', 'NPV', 'ACC']] \
    .head(1)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11623857205050066, 'fraction_numerical': 0.01567261645624728, 'mean_word_length': 5.602011494252873, 'pattern_counts': {'":': 1, '<': 13, '<?xml version=': 0, '>': 15, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '83193', 'license': 'MIT', 'n_tokens_mistral': 1537, 'n_tokens_neox': 1408, 'n_words': 440, 'path': 'notebook/6_visualise_binary_classifier_results.ipynb', 'repo_name': 'kylezeeuwen/ds-stackoverflow2020-analysis', 'size': '4594'}