<jupyter_start><jupyter_text>babyCARDS classifier training and evaluation This script loads the **training, validation and testing** data sourced from various **blogs, twitter, facebook and newspapers** and trains a classifier to classify these into **convinced** (0) and **contrarian** (1).  Packages<jupyter_code>from preprocess import denoise_text
from logistic import fit_logistic_classifier
import pandas as pd
from sklearn import metrics
import pickle
import warnings
warnings.filterwarnings("ignore", category=UserWarning, module='bs4')<jupyter_output><empty_output><jupyter_text>Load and preprocess the training, validation and testing data<jupyter_code>%%time
# Load the data
train = pd.read_csv('../data/babyCARDStrain.csv', lineterminator='\n')
valid = pd.read_csv('../data/babyCARDSvalid.csv', lineterminator='\n')
test = pd.read_csv('../data/babyCARDStest.csv', lineterminator='\n')
# Preprocess the text
train['text'] = train['text'].astype(str).apply(denoise_text)
valid['text'] = valid['text'].astype(str).apply(denoise_text)
test['text'] = test['text'].astype(str).apply(denoise_text)<jupyter_output>CPU times: user 5min 4s, sys: 8.06 s, total: 5min 12s
Wall time: 5min 12s<jupyter_text>Train the logistic classifier<jupyter_code>%%time
# Fit the classifier
data = train[['text','label']].values.tolist()
model = fit_logistic_classifier(data)<jupyter_output>CPU times: user 1h 13min 9s, sys: 1min 42s, total: 1h 14min 52s
Wall time: 17min 50s<jupyter_text>Encode and vectorize the validation and testing data<jupyter_code>%%time
# Encode the labels
y_valid = model['label_encoder'].transform(valid.label)
y_test = model['label_encoder'].transform(test.label)
# Vectorize the text
X_valid = model['vectorizer'].transform(valid.text)
X_test = model['vectorizer'].transform(test.text)<jupyter_output>CPU times: user 2min 8s, sys: 1.67 s, total: 2min 9s
Wall time: 2min 9s<jupyter_text>Inspect the classifier performance<jupyter_code>print('Validation data:\n')
print(metrics.classification_report(y_valid,model['clf'].predict(X_valid)))
print('\n\nTesting data (unseen bloggers, twitter accounts, facebook accounts and newspaper articles):\n')
print(metrics.classification_report(y_test,model['clf'].predict(X_test)))<jupyter_output>Validation data:

              precision    recall  f1-score   support

           0       0.87      0.90      0.88    509797
           1       0.91      0.88      0.90    597768

    accuracy                           0.89   1107565
   macro avg       0.89      0.89      0.89   1107565
weighted avg       0.89      0.89      0.89   1107565



Testing data (unseen bloggers, facebook users, twitter users and newspaper articles):

              precision    recall  f1-score   support

           0       0.71      0.90      0.80    180897
           1       0.90      0.70      0.79    221956

    accuracy                           0.79    402853
   macro avg       0.80      0.80      0.79    402853
weighted avg       0.81      0.79      0.79    402853<jupyter_text>Save the classifier<jupyter_code># Save the classifier
pkl_filename = "../classifiers/babyCARDS.pkl"
with open(pkl_filename, 'wb') as file:
    pickle.dump(model, file)
# Load classifier from file
# with open(pkl_filename, 'rb') as file:
#     model = pickle.load(file)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10997288339861404, 'fraction_numerical': 0.056040976197649896, 'mean_word_length': 3.7226173541963017, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '276178', 'license': 'Apache-2.0', 'n_tokens_mistral': 1203, 'n_tokens_neox': 1068, 'n_words': 274, 'path': 'code/babyCARDS_train_eval.ipynb', 'repo_name': 'MirjamOdile/babyCARDS', 'size': '3319'}