<jupyter_start><jupyter_text>This notebook illustrates every step of the process of training models for ic labeling.<jupyter_code>import matplotlib.pyplot as plt
import numpy as np
import pandas as pd

%matplotlib inline<jupyter_output><empty_output><jupyter_text>First of all, we need the dataset. We can download and it using the following commands.<jupyter_code>!curl -O http://alice.adase.org/media/downloads/dataset_Kids_v0.2_TLiCoxo.zip
!unzip dataset_Kids.zip<jupyter_output><empty_output><jupyter_text>Now, we load the dataset. For this we use the special function, which reads the csv-files from the directory and puts the data in a convenient format.<jupyter_code>from alice_ml.preprocessing import load_dataset

path = 'dataset_Kids'

data, raw_annotations = load_dataset(path, preload=False)<jupyter_output><empty_output><jupyter_text>Now, we calculate features for each ic in the dataset.<jupyter_code>from alice_ml.features import build_feature_df

features = build_feature_df(data, default=True, custom_features={})
features<jupyter_output><empty_output><jupyter_text>Targets are calculated as the average of the answers of multiple experts for each ic.<jupyter_code>def get_target_distribution(annotations):
    return annotations.groupby('ic_id').mean().apply(lambda x: x.value_counts())

get_target_distribution(raw_annotations)<jupyter_output><empty_output><jupyter_text>For now we do not want to distinguish between eye movement types, so we unite them under `'flag_eyes'`. We also presume the positive label for `'flag_alpha'` or `'flag_mu'` implies the positive label for `'flag_brain'`. Finally, since we do not have sufficient data for distinguishing line noise, we will not consider this flag.<jupyter_code>def propogate_labels(annotations):
    return annotations.assign(flag_brain=annotations[['flag_brain', 'flag_alpha', 'flag_mu']].any(axis=1)) \
                      .drop(columns=['flag_line_noise'])

annotations = propogate_labels(raw_annotations)
get_target_distribution(annotations)<jupyter_output><empty_output><jupyter_text>In order to get binary labels we apply threshold to these averages. By default, the each label has weight $\frac{1}{n}$, where $n$ is the number of labels given to the ic by this expert. We set the threshold to $0.33$ For flags `alpha` and `mu` we choose a simpler approach: each label has weight $1$. The threshold in this case is equal to $0.5$.<jupyter_code>from alice_ml.preprocessing import build_target_df
targets = build_target_df(annotations, weights='uniform', strategy='mean', threshold=0.33)

targets[['flag_alpha', 'flag_mu', 'flag_heart']] = build_target_df(annotations, ['flag_alpha', 'flag_mu', 'flag_heart'], strategy='majority')

print(targets.sum())
targets = targets.loc[:, (targets.mean(axis=0) > 0.05) & (targets.all(axis=0) < 0.95)]
targets.head()
features.to_csv('featrues_Kids_0.2.csv')
targets.to_csv('target_Kids_0.2.csv')<jupyter_output><empty_output><jupyter_text>For training we use XGboost, LinearSVM, and Logistic Regression with default parameters. We do a stratified train-test split with test size = 0.3.<jupyter_code>from sklearn.linear_model import LogisticRegression
from xgboost import XGBClassifier
from sklearn.svm import SVC
from sklearn.preprocessing import StandardScaler

models = {'LR': LogisticRegression(solver='liblinear', penalty='l2', C=1.0),
          'XGB': XGBClassifier(learning_rate=0.3, max_depth=4, n_estimators=30, eval_metric='logloss', use_label_encoder=False),
          'SVM': SVC(kernel='linear', probability=True)}
n_repeats = 50
from sklearn.model_selection import train_test_split

def repeated_train_test(features, target, model, n_repeats, test_size=0.3, axes=None, random_state=57):
    y_true = []
    y_pred = []
    y_proba = []
    np.random.seed(random_state)

    for _ in range(n_repeats):
        X_train, X_test, y_train, y_test = train_test_split(features, target, test_size=test_size, stratify=target)
        scaler = StandardScaler()
        X_train = scaler.fit_transform(X_train)
        X_test = scaler.transform(X_test)
        model.fit(X_train, y_train)
        y_true.append(y_test)
        y_pred.append(model.predict(X_test))
        y_proba.append(model.predict_proba(X_test)[:, 1])

    return np.array(y_true), np.array(y_pred), np.array(y_proba)

from alice_ml.vis import plot_aggregated_pr_curve, plot_aggregated_roc_curve

def plot_curves_grid(features, targets, models):

    fig_roc, axes_roc = plt.subplots(len(models), len(targets.columns), figsize=(4 * len(targets.columns), 3 * len(models)))
    fig_pr, axes_pr = plt.subplots(len(models), len(targets.columns), figsize=(4 * len(targets.columns), 3 * len(models)))

    for (model_name, model), row_roc, row_pr in zip(models.items(), axes_roc, axes_pr):
        for flag, ax_roc, ax_pr in zip(targets, row_roc, row_pr):
            y_true, _, y_proba = repeated_train_test(features, targets[flag], model, n_repeats)
            plot_aggregated_roc_curve(y_true, y_proba, ax_roc)
            plot_aggregated_pr_curve(y_true, y_proba, ax_pr)

        row_roc[0].set_ylabel(model_name, fontsize=28, labelpad=16)
        row_pr[0].set_ylabel(model_name, fontsize=28, labelpad=16)

        row_roc[-1].set_ylabel('True positive rate', fontsize=18)
        row_pr[-1].set_ylabel('Precision', fontsize=18)
        row_roc[-1].yaxis.set_label_position('right')
        row_pr[-1].yaxis.set_label_position('right')

    for target_name, ax_roc, ax_pr in zip(targets, axes_roc[0], axes_pr[0]):
        ax_roc.set_title(target_name[5:].capitalize(), fontsize=28, pad=16)
        ax_pr.set_title(target_name[5:].capitalize(), fontsize=28, pad=16)

    for ax_roc, ax_pr in zip(axes_roc[-1], axes_pr[-1]):
        ax_roc.set_xlabel('False positive rate', fontsize=18)
        ax_pr.set_xlabel('Recall', fontsize=18)

    fig_roc.subplots_adjust(wspace=0.1, hspace=0.1)
    fig_pr.subplots_adjust(wspace=0.1, hspace=0.1)

    for row_roc, row_pr in zip(axes_roc[:-1], axes_pr[:-1]):
        for ax_roc, ax_pr in zip(row_roc, row_pr):
            plt.setp(ax_roc.get_xticklabels(), visible=False)
            plt.setp(ax_pr.get_xticklabels(), visible=False)

    for row_roc, row_pr in zip(axes_roc, axes_pr):
        for ax_roc, ax_pr in zip(row_roc[:-1], row_pr[:-1]):
            plt.setp(ax_roc.get_yticklabels(), visible=False)
            plt.setp(ax_pr.get_yticklabels(), visible=False)

    return fig_roc, fig_pr
fig_roc, fig_pr = plot_curves_grid(features, targets, models)
from sklearn.metrics import auc, roc_curve, precision_recall_curve

def get_aucs(true, proba):
    roc_aucs, pr_aucs = [], []
    for true, proba in zip(y_true, y_proba):
        fpr, tpr, _ = roc_curve(true, proba, drop_intermediate=False)
        prec, rec, _ = precision_recall_curve(true, proba)
        roc_aucs.append(auc(fpr, tpr))
        pr_aucs.append(auc(rec, prec))
    return roc_aucs, pr_aucs
from itertools import product
from sklearn.metrics import f1_score

roc_aucs = pd.DataFrame(index=models, columns=targets.columns, dtype=float)
pr_aucs = pd.DataFrame(index=models, columns=targets.columns, dtype=float)
f1_scores = pd.DataFrame(index=models, columns=targets.columns, dtype=float)

for (model_name, model), flag in product(models.items(), targets):
        y_true, y_pred, y_proba = repeated_train_test(features, targets[flag], model, n_repeats)
        roc_auc_values, pr_auc_values = get_aucs(y_true, y_proba)
        f1_score_values = [f1_score(true, pred) for true, pred in zip(y_true, y_pred)]
        roc_aucs.loc[model_name, flag] = np.mean(roc_auc_values)
        pr_aucs.loc[model_name, flag] = np.mean(pr_auc_values)
        f1_scores.loc[model_name, flag] = np.mean(f1_score_values)

roc_aucs
pr_aucs
pr_aucs
f1_scores<jupyter_output><empty_output><jupyter_text>For each flag we choose the model based on ROC-AUC if the classes are balanced and on F1 scores otherwise. Save pretrained models<jupyter_code>import joblib

scaler = StandardScaler()
X_train = scaler.fit_transform(features)
joblib.dump(scaler, 'scaler.joblib')

for flag in targets.columns:
    model = LogisticRegression(solver='liblinear', penalty='l2', C=1.0)
    model.fit(X_train, targets[flag])
    joblib.dump(model, 'lr_' + flag + '.joblib')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1147838721394842, 'fraction_numerical': 0.012713403559752997, 'mean_word_length': 5.150409530900968, 'pattern_counts': {'":': 0, '<': 38, '<?xml version=': 0, '>': 38, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '299485', 'license': 'Apache-2.0', 'n_tokens_mistral': 2966, 'n_tokens_neox': 2844, 'n_words': 636, 'path': 'Basic Pipeline.ipynb', 'repo_name': 'lellight/alice-eeg-ml', 'size': '8259'}