<jupyter_start><jupyter_text>Plik z modelami do klasyfikacji. Tym razem modele testowałem już tylko na połączeniu wszystkich trzech plików z fingerprintami, ze względnu na to, że w przypadku regresji zawsze w ten sposób orzymywałem najlepsze wyniki. Testowałem również jak model radzi sobie w zależności od ustalonego progu IC50 (progu od którego klasyfikujemy związek jako toksyczny). Podobnie jak dla regresji, większa ilość zmiennych poprawia wyniki modelu, jednak korzystanie tylko z małej części zmiennych jest w stanie osiągać dość podobne wyniki, a przy tym czas wykonania jest zdecydowanie mniejszy. W notebooku przetestowałem sporo modeli płytkich oraz prostą sieć neuronową. Do oceny wyników używałem accuracy oraz f1_score dla mniej zbalansowanych progów IC50. Najlepsze wynik podobnie jak w przypadku regresji otrzymaweł w modelach opartych ns SVM, dla progu IC50=10 000(tyle samo związków toksycznych i nietoksycznych) dochododziłem do accuracy ~80%. W przypadku sieci neuronowej było to nieco mniej. Dokładniejsze wyniki można prześledzić poniżej. Ze względu na to, że testowałem dośc dużo kombinacji modeli/danych/parametrów cały notebook liczy się dość długo. W razie własnych testów polecam wywoływać funkcje z ograniczoną liczbą zmiennych, wtedy powinno działać szybciej. Powinno dać się to zrobić przez usunięcie częsci parametrów w miejscu gdzie wywołuje główne funckje.<jupyter_code>import numpy as np

from xgboost import XGBClassifier
from sklearn.svm import SVC, LinearSVC
from sklearn.neighbors import KNeighborsClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier

from sklearn.model_selection import train_test_split, KFold, GridSearchCV
from sklearn.feature_selection import SelectKBest, f_regression
from sklearn.metrics import accuracy_score, f1_score, precision_score, recall_score

from warnings import filterwarnings

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense

from tensorflow.keras.callbacks import History
from keras.callbacks import LearningRateScheduler
from keras.callbacks import ModelCheckpoint
from keras.callbacks import EarlyStopping
from keras.layers import Dropout
from keras.layers import BatchNormalization

import matplotlib.pyplot as plt

import utils
seed = 1
def SVC_RBF(X, y, cv):
    classifier = SVC(kernel='rbf')
    
    param_grid = {
        'C': [10, 100],
        'gamma':[0.001, 0.01, 0.1, 'auto'],
    }
    
    grid = GridSearchCV(classifier, param_grid, cv=cv, return_train_score=True)
    grid.fit(X, y)
    return grid
def SVC_Poly(X, y, cv):
    classifier = SVC(kernel='poly')
    
    param_grid = {
        'C': [0.01, 1],
        'coef0': [0.01, 1],
        'degree': [2]
    }
    
    grid = GridSearchCV(classifier, param_grid, cv=cv, return_train_score=True)
    grid.fit(X, y)
    return grid
def SVC_Linear(X, y, cv):
    classifier = LinearSVC(loss='squared_hinge', penalty='l2', dual=False, max_iter=50000)
    
    param_grid = {
        'C': [0.1, 1, 10],
    }
    
    grid = GridSearchCV(classifier, param_grid, cv=cv, return_train_score=True)
    grid.fit(X, y)
    return grid
def Logistic(X, y, cv):
    classifier = LogisticRegression(solver='liblinear')
    
    param_grid = {
        'C': [0.01, 0.1, 1, 10, 100, 1000],
    }
    
    grid = GridSearchCV(classifier, param_grid, cv=cv, return_train_score=True)
    grid.fit(X, y)
    return grid
def RandomForest(X, y, cv):
    classifier = RandomForestClassifier(random_state=seed, max_leaf_nodes=500)
    
    param_grid = {
        'n_estimators': [100, 500],
        'max_leaf_nodes': [2, 4, 8, 16]
    }
    
    grid = GridSearchCV(classifier, param_grid, cv=cv, return_train_score=True)
    grid.fit(X, y)
    return grid
def GradientBoosting(X, y, cv):
    classifier = GradientBoostingClassifier(random_state=seed)
    
    param_grid = {
        'n_estimators': [100, 500],
        'learning_rate': [0.01, 0.1]
    }
    
    grid = GridSearchCV(classifier, param_grid, cv=cv, return_train_score=True)
    grid.fit(X, y)
    return grid
def XGBoost(X, y, cv):
    classifier = XGBClassifier(use_label_encoder=False, verbosity=0)
    
    param_grid = {
        'max_depth': [3, 5, 8, 10],
        'learning_rate': [0.001, 0.01],
        'n_estimators': [50, 100],
    }
    
    grid = GridSearchCV(classifier, param_grid, cv=cv, return_train_score=True)
    grid.fit(X, y)
    return grid
def KNearestNeighbors(X, y, cv):
    classifier = KNeighborsClassifier(algorithm='ball_tree')
    
    param_grid = {
        'n_neighbors': [5, 10],
        'leaf_size': [20, 50],
    }
    
    grid = GridSearchCV(classifier, param_grid, cv=cv, return_train_score=True)
    grid.fit(X, y)
    return grid
def perform_classification(X, y):
    
    model_functions = [
    ("SVC RBF", SVC_RBF),
    ("SVC Poly", SVC_Poly),
    ("SVC Linear", SVC_Linear),
    ("Logistic", Logistic),
    ("Random Forest", RandomForest),
    ("Gradient Boosting", GradientBoosting),
    ("XGBoosting", XGBoost),
    ("K Nearest Neighbors", KNearestNeighbors),
    ]
    
    
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20, random_state=seed)

    kfold = KFold(n_splits=5, random_state=seed, shuffle=True)
    
    np.seterr(divide='ignore', invalid='ignore')
    filterwarnings('ignore')

    print(f"{'Classification model:'.ljust(22)}   {'accuracy:'.ljust(8)}   F1:")

    best_accuracy = 0
    best_acc_result = ("None", 0, {})
    
    best_f1 = 0
    best_precision = 0
    best_recall = 0
    best_f1_result = ("None", 0, {})

    for (name, function) in model_functions:
        model = function(X_train, y_train, kfold)
        prediction = model.best_estimator_.predict(X_test)
        accuracy = accuracy_score(y_test, prediction)
        precision = precision_score(y_test, prediction)
        recall = recall_score(y_test, prediction)
        f1 = f1_score(y_test, prediction)
        if (accuracy > best_accuracy):
            best_accuracy = accuracy
            best_acc_result = (name, accuracy, model.best_params_)
        if (f1 > best_f1):
            best_f1 = f1
            best_precision = precision
            best_recall = recall
            best_f1_result = (name, f1, model.best_params_)

        print(f"{name.ljust(22)}   {str(round(accuracy, 6)).ljust(9)}   {str(round(f1, 6)).ljust(9)}   {model.best_params_}")

    print(f"\nBest accuracy\n{best_acc_result[0].ljust(22)}   {str(round(best_acc_result[1], 6)).ljust(9)}   {best_acc_result[2]}\n")
    print(f"\nBest F1\n{best_f1_result[0].ljust(22)}   {str(round(best_f1_result[1], 6)).ljust(9)}   {best_f1_result[2]}")
    print(f"{'    preccision'.ljust(22)}   {best_precision}")
    print(f"{'    recall'.ljust(22)}   {best_recall}")
for perc in [0.9, 0.8, 0.5, 0.3, 0.1, 0.01]:
#     df_hashed = utils.get_hashed_fingerprints(min_perc_used=perc)
#     df_maccsfp = utils.get_MACCSFP_fingerprints(min_perc_used=perc)
#     df_klekota = utils.get_KlekotaRoth_fingerprints(min_perc_used=perc)
    df_mixed = utils.get_mixed_fingerprints(min_perc_used=perc)
    dfs = [
#         (df_hashed, "Hashed Extended Fingerprints"), 
#         (df_maccsfp, "MACCSFP Fingerprints"), 
#         (df_klekota, "Klekota&Roth Fingerprints"), 
        (df_mixed, "Mixed Fingerprints")
    ]

    print(f'\nUsing fetures that are present in at least {perc*100}% of substances.\n')
    for df, title in dfs:
        for IC50_threshold in [500, 5_000, 10_000, 50_000]:
            print(f'IC50 threshold for classification: {IC50_threshold} \n\n')
            df_class = utils.classify_on_IC50(df.copy(), IC50_threshold)
            X = df_class.drop('IC50', axis=1)
            y = df_class['IC50']
            print(f'{round(y[y==1].shape[0]/y.shape[0] * 100, 2)}% of substances classified as toxic, {round(y[y==0].shape[0]/y.shape[0] * 100, 2)}% classified as non toxic.' , '\n')
            print(title, '\n')
            perform_classification(X, y)
            print()
    print('\n\n')<jupyter_output>Preparing files for mixed fingerprints.

Preparing (ready_sets/cardiotoxicity_hERG_MACCSFP.csv) file.
DataFrame base shape: (11504, 167)
Shape after removing wrong values: (10635, 167)
Shape after removing least used features: (10635, 15)
Shape after removing outliers: (10396, 15)

Preparing (ready_sets/cardiotoxicity_hERG_KlekFP.csv) file.
DataFrame base shape: (11504, 4861)
Shape after removing wrong values: (10635, 4861)
Shape after removing least used features: (10635, 9)
Shape after removing outliers: (10396, 9)

Preparing (ready_sets/cardiotoxicity_hERG_ExtFP.csv) file.
DataFrame base shape: (11504, 1025)
Shape after removing wrong values: (10635, 1025)
Shape after removing least used features: (10635, 51)
Shape after removing outliers: (10396, 51)


Using fetures that are present in at least 90.0% of substances.

IC50 threshold for classification: 500 


11.55% of substances classified as toxic, 88.45% classified as non toxic. 

Mixed Fingerprints 

Classification model:    accu[...]<jupyter_text>Tutaj testuje metode SelectKBest do wyboru zmiennych.<jupyter_code># perc values adjusted to compare with previous method
for perc in [0.0119, 0.02, 0.059, 0.133, 0.21, 0.29]:
#     df_hashed = utils.get_hashed_fingerprints(min_perc_used=perc)
#     df_maccsfp = utils.get_MACCSFP_fingerprints(min_perc_used=perc)
#     df_klekota = utils.get_KlekotaRoth_fingerprints(min_perc_used=perc)
    df_mixed = utils.get_mixed_fingerprints(min_perc_used=perc)
    dfs = [
#         (df_hashed, "Hashed Extended Fingerprints"), 
#         (df_maccsfp, "MACCSFP Fingerprints"), 
#         (df_klekota, "Klekota&Roth Fingerprints"), 
        (df_mixed, "Mixed Fingerprints")
    ]

    print(f'\nUsing fetures that are present in at least {perc*100}% of substances.\n')
    for df, title in dfs:
        for IC50_threshold in [500, 5_000, 10_000, 50_000]:
            print(f'IC50 threshold for classification: {IC50_threshold} \n\n')
            df_class = utils.classify_on_IC50(df.copy(), IC50_threshold)
            X = df_class.drop('IC50', axis=1)
            y = df_class['IC50']
            select = int(X.shape[1] * perc)
            print(f'{round(y[y==1].shape[0]/y.shape[0] * 100, 2)}% of substances classified as toxic, {round(y[y==0].shape[0]/y.shape[0] * 100, 2)}% classified as non toxic.' , '\n')
            np.seterr(divide='ignore', invalid='ignore')
            X = SelectKBest(f_regression, k=select).fit_transform(X, y)
            print(title, '\n')
            perform_classification(X, y)
            print()
    print('\n\n')<jupyter_output>Preparing files for mixed fingerprints.

Preparing (ready_sets/cardiotoxicity_hERG_MACCSFP.csv) file.
DataFrame base shape: (11504, 167)
Shape after removing wrong values: (10635, 167)
Shape after removing least used features: (10635, 138)
Shape after removing outliers: (10396, 138)

Preparing (ready_sets/cardiotoxicity_hERG_KlekFP.csv) file.
DataFrame base shape: (11504, 4861)
Shape after removing wrong values: (10635, 4861)
Shape after removing least used features: (10635, 565)
Shape after removing outliers: (10396, 565)

Preparing (ready_sets/cardiotoxicity_hERG_ExtFP.csv) file.
DataFrame base shape: (11504, 1025)
Shape after removing wrong values: (10635, 1025)
Shape after removing least used features: (10635, 1008)
Shape after removing outliers: (10396, 1008)


Using fetures that are present in at least 1.1900000000000002% of substances.

IC50 threshold for classification: 500 


11.55% of substances classified as toxic, 88.45% classified as non toxic. 

Mixed Fingerprints 

Class[...]<jupyter_text>Na koniec przetestowałem jeszcze prostą sieć neuronową<jupyter_code>def perform_NN_classification(X, y):
    history = History()
    model = Sequential()

    size = X.shape[1]
    model.add(Dense(size//2, activation="relu",input_shape=(size,)))
    model.add(BatchNormalization())
    model.add(Dropout(0.5))

    model.add(Dense(size//4, activation="relu"))
    model.add(BatchNormalization())
    model.add(Dropout(0.5))

    model.add(Dense(size//8, activation="relu"))
    model.add(BatchNormalization())
    model.add(Dropout(0.5))

    model.add(Dense(size//16, activation="relu"))
    model.add(BatchNormalization())
    model.add(Dropout(0.5))

    model.add(Dense(size//32, activation="relu"))
    model.add(BatchNormalization())
    model.add(Dropout(0.5))

    model.add(Dense(1,activation="sigmoid"))
    model.summary()


    early_stopping = EarlyStopping(patience=30, monitor="val_loss")
    model.compile(loss='binary_crossentropy', optimizer="Adam", metrics=["accuracy"])

    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20, random_state=seed)

    model.fit(X_train, y_train, validation_data= (X_test, y_test), batch_size=256, epochs=200, validation_split=0.2, callbacks=[early_stopping, history], verbose=0)
    prediction = (model.predict(X_test) > 0.5).astype("int32")
    accuracy = accuracy_score(y_test, prediction)
    f1 = f1_score(y_test, prediction)
    print(f'\nFinal accuracy:   {accuracy}')
    print(f'\nFinal f1 score:   {f1}')
    print()
    
for perc in [0.9, 0.8, 0.5, 0.3, 0.1, 0.01]:
#     df_hashed = utils.get_hashed_fingerprints(min_perc_used=perc)
#     df_maccsfp = utils.get_MACCSFP_fingerprints(min_perc_used=perc)
#     df_klekota = utils.get_KlekotaRoth_fingerprints(min_perc_used=perc)
    df_mixed = utils.get_mixed_fingerprints(min_perc_used=perc)
    dfs = [
#         (df_hashed, "Hashed Extended Fingerprints"), 
#         (df_maccsfp, "MACCSFP Fingerprints"), 
#         (df_klekota, "Klekota&Roth Fingerprints"), 
        (df_mixed, "Mixed Fingerprints")
    ]

    print(f'\nUsing fetures that are present in at least {perc*100}% of substances.\n')
    for df, title in dfs:
        for IC50_threshold in [500, 5_000, 10_000, 50_000]:
            print(f'IC50 threshold for classification: {IC50_threshold} \n\n')
            df_class = utils.classify_on_IC50(df.copy(), IC50_threshold)
            X = df_class.drop('IC50', axis=1)
            y = df_class['IC50']
            print(f'{round(y[y==1].shape[0]/y.shape[0] * 100, 2)}% of substances classified as toxic, {round(y[y==0].shape[0]/y.shape[0] * 100, 2)}% classified as non toxic.' , '\n')
            print(title, '\n')
            perform_NN_classification(X, y)
            print()
    print('\n\n')<jupyter_output>Preparing files for mixed fingerprints.

Preparing (ready_sets/cardiotoxicity_hERG_MACCSFP.csv) file.
DataFrame base shape: (11504, 167)
Shape after removing wrong values: (10635, 167)
Shape after removing least used features: (10635, 15)
Shape after removing outliers: (10396, 15)

Preparing (ready_sets/cardiotoxicity_hERG_KlekFP.csv) file.
DataFrame base shape: (11504, 4861)
Shape after removing wrong values: (10635, 4861)
Shape after removing least used features: (10635, 9)
Shape after removing outliers: (10396, 9)

Preparing (ready_sets/cardiotoxicity_hERG_ExtFP.csv) file.
DataFrame base shape: (11504, 1025)
Shape after removing wrong values: (10635, 1025)
Shape after removing least used features: (10635, 51)
Shape after removing outliers: (10396, 51)


Using fetures that are present in at least 90.0% of substances.

IC50 threshold for classification: 500 


11.55% of substances classified as toxic, 88.45% classified as non toxic. 

Mixed Fingerprints 

Model: "sequential"
_________[...]
{'fraction_non_alphanumeric': 0.11486355091722306, 'fraction_numerical': 0.04887534841511636, 'mean_word_length': 3.8333333333333335, 'pattern_counts': {'":': 0, '<': 10, '<?xml version=': 0, '>': 13, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '236456', 'license': 'Apache-2.0', 'n_tokens_mistral': 6076, 'n_tokens_neox': 5556, 'n_words': 1332, 'path': 'classification.ipynb', 'repo_name': 'mateuszpoleski/umwpl-cardiotoxicity-prediction', 'size': '15427'}