<jupyter_start><jupyter_text>Sliding windows and ClassifiersThis is the final procedure of features engineerings. It should be done after the features extraction and after the one-hot-encoding procedure.The end of the notebook has an example of a some classifers which try to predict single porteins  Some libraries<jupyter_code># Add directory root path
import sys
sys.path.append("..") 
#Standard
import pandas as pd
import numpy as np
import time
import warnings
import matplotlib.pyplot as plt
import pickle
from joblib import dump, load
#Sliding windows
from scipy import signal
#Module
from  modules.feature_extraction import *
from  modules.feature_preprocessing import *
from  modules.models import *
#Classifiers
from sklearn.linear_model import LogisticRegression
from sklearn.neural_network import MLPClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.svm import SVC
from sklearn.gaussian_process import GaussianProcessClassifier
from sklearn.gaussian_process.kernels import RBF
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier, AdaBoostClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.discriminant_analysis import QuadraticDiscriminantAnalysis
#Metricsc
from sklearn import metrics
print('Done')<jupyter_output>Done<jupyter_text>Importing DataFrame 'one-hot-enc'<jupyter_code>#Import data
df = pd.read_csv(r'.\..\datasets\one-hot-enc.csv')
#Get data ready
LIP = df.LIP
df.drop(['LIP_SCORE', 'LIP'], inplace = True, axis = 1)
#Elimate NaN
df.loc[df.REL_ASA.isna(), 'REL_ASA'] = df.REL_ASA.mean()
df.head()
df.columns<jupyter_output><empty_output><jupyter_text>Apply sliding windowsUse odd numbers for windows or some extra NaN could be generated<jupyter_code>df_slided = sliding_window(data=df, k=5, sd=1)
df_slided.head()<jupyter_output><empty_output><jupyter_text>Model - Leave-One-Out of ProteinHere we run a cycle that for every protein of the dataset it use that protein as test set and the rest as training. We print Accuracy, Precision, Recall and Confusion Matrix  Build Classifier<jupyter_code>clf = RandomForestClassifier(n_estimators=100)<jupyter_output><empty_output><jupyter_text>Validation phase - LOOCVVery long procedure for some classifiers. Use verbose to see results for each protein (advised if testing the model for the first time)<jupyter_code>results = loo_cv(data = df_slided, clf=clf, 
                 target = LIP,
                 bad_condition=(0.75, 0.75),
                 ign_warnings = True, 
                 get_times = True, verbose = True)<jupyter_output>Ieration number: 1
1cee
Accuracy: 0.971
Balanced Accuracy: 0.95
Precision: 0.935
Recall: 0.915
F1 score: 0.925
[[188   3]
 [  4  43]]
______________________________________________________
Ieration number: 2
1dev
Accuracy: 1.0
Balanced Accuracy: 1.0
Precision: 1.0
Recall: 1.0
F1 score: 1.0
[[194   0]
 [  0  41]]
______________________________________________________
Ieration number: 3
1dow
Accuracy: 1.0
Balanced Accuracy: 1.0
Precision: 1.0
Recall: 1.0
F1 score: 1.0
[[206   0]
 [  0  31]]
______________________________________________________
Ieration number: 4
1fqj
Accuracy: 0.972
Balanced Accuracy: 0.985
Precision: 0.714
Recall: 1.0
F1 score: 0.833
[[321  10]
 [  0  25]]
______________________________________________________
Ieration number: 5
1g3j
Accuracy: 1.0
Balanced Accuracy: 1.0
Precision: 1.0
Recall: 1.0
F1 score: 1.0
[[522   0]
 [  0  41]]
______________________________________________________
Ieration number: 6
1hrt
Accuracy: 0.677
Balanced Accuracy: 0.58
Precision: 0.672
Re[...]<jupyter_text>Get some results of the model<jupyter_code>print('The avarage balanced accuracy is {}\n'.format(results[1]))
print('The avarage f1 score is {}\n'.format(results[3]))
print('The proteins which gave bad results are:\n{}\n'.format(results[4]))
print('Number of proteins which gave bad results: {}\n'.format(len(results[4])))<jupyter_output>The avarage balanced accuracy is 0.9338611111111111

The avarage f1 score is 0.8940555555555556

The proteins which gave bad results are:
[('1hrt', 0.58, 0.788), ('1i7w', 1.0, 0.0), ('1kil', 0.409, 0.581), ('1rf8', 0.5, 0.0), ('1sc5', 0.762, 0.638), ('1sqq', 0.986, 0.719), ('1ymh', 0.668, 0.782), ('2a6q', 0.715, 0.889), ('1mv0', 0.769, 0.7), ('1ozs', 0.876, 0.727), ('1j2x', 0.49, 0.292), ('2b3g', 0.906, 0.743)]

Number of proteins which gave bad results: 12<jupyter_text>Example of testing for 1cee<jupyter_code>X_train = np.array(df_slided.iloc[238:, 3:])
y_train = np.array(LIP[238:])
X_test = np.array(df_slided.iloc[:238, 3:])
y_test = np.array(LIP[:238])

clf.fit(X_train, y_train)
metrics.balanced_accuracy_score(y_test, clf.predict(X_test))<jupyter_output><empty_output><jupyter_text>Features Selection<jupyter_code>clf.fit(np.array(df_slided.iloc[:, 3:]), LIP)
if type(clf) == RandomForestClassifier:    
    feat_imp = []
    for i,j in zip(clf.feature_importances_, df_slided.columns[3:]):
        value.append(i)
        names.append(j)
        feat_imp.append((i, j))

feat_imp = sorted(feat_imp, reverse=True)
feat_imp
ar, ind = [i[0] for i in feat_imp[0:15]], [i[1] for i in feat_imp[0:15]]
fig, ax = plt.subplots(1, 1, figsize = (25, 7))
_ = ax.bar(ind, ar)<jupyter_output><empty_output><jupyter_text>New DataSet with less features<jupyter_code>low_feat = list(df_slided.columns[47:64])
df_new = df_slided.copy()
df_new.drop(low_feat, axis = 1, inplace = True)
df_new.head()<jupyter_output><empty_output><jupyter_text>Re-test the model to check if eliminating some features didn't compromize the accuracy<jupyter_code>clf = RandomForestClassifier(n_estimators=100)

results = loo_cv(data = df_new, clf=clf, 
                 target = LIP,
                 bad_condition=(0.75, 0.75),
                 ign_warnings = True, 
                 get_times = True, verbose = True)
print('The avarage balanced accuracy is {}\n'.format(results[1]))
print('The avarage f1 score is {}\n'.format(results[3]))
print('The proteins which gave bad results are:\n{}\n'.format(results[4]))
print('Number of proteins which gave bad results: {}\n'.format(len(results[4])))<jupyter_output>The avarage balanced accuracy is 0.9329444444444444

The avarage f1 score is 0.8942777777777778

The proteins which gave bad results are:
[('1hrt', 0.583, 0.804), ('1i7w', 1.0, 0.0), ('1kil', 0.44, 0.584), ('1rf8', 0.5, 0.0), ('1sc5', 0.733, 0.58), ('1sqq', 0.986, 0.719), ('1ymh', 0.487, 0.735), ('2a6q', 0.691, 0.963), ('1mv0', 0.769, 0.7), ('1ozs', 0.876, 0.727), ('1j2x', 0.566, 0.337)]

Number of proteins which gave bad results: 11<jupyter_text>Train the model with all the data available and save it<jupyter_code>clf = RandomForestClassifier(n_estimators=100)
clf.fit(df_new.iloc[:, 3:], LIP)
dump(clf, 'random_forest.joblib')<jupyter_output><empty_output><jupyter_text>Raw n_estimator and sliding size search<jupyter_code>df_slided = sliding_window(data=df, k=j, sd=1)
low_feat = list(df_slided.columns[47:64])
df_new = df_slided.copy()
df_new.drop(low_feat, axis = 1, inplace = True)

n_estimator = [80, 100, 120, 140] 
win_size = [3, 5, 7]
for i in n_estimator:
    for j in win_size:
        if (i == 80) and (j == 3):
            continue
        
        clf = RandomForestClassifier(n_estimators=i)
        results = loo_cv(data = df_new, clf=clf, 
                         target = LIP,
                         bad_condition=(0.75, 0.75),
                         ign_warnings = True, 
                         get_times = True, verbose = False)
        
        print('-------------------------------------------------------------------------')
        print('-------------------------------------------------------------------------\n')
        print('MODEL: n_estimator = {}; sliding_size = {}\n'.format(i, j))
        print('The avarage balanced accuracy is {}\n'.format(results[1]))
        print('The avarage f1 score is {}\n'.format(results[3]))
        print('Number of proteins which gave bad results: {}\n'.format(len(results[4])))
        print('-------------------------------------------------------------------------')
        print('-------------------------------------------------------------------------\n')<jupyter_output>Time taken: 231.5304365158081

-------------------------------------------------------------------------
-------------------------------------------------------------------------

MODEL: n_estimator = 80; sliding_size = 5

The avarage balanced accuracy is 0.937236111111111

The avarage f1 score is 0.8977777777777778

Number of proteins which gave bad results: 9

-------------------------------------------------------------------------
-------------------------------------------------------------------------
{'fraction_non_alphanumeric': 0.1822462103812586, 'fraction_numerical': 0.06419384474046853, 'mean_word_length': 3.19306692344728, 'pattern_counts': {'":': 0, '<': 44, '<?xml version=': 0, '>': 44, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 6, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '295694', 'license': 'MIT', 'n_tokens_mistral': 3283, 'n_tokens_neox': 2898, 'n_words': 769, 'path': 'notebooks/sliding_windows_and_models.ipynb', 'repo_name': 'f-grimaldi/Linear-Interactive-Peptides-LIPs-Predictor', 'size': '8708'}