<jupyter_start><jupyter_text>ROC Analysis to Determine of Optimum Threshold for NHISS In this notebook, ROC curve was analyzed to determine optimum NHISS threshold for classifying INP former and non-former drugs. We included all experimental data we have gathered until now (N=60).  Our criteria for optimum threshold value was the point that maximizes sum of specificity and sensitivity.  sensitivity + specificity = true positive rate - false positive rate.<jupyter_code>import pandas as pd
import numpy as np
import os
import re
from __future__ import print_function, division
import matplotlib.pyplot as plt
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Import experimental data and NHISS values<jupyter_code>df_molecules = pd.read_csv("df_molecules.csv")
df_molecules.head()

df_molecules["Experimental Category"]=None

for i,row in enumerate(df_molecules.iterrows()):
    if df_molecules.ix[i,"Experimental INP Formation"] == "Yes" :
        df_molecules.ix[i, "Experimental Category"] = int(1)
    else:
        df_molecules.ix[i, "Experimental Category"] = int(0)

df_molecules.head()<jupyter_output><empty_output><jupyter_text>1. NHISS Logistic Regression<jupyter_code># independent variable
X_train = df_molecules["NHISS"].reshape(-1, 1)

# dependent classification
y_train = np.array(df_molecules.ix[:,"Experimental Category"], dtype=int)

# train a logistic regression model on the training set
from sklearn.linear_model import LogisticRegression
logreg = LogisticRegression()
logreg.fit(X_train, y_train)
# make class predictions for the training set
y_pred_class = logreg.predict(X_train)

# print first 35 observations and prediction classes
print('True:', y_train[0:35])
print('Pred:', y_pred_class[0:35])
# store the predicted probabilities for class 1
y_pred_prob = logreg.predict_proba(X_train)[:, 1]
y_pred_prob<jupyter_output><empty_output><jupyter_text>2. NHISS ROC Curve Analysis for threshold determination<jupyter_code>from sklearn import metrics

# IMPORTANT: first argument is true values, second argument is predicted probabilities
fpr, tpr, thresholds = metrics.roc_curve(y_train, y_pred_prob)
plt.plot(fpr, tpr)
plt.xlim([-0.01, 1.01])
plt.ylim([-0.01, 1.01])
plt.title('ROC curve for diabetes classifier')
plt.xlabel('False Positive Rate (1 - Specificity)')
plt.ylabel('True Positive Rate (Sensitivity)')
plt.grid(True)
print("TPR:", tpr)
print("FPR:", fpr)
print("Tresholds:", thresholds)
# Optimum threshold criteria is maximum specificity and sensitivity.
sum_of_specificity_and_sensitivity = np.full(thresholds.size, np.nan)

for i, threshold in enumerate(np.nditer(thresholds)):
    # print(i, threshold)
    sensitivity = tpr[i]
    specificity = 1 - fpr[i]
    sum_of_specificity_and_sensitivity[i] = sensitivity + specificity

# Determine optimum threshold based on maximum specificity and sensitivity
optimum_threshold_probability = thresholds[sum_of_specificity_and_sensitivity == max(sum_of_specificity_and_sensitivity)]
print("Optimum threshold probability: ", optimum_threshold_probability[0])

# Let's find the NHISS threshold value that corresponds to threshold probability
NHISS_range = np.arange(min(X_train)[0], max(X_train)[0], 0.01)

for NHISS in NHISS_range:
    probability = logreg.predict_proba(NHISS)[0][1]
    if probability >= optimum_threshold_probability:
        print("NHISS threshold: ", NHISS)
        break<jupyter_output>Optimum threshold probability:  0.762690734246
NHISS threshold:  4.0
{'fraction_non_alphanumeric': 0.09507346585998272, 'fraction_numerical': 0.01584557764333045, 'mean_word_length': 6.3559322033898304, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 17, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '45155', 'license': 'MIT', 'n_tokens_mistral': 1163, 'n_tokens_neox': 1053, 'n_words': 304, 'path': 'NHISS_discrimination_performance_analysis/determining_NHISS_threshold_with_ROC_curve.ipynb', 'repo_name': 'choderalab/IR783-QSNAP', 'size': '3471'}