<jupyter_start><jupyter_text>SDMT Validity  --- 1. Introduction The concept of validity was formulated by Kelly (1927, p. 14) who stated that a test is valid if it measures what it claims to measure. For example a test of intelligence should measure intelligence and not something else (such as memory). Who stated that a test is valid if it measures what it claims to measureFor example, the validity of a cognitive test for job performance is the correlation between test scores and, for example, supervisor performance ratings. Such a cognitive test would have predictive validity if the observed correlation were statistically significant.Rather than accuracy, validity refers to the meaning of a test score. There are multiple aspects of validity in psychometric science. Does a low score, for example, represent the presence of neuropsychological dysfunction, a particular disease state such as MS, a high likelihood of brain atrophy or some other marker of cerebral involvement, or perhaps an increased risk of failing at work? Correlations between test scores and other measures [ie validity coefficient r] and comparing differences in the performance from specified samples [eg, MS vs controls; employed vs disabled MS patients] are common methods of investigation.The BICAMS committee decided that the most important aspect of validity for clinical purposes is criterion-related validity, most notably differentiating MS patients from healthy controls. All of the BICAMS tests discriminate well with SDMT most often the most sensitive measure in NP batteriesThe first step is the standardisation of the test, to ensure that test stimuli and administration procedures have face validity and consistent stimulus presentation. The CSCT is presented on a computer in a consistent manner for each session. The instructions are standardised and they appear on the computer screen. The construction of the test was inspired by other well-known IPS measures, such as the digit-symbol subtests of the WAIS-R,14 the SDMT17 that ensures its face validity<jupyter_code>import sys
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import scipy.stats as stats
import warnings
warnings.filterwarnings('ignore')
%matplotlib inline
!pwd
sys.path.insert(0, '/Users/pedrohserrano/neuroscience/utils') #internal packages
import buildms as ms 
import statsms as stms
color_ms = '#386cb0' #blue, This is the color chosen for patients with Multiple Sclerosis
color_hc = 'red'#This is the color chosen for health control participants<jupyter_output><empty_output><jupyter_text>Recommended Step-by-Step Protocol for BICAMS Validation **Step 1, Standardization and Translation of Test Stimuli**  For visual stimuli, determine if there are any semantic associations to stimuli in the culture or language under consideration. For CVLT2 must match new words on word frequency and appropriate similarity ofmeaning. If these parameters cannot be applied scientifically, then ex- pert review andperformance on test by appropriate partici- pants will be utilized to assess translation **Step 2, Standardization and Translation of Test  Instructions**  All information from the testmanual necessary for administration and interpretation must be translated, back translated,and checked for errors. Where possible the translated instructions should be validated againstexpected participant performance in terms of accuracy and error profile. Step 4 will also contribute to the accuracy of the test instructions  **Step 3, Normalization**  Large samples of 150 or more healthy persons are needed for data applicable to persons of all ages and diverse ethnicity. The minimum sample size is 65 healthy volunteers, provided they are group matched on demographics to either a concurrent MS sample, or matched to samples in other published descriptive MS studies. Where possible, the distribution of test scores and error profile of the normalization sample shouldbe examined and compared to published distributions from other language groups **Step 4, Test-Retest Reliability**  Assessment of this criterion can be achieved by evaluatingan MS and/or healthy volunteer sample on two occasions separated by 1–3 weeks. This is thegold standard separation where the question is only test reliability, controlling for maturation effects. A Pearson’s correlation coefficient >0.70 will usually be required.  **Step 5, Criterion-Related Validity**  This step can be pursued in conjunction with Step 3, inthat an MS sample can be compared to a healthy control group that also serves for normalization. To determine if a new Italian BVMTR is sensitive to MS disease state, for example, compare 50 patients to the healthy controls in Step 3. After the study, the investigator adds another 35 healthy volunteers to round out the normalization sample Step 1 and 2 = True (The test it is performed in an application on a controled group  Step 4 = True (The study was driven on the last stories), Step 3= Datapoints count? 246,23Step 5 = Predictive ---  Loading Datasets<jupyter_code>!python ~/neuroscience/utils/create_dataset.py ~/neuroscience/data/processed/dataset.csv
df_measures = pd.read_csv('../data/interim/df_measures.csv', encoding="utf-8")
df_measures_users = pd.read_csv('../data/interim/df_measures_users.csv', encoding="utf-8")<jupyter_output><empty_output><jupyter_text>--- 2. Methods In order to prove validity, we need to be able to predict who is an MS person Predictive validityThis is the degree to which a test accurately predicts a criterion that will occur in the future. For example, a prediction may be made on the basis of a new intelligence test, that high scorers at age 12 will be more likely to obtain university degrees several years later. If the prediction is born out then the test has predictive validity.  2.1 Predict MS Via Classification<jupyter_code>from sklearn.model_selection import train_test_split
from sklearn import preprocessing
from sklearn import metrics
from sklearn.metrics import roc_curve, auc
from sklearn.cross_validation import cross_val_score<jupyter_output>/Users/pedrohserrano/.pyenv/versions/3.4.2/lib/python3.4/site-packages/sklearn/cross_validation.py:41: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.
  "This module will be removed in 0.20.", DeprecationWarning)<jupyter_text>We choose 3 different features that help to distinguish an MS person   - Score of Correct Answers   - Number of Distraction Points   - Median Time of Response in Miliseconds<jupyter_code>X = df_measures[['correct.answers','distract_points','response_ms_med']]
y = df_measures['ms']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33,random_state=4)
# k-Nearest Neighbor
from sklearn.neighbors import KNeighborsClassifier
clf = KNeighborsClassifier()
clf.fit(X_train, y_train)
print(metrics.classification_report(y_test, clf.predict(X_test)))
print('Accuracy: {}'.format(round(clf.score(X_test, y_test),2)))
# Gaussian Naive Bayes
from sklearn.naive_bayes import GaussianNB
clf = GaussianNB()
clf.fit(X_train, y_train)
print(metrics.classification_report(y_test, clf.predict(X_test)))
print('Accuracy: {}'.format(round(clf.score(X_test, y_test),2)))
# Logistic Regression
from sklearn.linear_model import LogisticRegression
clf = LogisticRegression()
clf.fit(X_train, y_train)
print(metrics.classification_report(y_test, clf.predict(X_test)))
print('Accuracy: {}'.format(round(clf.score(X_test, y_test),2)))
plt.figure(figsize=(12, 6))
plt.plot([0, 1], [0, 1], 'k--'); plt.xlim([0.0, 1.0]); plt.ylim([0.0, 1.05])
# Determine the false positive and true positive rates
fpr, tpr, _ = roc_curve(y_test, clf.predict_proba(X_test)[:,1])
roc_auc = auc(fpr, tpr)
plt.plot(fpr, tpr, color='red', label='ROC curve (area = %0.2f)' % roc_auc)
print ('ROC AUC: {}'.format(roc_auc))

plt.xlabel('1-Specificity')
plt.ylabel('Recall, Sensitivity')
plt.legend(loc="lower right")
plt.savefig('../reports/roc_logistic.png')<jupyter_output>ROC AUC: 0.8024425287356323<jupyter_text>2.2 Scores Point Estimate<jupyter_code>## MS Person
import scipy.stats as stats
score_variable = 'correct.answers'
df_ms = df_measures[df_measures['ms']==1][['userId','correct.answers','ms']]
ms_norm = stats.norm(df_ms[score_variable].mean(), df_ms[score_variable].std())
ms_sample = ms_norm.rvs(10000)

df_hc = df_measures[df_measures['ms']==0][['userId','correct.answers','ms']]
hc_norm = stats.norm(df_hc[score_variable].mean(), df_hc[score_variable].std())
hc_sample = hc_norm.rvs(10000)
fig = plt.figure(figsize=(20,10))
N=100 #number of elements randomly grab with replacement
M=100 #number of bootsraps

means, intervals = stms.Intervals(df_ms[score_variable]).create_int(N,M) #ms_sample)
plt.errorbar(x=np.arange(0.1, M, 1), y=means, yerr=[(top-bot)/2 for top,bot in intervals], 
             color=color_ms,fmt='o', label='MS')
plt.hlines(xmin=0, xmax=M, y=df_ms[score_variable].mean(), color=color_ms, linestyles='--')

means, intervals = stms.Intervals(df_hc[score_variable]).create_int(N,M) #hc_sample)
plt.errorbar(x=np.arange(0.1, M, 1), y=means, yerr=[(top-bot)/2 for top,bot in intervals], 
             color=color_hc,fmt='o', label='HC')
plt.hlines(xmin=0, xmax=M, y=df_hc[score_variable].mean(), color=color_hc, linestyles='--')

plt.xlabel('Score Inference')
plt.xlabel('Experiments')
plt.legend(loc='best')<jupyter_output><empty_output><jupyter_text>We found a person needs 16 trials to find out wich group represent its score, MS or HC<jupyter_code>#Run a Bootstrap with 100 Experiments
print('Point estimator on 10k Norms: \n MS: {} (SD {}) \n HC: {} (SD {}) \n Tradeoff: {}'.format(
ms_sample.mean(), ms_sample.std(), hc_sample.mean(), hc_sample.std(), (ms_sample.mean()+hc_sample.mean())/2)) 
imp_value = (ms_sample.mean()+hc_sample.mean())/2
impairment = df_measures[df_measures[score_variable] < imp_value]
not_imp = df_measures[df_measures[score_variable] >= imp_value]
#True Positives
tp = impairment['ms'].sum()
tn = len(not_imp) - not_imp['ms'].sum()
fn = not_imp['ms'].sum()
fp = len(impairment) - impairment['ms'].sum()
accu = (tp+tn)/len(df_measures)
sens = tp/(tp+fn)
spec = tn/ (tn+fp)
print('Performance: \n Accuracy: {} \n Sensitivity: {} \n Specificity: {}'.format(
    round(accu,2), round(sens,2), round(spec,2) ))<jupyter_output>Performance: 
 Accuracy: 0.67 
 Sensitivity: 0.66 
 Specificity: 0.69
{'fraction_non_alphanumeric': 0.09076312806951266, 'fraction_numerical': 0.016244805440120892, 'mean_word_length': 5.836023240800516, 'pattern_counts': {'":': 0, '<': 23, '<?xml version=': 0, '>': 24, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '252935', 'license': 'MIT', 'n_tokens_mistral': 3249, 'n_tokens_neox': 2982, 'n_words': 1200, 'path': 'notebooks/Story 5 - Validity.ipynb', 'repo_name': 'pedrohserrano/Neuroscience', 'size': '10588'}