<jupyter_start><jupyter_text>EXP1a : internal validation  - In this EXP I choose 16 OK from COBRE and 17 fail, 17 maybe  from ADHD200 total of 50 subjects- Pierre and I Rated them **2 times** each- I'll compute kappa intra and inter rater<jupyter_code># Load library
import pandas as pd
import os
import numpy as np
# paths
path_root = "/home/yassinebha/Drive"
path_qc = os.path.join(path_root,"QC_zooniverse/zooqc_internal_validation/EXP1a")<jupyter_output><empty_output><jupyter_text>Calculate Kappa intra rater---------------------------------** PIERRE **<jupyter_code># read CSV's
qc_pierre_1 = pd.read_csv(os.path.join(path_qc,"pierre/pierre_1/T0Rate_the_match_summary.csv"))
qc_pierre_2 = pd.read_csv(os.path.join(path_qc,"pierre/pierre_2/T0Rate_the_match_summary.csv"))
qc_pierre_1[["subject_id","most_likely"]].head()
qc_pierre_2[["subject_id","most_likely"]].head()
# match SubjID with zooniverse ID for Pierre_1 and 2
zoo_id = pd.read_csv(os.path.join(path_qc,"brain-match-subjects_exp1a_raw.csv"))
zoo_id[["subject_id","id"]].head()
# Add subject id Correspondance for Pierre_1
qc_pierre_1_id = pd.merge(qc_pierre_1,zoo_id[["subject_id","id"]], how = 'left', on = 'subject_id')
qc_pierre_1_id[["id","most_likely","subject_id"]].head()
# Add subject id Correspondance for Pierre_2
qc_pierre_2_id = pd.merge(qc_pierre_2,zoo_id[["subject_id","id"]], how = 'left', on = 'subject_id')
qc_pierre_2_id[["id","most_likely","subject_id"]].head()
# Merge Pierre_1 and 2
qc_pierre_1_2 = pd.merge(qc_pierre_1_id[["id","most_likely"]],
                         qc_pierre_2_id[["id","most_likely"]], 
                         how = 'left', on = "id")
qc_pierre_1_2.loc[:,"rating_match_pierre"] = [ row["most_likely_x"] == row["most_likely_y"] for ind,row in qc_pierre_1_2.iterrows()]
qc_pierre_1_2
# number of mismatch intra rater
50 - np.sum(qc_pierre_1_2["rating_match_pierre"])
# Replace OK:1 , Maybe:2 and Fail:3
qc_pierre_1_2.replace({'OK':1,'Maybe':2, 'Fail':3}, inplace=True)
qc_pierre_1_2.head()
# Calculate kappa for pierre 1 and 2
from sklearn import metrics
kappa = metrics.cohen_kappa_score(qc_pierre_1_2[['most_likely_x']],qc_pierre_1_2[['most_likely_y']])
print("The kappa for Pierre is %f " %(kappa))<jupyter_output>The kappa for Pierre is 0.814356<jupyter_text>Calculate Kappa intra rater------------------------------** YASSINE **<jupyter_code># read CSV's
qc_yassine_summary = pd.read_csv(os.path.join(path_qc,"yassine/T0Rate_the_match_summary.csv"))
qc_yassine_detail =  pd.read_csv(os.path.join(path_qc,"yassine/T0Rate_the_match.csv"))
qc_yassine_summary.head()
qc_yassine_detail.head()
# Merge summary and detail for yassine
qc_yassine_raw = pd.merge(qc_yassine_summary,qc_yassine_detail, how = 'left', on = 'subject_id')
qc_yassine = qc_yassine_raw [["subject_id","most_likely","p(OK)","p(Maybe)","p(Fail)"]]
qc_yassine
# number of mismatch intra rater
np.sum(qc_yassine[["p(OK)","p(Maybe)","p(Fail)"]].isin([0.5]))<jupyter_output><empty_output><jupyter_text>Calculate Kappa inter-rater------------------------------** PIERRE VS YASSINE **<jupyter_code># Add subject id Correspondance for yassine
qc_yassine_id = pd.merge(qc_yassine,zoo_id[["subject_id","id"]], how = 'left', on = 'subject_id')
qc_yassine_id.head()
# merge Pierre and Yassine 
qc_pieyass = pd.merge(qc_pierre_1_2,qc_yassine_id, how = 'left', on = 'id')
qc_pieyass
# Replace OK:1 , Maybe:2 and Fail:3
qc_pieyass.replace({'OK':1,'Maybe':2, 'Fail':3}, inplace=True)
## first pierre's rating vs yassine
kappa = metrics.cohen_kappa_score(qc_pieyass[['most_likely_x']],qc_pieyass[['most_likely']])
print("The kappa for Pierre first rating vs yassine is %f " %(kappa))
## second pierre's rating vs yassine
kappa = metrics.cohen_kappa_score(qc_pieyass[['most_likely_y']],qc_pieyass[['most_likely']])
print("The kappa for Pierre second rating vs yassine is %f " %(kappa))<jupyter_output>The kappa for Pierre second rating vs yassine is 0.816850
{'fraction_non_alphanumeric': 0.16832440703902066, 'fraction_numerical': 0.02320836521295588, 'mean_word_length': 6.07942238267148, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 15, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '536950', 'license': 'MIT', 'n_tokens_mistral': 1627, 'n_tokens_neox': 1550, 'n_words': 284, 'path': 'zooqc_internal_rating_exp1a.ipynb', 'repo_name': 'SIMEXP/zooniverse_qc', 'size': '3921'}