<jupyter_start><jupyter_text>Exctract demographic info and cognitive metrics from the HCP behavioral data<jupyter_code>import os, sys, time, h5py, zipfile
import numpy as np
import pandas as pd
from statistics import median
url = 'https://db.humanconnectome.org/REST/search/dict/Subject%20Information/results?format=csv&removeDelimitersFromFieldValues=true&restricted=0&project=HCP_1200'

hcp =pd.read_csv(url)<jupyter_output><empty_output><jupyter_text>Loading the HCP spreadsheet and select the subset of columns we are interesting on working with:- Subject ID- Age- Gender- Mini Mental State Examination (MMSE) total score- Penn matrix to measure fluid intelligence (number of correct answers)- Speed of processing score<jupyter_code>hcp_behav = pd.read_csv('/home/anna/Scrivania/NeuroHackademy2019/unrestricted_annatruzzi91_8_5_2019_13_21_16.csv')
hcp_behav.keys()
behav_subset = hcp_behav[['Subject','Age','Gender','MMSE_Score','PMAT24_A_CR','ProcSpeed_AgeAdj']]<jupyter_output><empty_output><jupyter_text>Inspect variables of interest (i.e. range, count occurrences, median)<jupyter_code>## Age
ageunique, agecounts = np.unique(min_age_vector, return_counts=True)
print('Age')
print(dict(zip(ageunique, agecounts)))
agelist = ageunique

## MMSE 
unique, counts = np.unique(np.array(hcp_behav['MMSE_Score']), return_counts=True)
print('MMSE')
print(dict(zip(unique, counts)))

## Pemm matrices
print('Pemm')
print('min Pemm score is '+str(min(hcp_behav['PMAT24_A_CR'])))
print('max Pemm score is '+str(max(hcp_behav['PMAT24_A_CR'])))
print('median Pemm score is '+str(median(hcp_behav['PMAT24_A_CR'])))

## Processing speed
print('Processing speed')
print('min Processing speed score is '+str(min(hcp_behav['ProcSpeed_AgeAdj'])))
print('max Processing speed score is '+str(max(hcp_behav['ProcSpeed_AgeAdj'])))
print('median Processing speed score is '+str(median(hcp_behav['ProcSpeed_AgeAdj'])))<jupyter_output>Age
{22: 247, 26: 527, 31: 418, 36: 14}
MMSE
{23: 1, 24: 1, 26: 31, 27: 84, 28: 197, 29: 440, 30: 452}
Pemm
min Pemm score is 4.0
max Pemm score is 24.0
median Pemm score is 10.0
Processing speed
min Processing speed score is 45.31
max Processing speed score is 149.3
median Processing speed score is 103.13<jupyter_text>Ages are expressed in intervals (the actual ace is a Tier 1 restricted information).Therefore we need to extract the min age for each participants in order to create 4 groups (one per age interval)<jupyter_code>min_age_vector=[]
max_age_vector=[]
for item in behav_subset['Age']:
    if '+' in item:
        min_age_vector.append(int(item[0:2]))
        max_age_vector.append(90)
    else:
        min_age_vector.append(int(item[0:2]))
        max_age_vector.append(int(item[-2:]))

min_age_vector=np.asarray(min_age_vector)
max_age_vector=np.asarray(max_age_vector)<jupyter_output><empty_output><jupyter_text>Add columns to the dataframe in which to define the category each participant is in for each variable of interest:- Age --> 0 (min age = 22),1 (min age = 26),2 (min age = 31),3 (min age = 36)- MMSE --> o (lower than median), 1 (higher than median)- Pemm matrices --> o (lower than median), 1 (higher than median)- Processing speed --> o (lower than median), 1 (higher than median)<jupyter_code>## Age
age_category = np.zeros(shape=len(behav_subset['Age']))
behav_subset['min_age'] = pd.Series(min_age_vector)
behav_subset['age_category'] = pd.Series(age_category)
behav_subset.loc[behav_subset['min_age'] == agelist[0],['age_category']] = 0
behav_subset.loc[behav_subset['min_age'] == agelist[1],['age_category']] = 1
behav_subset.loc[behav_subset['min_age'] == agelist[2],['age_category']] = 2
behav_subset.loc[behav_subset['min_age'] == agelist[3],['age_category']] = 3

## MMSE
MMSE_level = np.zeros(shape=len(behav_subset['MMSE_Score']))
behav_subset['MMSE_Level'] = pd.Series(MMSE_level)
behav_subset.loc[behav_subset['MMSE_Score'] <= median(behav_subset['MMSE_Score']),['MMSE_Level']] = 0
behav_subset.loc[behav_subset['MMSE_Score'] > median(behav_subset['MMSE_Score']),['MMSE_Level']] = 1

## Penn matrices
Penn_level = np.zeros(shape=len(behav_subset['PMAT24_A_CR']))
behav_subset['Penn_Level'] = pd.Series(Penn_level)
behav_subset.loc[behav_subset['PMAT24_A_CR'] <= median(behav_subset['PMAT24_A_CR']),['Penn_Level']] = 0
behav_subset.loc[behav_subset['PMAT24_A_CR'] > median(behav_subset['PMAT24_A_CR']),['Penn_Level']] = 1

## Processing speed
ProcSpeed_level = np.zeros(shape=len(behav_subset['ProcSpeed_AgeAdj']))
behav_subset['ProcSpeed_Level'] = pd.Series(ProcSpeed_level)
behav_subset.loc[behav_subset['ProcSpeed_AgeAdj'] <= median(behav_subset['ProcSpeed_AgeAdj']),['ProcSpeed_Level']] = 0
behav_subset.loc[behav_subset['ProcSpeed_AgeAdj'] > median(behav_subset['ProcSpeed_AgeAdj']),['ProcSpeed_Level']] = 1

behav_subset<jupyter_output>/home/anna/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:3: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  This is separate from the ipykernel package so we can avoid doing imports until
/home/anna/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:4: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  after removing the cwd from sys.path.
/home/anna/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:12: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_in[...]
{'fraction_non_alphanumeric': 0.132276954874268, 'fraction_numerical': 0.028246641405442644, 'mean_word_length': 7.110335195530726, 'pattern_counts': {'":': 0, '<': 22, '<?xml version=': 0, '>': 26, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '540086', 'license': 'MIT', 'n_tokens_mistral': 2242, 'n_tokens_neox': 2053, 'n_words': 466, 'path': '.ipynb_checkpoints/GAN_DL_AgeFromSpreadsheet-checkpoint.ipynb', 'repo_name': 'meltemiatay/neurohack_dl_gan', 'size': '5806'}