<jupyter_start><jupyter_text>DK connectome ordering (and reordering) In the Raj lab we have collected several connectivity matrices along the years.For the [Desikan-Killiani](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.458.1615&rep=rep1&type=pdf)atlas, we have 3 matrices (see `../data/` folder):- `dk_ADNI_averaged.mat`:     - Origin: Average over 37 young subjects from ADNI    - Low quality (dTI from ADNI is known to be pretty low resolution)- `mean80_fibercount.csv`:     - Origin: Cohort of 80 (young?) healthy subjects    - Probably young HC- `dk_conn_meanACS69.mat`:     - Origin: Cohort of 69 (young?) healthy subjects.    - Subset of `mean80_fibercount`.        The goal of this notebook is to reorder these connectivity matrices following FreeSurfer ordering (see `../data/dk_all.csv` file)and export them as csv with labels built in.<jupyter_code>import matplotlib.pyplot as plt
from scipy.io import loadmat
import pandas as pd
from scipy.stats import pearsonr
ADNI_conn = loadmat('../data/dk_ADNI_averaged.mat')['connectome_avg']
meanACS_conn = loadmat('../data/dk_conn_meanACS69.mat')['meanACS']
mean80_conn = pd.read_csv('../data/mean80_fibercount.csv')
mean80_conn = mean80_conn.set_index(mean80_conn.columns)
# load rajutils
import importlib.util
raj_utils_path = "../utils/atlas_utils.py"
spec = importlib.util.spec_from_file_location("atlas-utils.load_atlas", raj_utils_path)
raj_utils = importlib.util.module_from_spec(spec)
spec.loader.exec_module(raj_utils)
# load desikan utils info. Index is in Free-Surfer ordering
dk_atlas_info = raj_utils.load_atlas(atlas="DK").set_index('Name')

# Get the cortical regions
right_cortical = dk_atlas_info.loc[dk_atlas_info['Hemisphere'] == 'Right'].loc[dk_atlas_info['Cortex'] == 'cortical']
left_cortical = dk_atlas_info.loc[dk_atlas_info['Hemisphere'] == 'Left'].loc[dk_atlas_info['Cortex'] == 'cortical']

# Get the subcortical regions
left_subcortical = dk_atlas_info.loc[dk_atlas_info['Hemisphere'] == 'Left'].loc[dk_atlas_info['Cortex'] == 'subcortical']
right_subcortical = dk_atlas_info.loc[dk_atlas_info['Hemisphere'] == 'Right'].loc[dk_atlas_info['Cortex'] == 'subcortical']

#Make DFs following the order above
LRLR = pd.concat([left_cortical, right_cortical, left_subcortical, right_subcortical])
LRRL = pd.concat([left_cortical, right_cortical, right_subcortical, left_subcortical])

#Get the region names of these dataframes
LRLR_names = list(LRLR.index)
LRRL_names = list(LRRL.index)<jupyter_output><empty_output><jupyter_text>Creating dataframes for connectivities<jupyter_code>plt.figure(figsize=(20,20))

plt.subplot(1, 3, 1)
plt.title('ADNI averaged connectome')
_ = plt.matshow(ADNI_conn, fignum=False)

plt.subplot(1, 3, 2)
plt.title("mean ACS69")
_ = plt.matshow(meanACS_conn, fignum=False)

plt.subplot(1, 3, 3)
plt.title("mean80 fibercount")
_ = plt.matshow(mean80_conn, fignum=False)<jupyter_output><empty_output><jupyter_text>From the above exercise:- `ADNI` is LRLR, CCSS- `meanACS69` is LRRL, CCSS- `mean80` is LRLR, SSCCWhere L/R mean Left/Right hemispheres. C/S means Cortical/Subcortical structures.`ADNI` and `meanACS69` were commonly used in the lab and it is expected (although not 100% certain) that the order follows FreeSurfer (therefore not alphabetical).`mean80` is already in the dictionary form although right now I'm not able to track it down its origin."  1. Fixing ADNI ordering to FreeSurfer<jupyter_code>ADNI_conn_df = pd.DataFrame(ADNI_conn, columns=LRLR_names, index=LRLR_names)
ADNI_conn_df.head(3)
# ADNI connectivity is already in FreeSurfer order, but for completion:
ADNI_conn_fs_order = ADNI_conn_df[LRLR_names].loc[LRLR_names]<jupyter_output><empty_output><jupyter_text>2. Fixing meanACS ordering to FreeSurfer<jupyter_code>meanACS_conn_df = pd.DataFrame(meanACS_conn, columns=LRRL_names, index=LRRL_names)
meanACS_conn_df.head(3)
meanACS_conn_fs_order = meanACS_conn_df[LRLR_names].loc[LRLR_names]
meanACS_conn_fs_order.head(3)<jupyter_output><empty_output><jupyter_text>3. Fixing mean80 ordering to FreeSurfer<jupyter_code>#mean80 is already a pandas dataframe but 2 names are incomplete / different
mean80_conn_df = pd.DataFrame(mean80_conn)

mean80_conn_df = mean80_conn_df.rename(columns={'Left-VentralDC':'Left-VentralDC/Hypothalamus', 'Right-VentralDC':'Right-VentralDC/Hypothalamus'})
mean80_conn_df = mean80_conn_df.rename(index={'Left-VentralDC':'Left-VentralDC/Hypothalamus', 'Right-VentralDC':'Right-VentralDC/Hypothalamus'})

mean80_conn_df.head(3)
mean80_conn_fs_order = mean80_conn_df[LRLR_names].loc[LRLR_names]
mean80_conn_fs_order.head(3)<jupyter_output><empty_output><jupyter_text>4. New plots<jupyter_code>plt.figure(figsize=(20,20))

plt.subplot(1, 3, 1)
plt.title('Reordered ADNI averaged connectome')
_ = plt.matshow(ADNI_conn_fs_order, fignum=False)

plt.subplot(1, 3, 2)
plt.title("Reordered mean ACS69")
_ = plt.matshow(meanACS_conn_fs_order, fignum=False)

plt.subplot(1, 3, 3)
plt.title("Reordered mean80 fibercount")
_ = plt.matshow(mean80_conn_fs_order, fignum=False)<jupyter_output><empty_output><jupyter_text>5. Export to CSV and pickles<jupyter_code>ADNI_conn_fs_order.to_csv('../data/postprocessing/ADNI37_conn_fs_order.csv')
meanACS_conn_fs_order.to_csv('../data/postprocessing/meanACS69_conn_fs_order.csv')
mean80_conn_fs_order.to_csv('../data/postprocessing/mean80_conn_fs_order.csv')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1191449814126394, 'fraction_numerical': 0.023977695167286244, 'mean_word_length': 7.329721362229102, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '349301', 'license': 'Unlicense', 'n_tokens_mistral': 2167, 'n_tokens_neox': 1996, 'n_words': 378, 'path': 'cortography/notebooks/1.DK_connectome_reordering.ipynb', 'repo_name': 'pfdamasceno/rajutils', 'size': '5380'}