<jupyter_start><jupyter_text>---&nbsp; The chromosome rearrangement scoring data by subtelomere Directional Genomic Hybridization (subtelo-dGH) was originally stored inindvidiual excel sheets. However, the data was previously combined into one sheet. The same is true of the telomeric aberrations by TeloFISH - it's in the same file as the subtelo-dGH data. subtelo-dGH was performed on pre-/post-flight samples most distal to spaceflight and on midflight samples for astros 1, 2, & 5; as well as distal pre-flight/post-flight samples for all other astros. Telomeric aberrations were scored for astros 1, 2, & 5 at pre-/post-flight most proximal to spaceflight and on midflight samples.We'll extract all of the data from this excel sheet. We'll generate a dataframe & save it for later use.&nbsp; ---  Extracting all telomeric / chr aberr data into one dataframe<jupyter_code># pulling ALL of the data in the sheet into one dataframe

chr_data = '../data/raw data/Chromosome_Aberrations_telodGH_unrelatedAstros+SK_complete_TeloAberr_astros125___graphs_7_17_19_227pm.xlsx'
nasa_chr_data = pd.read_excel(chr_data)<jupyter_output><empty_output><jupyter_text>Parsing pre-flight, midflight1&2, and post-flight telomeric aberration data<jupyter_code># parsing pre-flight, midflight 1/2, and post-flight telomeric aberration data

pre_f_telo_aberr = nasa_chr_data.iloc[0:90, 147:155]
mid_f1_telo_aberr = nasa_chr_data.iloc[0:90, 155:163]
mid_f2_telo_aberr = nasa_chr_data.iloc[0:90, 163:171]
post_f_telo_aberr = nasa_chr_data.iloc[0:90, 179:187]

print(pre_f_telo_aberr.columns)

display(
pre_f_telo_aberr.head(1),
mid_f1_telo_aberr.head(1),
mid_f2_telo_aberr.head(1),
post_f_telo_aberr.head(1))<jupyter_output>Index(['astro id', 'flight status.3', 'Cell Number.5', 'Image File Number.6',
       '# of Fragile Telos', '# of STL-complete', '#  of STL-hetero',
       '#  of potential threaded chr'],
      dtype='object')<jupyter_text>Renaming columns to avoid formatting errors<jupyter_code># we need to rename the columns for these sheets; column names were changed in loading

telo_aberr_cols = ['astro id', 'flight status', 'Cell Number',
       'Image File Number', '# of Fragile Telos', '# of STL-complete',
       '# of STL-hetero', '# of sat associations']

pre_f_telo_aberr.columns = telo_aberr_cols
mid_f1_telo_aberr.columns = telo_aberr_cols
mid_f2_telo_aberr.columns = telo_aberr_cols
post_f_telo_aberr.columns = telo_aberr_cols<jupyter_output><empty_output><jupyter_text>Combining pre-flight, midflight1&2, post-flight telomeric aberr data into one dataframe<jupyter_code>all_astro_telo_aberr = pd.concat([pre_f_telo_aberr, mid_f1_telo_aberr, mid_f2_telo_aberr, post_f_telo_aberr], axis=0, ignore_index=True)
all_astro_telo_aberr = all_astro_telo_aberr.drop(['Cell Number', 'Image File Number'], axis=1)

print(all_astro_telo_aberr.shape)
all_astro_telo_aberr.head(4)<jupyter_output>(360, 6)<jupyter_text>Saving all telomeric aberration data to dataframe for later retrieval<jupyter_code>all_astro_telo_aberr.to_csv('../data/compiled data/All_astronauts_telomeric_aberrations.csv', index = False)<jupyter_output><empty_output><jupyter_text>Making a tidy-data formatted dataframe for telomeric aberration data<jupyter_code>melt_all_astro_telo_aberr = pd.melt(all_astro_telo_aberr,
                                   id_vars=['astro id', 'flight status'],
                                   var_name='aberration type',
                                   value_name='count per cell')

melt_all_astro_telo_aberr.head(4)<jupyter_output><empty_output><jupyter_text>Saving telomeric aberration in tidy-data format dataframe for later retrieval<jupyter_code>melt_all_astro_telo_aberr.to_csv('../data/compiled and processed data/melt_all_astro_telo_aberr.csv', index = False)<jupyter_output><empty_output><jupyter_text>Parsing pre-flight, midflight1&2, and post-flight chromosome rearrangement data<jupyter_code># parsing pre-flight, midflight 1/2, and post-flight chromosome rearrangement data

pre_f = nasa_chr_data.iloc[0:404, 1:15]
mid_f_1 = nasa_chr_data.iloc[35:109, 25:39]
mid_f_2 = nasa_chr_data.iloc[0:111, 49:63]
post_f = nasa_chr_data.iloc[0:400, 96:110]
# renaming columns to avoid errors when combining dataframes

chr_aberr_cols = ['astro id', 'flight status', 'cell number', 'image file number', 
                  'dicentrics', 'translocations', 'inversions', 
                  'terminal inversions', 'terminal SCEs paint cis', 
                  'terminal SCEs dark cis', 'subtelo SCEs', 
                  'sister chromatid exchanges', 'insertions', 'satellite associations']

pre_f.columns = chr_aberr_cols
mid_f_1.columns = chr_aberr_cols
mid_f_2.columns = chr_aberr_cols
post_f.columns = chr_aberr_cols<jupyter_output><empty_output><jupyter_text>Combining chromosome rearrangement data<jupyter_code>all_astro_chr_aberr = pd.concat([pre_f, mid_f_1, mid_f_2, post_f], axis=0, ignore_index=True)
all_astro_chr_aberr = all_astro_chr_aberr.drop(['cell number', 'image file number', 'insertions'], 
                                               axis=1).dropna().reset_index(drop=True)
print(all_astro_chr_aberr.columns)
all_astro_chr_aberr.head(4)<jupyter_output>Index(['astro id', 'flight status', 'dicentrics', 'translocations',
       'inversions', 'terminal inversions', 'terminal SCEs paint cis',
       'terminal SCEs dark cis', 'subtelo SCEs', 'sister chromatid exchanges',
       'satellite associations'],
      dtype='object')<jupyter_text>Generating total inversions & terminal SCEs columns<jupyter_code>all_astro_chr_aberr['total inversions'] = all_astro_chr_aberr['terminal inversions'] + all_astro_chr_aberr['inversions']
all_astro_chr_aberr['terminal SCEs'] = all_astro_chr_aberr['terminal SCEs paint cis'] + all_astro_chr_aberr['terminal SCEs dark cis']<jupyter_output><empty_output><jupyter_text>Saving all astronauts chr aberr to csv for later retrieval<jupyter_code>all_astro_chr_aberr.to_csv('../data/compiled data/all_astro_chr_aberr.csv', index=False)<jupyter_output><empty_output><jupyter_text>Generating tidy-data formatted dataframe for all astronaut chromosome rearrangement data<jupyter_code>copy_all_astro_chr_aberr = all_astro_chr_aberr.drop(['terminal inversions', 'inversions', 
                                                     'terminal SCEs paint cis', 'terminal SCEs dark cis'], axis=1)

melt_all_astro_chr_aberr = pd.melt(copy_all_astro_chr_aberr,
       id_vars=['astro id', 'flight status'],
       var_name='aberration type',
       value_name='count per cell')

melt_all_astro_chr_aberr.head(4)<jupyter_output><empty_output><jupyter_text>Saving all astronaut chromosome rearrangement data in tidy-data format for later retrieval<jupyter_code>melt_all_astro_chr_aberr.to_csv('../data/compiled and processed data/All_astronauts_chromosome_aberration_data_tidy_data.csv', index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09713120722295034, 'fraction_numerical': 0.01893111984855104, 'mean_word_length': 5.503787878787879, 'pattern_counts': {'":': 0, '<': 50, '<?xml version=': 0, '>': 50, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '541708', 'license': 'MIT', 'n_tokens_mistral': 2595, 'n_tokens_neox': 2348, 'n_words': 505, 'path': 'NASA_chr_aberr/01b NASA Astros Chr and Telo Aberr Data Extraction and Cleaning.ipynb', 'repo_name': 'j-Luxton/NASA-astronauts-telomeres-chromos', 'size': '6867'}