<jupyter_start><jupyter_text>the data has been read in, but the cols are not named. This will not do.<jupyter_code>names = [
"call",
"pgene",
"ctregion_chrom",
"ctregion_start",
"ctregion_stop",
"ptregion_chrom",
"ptregion_start",
"ptregion_stop",
"perid",
"meanperid",
"ks",
"meanks",
"cgene",
"cgregion_chrom",
"cgregion_start",
"cgregion_stop",
"pgregion_chrom",
"pgregion_start",
"pgregion_stop",
"ka",
"meanka",
"perid_len",
"ka_len",
"ks_len",
"syn_len"]
ks_dat.columns = names<jupyter_output><empty_output><jupyter_text>Now that we have names for columns we can begin to work with data.<jupyter_code>ks_measured = ks_dat.loc[~ks_dat['ks'].isin(['None','NA','none','nan','na'])]<jupyter_output><empty_output><jupyter_text>We have thrown out all unmeasured ks values. It is time to do some graphing<jupyter_code>print(ks_measured.head())
print(ks_measured.describe())
ks_measured.plot(x='ka', y='ks',
                 kind='scatter')
ks_measured.plot(x='ka', y='perid',
                 kind='scatter')
ks_measured.plot(x='ks', y='perid',
                 kind='scatter')
numeric_ksmean =  ks_dat.loc[~ks_dat['meanks'].isin(['None','NA','none',
                                                     'nan',
                                                     'na',
                                                     'NaN'])]
print(numeric_ksmean['meanks'].describe())
print(numeric_ksmean['meanka'].describe())
print(numeric_ksmean['meanka'].head())
print(numeric_ksmean['meanks'].head())
numeric_ksmean.hist(column='ks')<jupyter_output><empty_output><jupyter_text>`>awk 'OFS=","{print $3"_"$4"_"$5,$12, $10}' bog_sep17_bag_of_genes.tsv | grep -v None | sort | uniq >reg_meaks_meanpid.tsv`having trouble getting the exact right filtering. I can figure it out later in pandas. Right now this will do.<jupyter_code>meanks_by_region = pd.read_csv(
    '/home/ndh0004/code/coge_tools/test_out/reg_meaks_meanpid.tsv',
    sep=','
 )
names = ['reg','meanks','meanpid']
meanks_by_region.columns = names
print(meanks_by_region.head())
meanks_by_region.hist(column='meanks',
                      bins=20)
meanks_by_region.plot(x='meanks', y='meanpid',
                      kind='scatter')
calls = ks_dat.loc[ks_dat['call'].isin(['A','B'])]
calls.boxplot(by='call', column='perid')
calls.boxplot(by='call',column='meanperid')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.16957076073098173, 'fraction_numerical': 0.006374840628984276, 'mean_word_length': 4.162280701754386, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 19, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '424938', 'license': 'BSD-2-Clause', 'n_tokens_mistral': 867, 'n_tokens_neox': 831, 'n_words': 138, 'path': 'images_notes/ks_analysis_sep17.ipynb', 'repo_name': 'NDHall/coge_tools', 'size': '2353'}