<jupyter_start><jupyter_text>GI ScoresLoad data from [Mapping the Genetic Landscape of Human Cells](https://www.sciencedirect.com/science/article/pii/S0092867418307359?via%3Dihubmmc2) paper. > [**Table S5.**](https://www.sciencedirect.com/science/article/pii/S0092867418307359?via%3Dihubmmc5)  Statistical Test   Investigate expression patterns   1st-  [Acute Myeloid Leukemia (OHSU, Nature 2018)](https://www.cbioportal.org/study?id=aml_ohsu_2018)<jupyter_code># ls cBioPortal/aml_ohsu_2018
# print meta data
with open('cBioPortal/aml_ohsu_2018/meta_study.txt') as fp:
    print( fp.read() )<jupyter_output>type_of_cancer: aml
cancer_study_identifier: aml_ohsu_2018
name: Acute Myeloid Leukemia (OHSU, Nature 2018)
description: Whole-exome sequencing of 672 acute myeloid leukemia samples (with 454 matched normals) from the Beat AML program.
citation: Tyner et al. Nature 2018
pmid: 30333627
short_name: AML (OHSU)
groups: PUBLIC<jupyter_text>Chi-square Test Low-Low gene pairs for **K562** GI pairs:<jupyter_code>ohsu_K562_test_res, ohsu_K562_random_test_res = SL_stat_test(
    'cBioPortal/aml_ohsu_2018/data_RNA_Seq_mRNA_median_Zscores.txt', cell_line = 'K562')<jupyter_output><empty_output><jupyter_text>> Low Low $Obs/Exp <  $ High High $Obs/Exp$<jupyter_code>stat_test_plot(ohsu_K562_test_res, ohsu_K562_random_test_res, x_max = 40, y_max = 0.2)
ohsu_K562_test_res[
    (ohsu_K562_test_res.Obs_Low_Low < ohsu_K562_test_res.Obs_High_High) &
    (ohsu_K562_test_res.Obs_Low_Low < ohsu_K562_test_res.Obs_Low_High) &
    (ohsu_K562_test_res.Obs_Low_Low < ohsu_K562_test_res.Obs_High_Low)
].sort_values('GI_Score', ascending = False)
ohsu_K562_random_test_res.sort_values('Obs_Low_Low', ascending = True)<jupyter_output><empty_output><jupyter_text>Low-Low gene pairs for **Jurkat** GI pairs:<jupyter_code>ohsu_Jurkat_test_res, ohsu_Jurkat_random_test_res = SL_stat_test(
    'cBioPortal/aml_ohsu_2018/data_RNA_Seq_mRNA_median_Zscores.txt', cell_line = 'Jurkat')<jupyter_output>K562: 0.67%	677 SLs from 100576 unique gene pairs
Jurkat: 0.4%	300 SLs from 75466 unique gene pairs 
done in 9.263637s
done in 68.139052s<jupyter_text>> Low Low $Obs/Exp <  $ High High $Obs/Exp$<jupyter_code>stat_test_plot(ohsu_Jurkat_test_res, ohsu_Jurkat_random_test_res, x_max = 50, y_max = 0.2)
ohsu_Jurkat_test_res[
    (ohsu_Jurkat_test_res.Obs_Low_Low < ohsu_Jurkat_test_res.Obs_High_High) &
    (ohsu_Jurkat_test_res.Obs_Low_Low < ohsu_Jurkat_test_res.Obs_Low_High) &
    (ohsu_Jurkat_test_res.Obs_Low_Low < ohsu_Jurkat_test_res.Obs_High_Low)
].sort_values('Obs_Low_Low', ascending = False)<jupyter_output><empty_output><jupyter_text>Survival analysis  https://plot.ly/python/v3/ipython-notebooks/survival-analysis-r-vs-python/  2nd-  [Pediatric Acute Lymphoid Leukemia - Phase II (TARGET, 2018)](https://www.cbioportal.org/study?id=all_phase2_target_2018_pub)<jupyter_code># meta data
filepath = 'cBioPortal/all_phase2_target_2018_pub/meta_study.txt'
with open(filepath) as fp:
    print( fp.read() )
# ls cBioPortal/all_phase2_target_2018_pub
data = make_study_dataset('cBioPortal/all_phase2_target_2018_pub/data_RNA_Seq_mRNA_median_Zscores.txt')<jupyter_output>done in 10.104318s<jupyter_text>Chi-square Test Low-Low gene pairs for **K562** GI pairs:<jupyter_code>test_res, random_test_res = SL_stat_test(
    'cBioPortal/all_phase2_target_2018_pub/data_RNA_Seq_mRNA_median_Zscores.txt', cell_line = 'K562')
with open('cBioPortal/all_phase2_target_2018_pub/data_RNA_Seq_mRNA_median_Zscores.txt') as fp:
    lines = [l.split('\t') for l in fp.readlines()]
    # switch NAs -> 0s
    for i,l in enumerate(lines):
        if 'NA' in l:
            n = len(l[2:])
            l = l[0:2]
            for z in [0]*n:
                l.append(z)
        lines[i] = l
    genes = [d[0] for d in lines[1:]]
    RNA_seq = [np.array(d[2:], dtype = float) for d in lines[1:]]

    Q1 = [np.quantile(p, 0.25) for p in RNA_seq]       # Low expression threshold
    Q3 = [np.quantile(p, 0.75) for p in RNA_seq]       # High expression threshold        

data = pd.DataFrame(data = [i + j for i,j in zip([-1*(p <= q1) for p,q1 in zip(RNA_seq, Q1)], 
                                                 [ 1*(p >= q3) for p,q3 in zip(RNA_seq, Q3)])],
                    index = [d[0] for d in lines[1:]], columns= lines[0][2:])


RNA_seq = pd.DataFrame(data = RNA_seq, index = [d[0] for d in lines[1:]],
                       columns= lines[0][2:])


cell_line = 'Jurkat'
SLdata = get_SLdataset()
tmp = SLdata[cell_line]
# filter out missing gene names between GI and expression studies 
tmp = tmp[pd.DataFrame(tmp.Gene1.tolist()).isin(data.index.tolist()).any(1)].reset_index(drop=True)
tmp = tmp[pd.DataFrame(tmp.Gene2.tolist()).isin(data.index.tolist()).any(1)].reset_index(drop=True)
Cors = []
Obs = []
for g1,g2 in zip(tmp.Gene1,tmp.Gene2):    # g1   g2
    Obs.append([
        [sum(np.array(data.loc[g1] < 0)
         * np.array(data.loc[g2] < 0)),   # low_low
         sum(np.array(data.loc[g1] > 0)
         * np.array(data.loc[g2] < 0))],   # high_low
        [sum(np.array(data.loc[g1] < 0)
         * np.array(data.loc[g2] > 0)),  # low_high
         sum(np.array(data.loc[g1] > 0)
         * np.array(data.loc[g2] > 0))]   # high_high
     ])
    Cors.append (spearmanr(np.array(data.loc[g1]), np.array(data.loc[g2]))[0])
# do chi-square test:
chi2_res = [chi2_contingency(o) for o in np.array(Obs)]
Chi2 = [c[0] for c in chi2_res]
ChiPval = [p[1] for p in chi2_res]
Exp = [np.concatenate((x[3][0], x[3][1])) for x in chi2_res]
Obs = [np.concatenate((o[0], o[1])) for o in Obs]         
# make the test result dataframe:
test_res = pd.DataFrame(data= np.concatenate((
    np.stack(( tmp.Gene1 ,tmp.Gene2, tmp[cell_line]), axis = 1),
    np.array(Obs), np.array(Exp), np.stack(( ChiPval, Chi2, Cors), axis = 1),
    ), axis = 1), columns=[
    'Gene1', 'Gene2', 'GI_Score',
    'Obs_Low_Low', 'Obs_High_Low', 'Obs_Low_High', 'Obs_High_High',
    'Exp_Low_Low' ,'Exp_High_Low', 'Exp_Low_High', 'Exp_High_High', 
    'ChiPval', 'Chi2', 'Correlation'])<jupyter_output>K562: 18.08%	18188 SLs from 100576 unique gene pairs
Jurkat: 13.19%	9954 SLs from 75466 unique gene pairs<jupyter_text>> Low Low $Obs/Exp <  $ High High $Obs/Exp$<jupyter_code>test[
#     ((test.Obs_Low_Low/ test.Exp_Low_Low) < (test.Obs_High_High/test.Exp_High_High)) 
    (test.Obs_Low_Low < test.Obs_High_High) &
    (test.Obs_Low_Low < test.Obs_Low_High) &
    (test.Obs_Low_Low < test.Obs_High_Low)
].sort_values('Obs_Low_Low', ascending = False)

r_test[
#     ((test.Obs_Low_Low/ test.Exp_Low_Low) < (test.Obs_High_High/test.Exp_High_High)) 
    (r_test.Obs_Low_Low < r_test.Obs_High_High) &
    (r_test.Obs_Low_Low < r_test.Obs_Low_High) &
    (r_test.Obs_Low_Low < r_test.Obs_High_Low)
    & r_test.ChiPval < 1e-10
].sort_values('Obs_Low_Low', ascending = False)

stat_test_plot(test,random_test)<jupyter_output><empty_output><jupyter_text>Low-Low gene pairs for **Jurkat** GI pairs:<jupyter_code>test_res, random_test_res = SL_stat_test(
    'cBioPortal/all_phase2_target_2018_pub/data_RNA_Seq_mRNA_median_Zscores.txt', cell_line = 'Jurkat')<jupyter_output><empty_output><jupyter_text>> Low Low $Obs/Exp <  $ High High $Obs/Exp$<jupyter_code>test[
#     ((test.Obs_Low_Low/ test.Exp_Low_Low) < (test.Obs_High_High/test.Exp_High_High)) 
    (test.Obs_Low_Low < test.Obs_High_High) &
    (test.Obs_Low_Low < test.Obs_Low_High) &
    (test.Obs_Low_Low < test.Obs_High_Low)
    & test.ChiPval < 1e-10
].sort_values('Obs_Low_Low', ascending = False)

r_test[
#     ((test.Obs_Low_Low/ test.Exp_Low_Low) < (test.Obs_High_High/test.Exp_High_High)) 
    (r_test.Obs_Low_Low < r_test.Obs_High_High) &
    (r_test.Obs_Low_Low < r_test.Obs_Low_High) &
    (r_test.Obs_Low_Low < r_test.Obs_High_Low)
    & r_test.ChiPval < 1e-10
].sort_values('Obs_Low_Low', ascending = False)

stat_test_plot(test,r_test)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13638670388226337, 'fraction_numerical': 0.04541994417660492, 'mean_word_length': 4.578910120311394, 'pattern_counts': {'":': 0, '<': 71, '<?xml version=': 0, '>': 48, 'https://': 5, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '561522', 'license': 'MIT', 'n_tokens_mistral': 3640, 'n_tokens_neox': 3212, 'n_words': 541, 'path': 'chi2model.ipynb', 'repo_name': 'abearab/GI-predictor', 'size': '7882'}