<jupyter_start><jupyter_text>17BR_magn_coop_diff_cell type7/29/21A. The degree predicted cell type specific vocabularies actually drove cell type specific expression in the MPRA. This point is actually pretty opaque to me still as the paper and figures were configured initially and if we had some correlation matrix to specifically show this, it could help make the central point of the paper. So, putting cell type specific DNA motif combinations [maybe colored by cell type] on one edge of the matrix and cell type-specific MPRA activity on the other edge with enrichment score heat map would be one way to address this central and critical gap in the paper. You can be certain that not only the Reviewers but even the Editors will kill the paper on initial screening without this clear information. I apologize that I did not push for clarity on this as the figures were being drafted. B. The degree to which malignant cells A) engage activity of new synergistic DNA motif combinations (DMCs) versus B) differentially modulating existing normal lineage DMCs and C) the putative target genes and their biological processes for each. As configured, the figures don’t address this and I cant figure it out but you can bet the Reviewers will demand this also.Neoplastic cells engage new DMCs – these need to be more clearly shown, quantified, and maybe should go into a supplementary tableNeoplastic cells differentially modulate DMCs active in their normal counterparts that mediate neoplasia-relevant processes – the same, with the biological impact emphasized – a venn diagram for KC/cSCC and MC/MM of shared and unique DMCs would help address both pointsThe target genes of the normal unique DMCs, the cancer unique DMCs, and the overlapping DMCs in GO would be critical hereupdate 10/24/21 'cell type-specific DMC' panel? Essentially, this is what I'm envisioning:4 columns x rows for all functional DMCs as determined by MPRAColumn 1 = rows of DMCs grouped and colored by cell typeColon (yellow), GM (grey), KC (red), MC (blue), SCC (green), MM (orange)Column 2 = rows of DMCs grouped and colored by functional categoryRedundant (yellow), Buffer (green), Synergy (pink), Single Driver (purple)Column 3 = rows of DMCS with heat map of p-value for cell type-specificityI'm thinking maybe the color scale could mimic the ones you used for the PPV matrices? Light red = p-value <0.1 to purple = not significantMaybe add stars to the DMCs with p values <0.1Column 4 = rows of DMC labels<jupyter_code>library(tidyverse)
library(pheatmap)
library(RColorBrewer)
save_pheatmap_png <- function(x, filename, width=1200, height=1000, res = 200) {
  png(filename, width = width, height = height, res = res)
  grid::grid.newpage()
  grid::grid.draw(x$gtable)
  dev.off()
}
save_pheatmap_pdf <- function(x, filename, width=7, height=7) {
   stopifnot(!missing(x))
   stopifnot(!missing(filename))
   pdf(filename, width=width, height=height)
   grid::grid.newpage()
   grid::grid.draw(x$gtable)
   dev.off()
}

save_prefix = '../data/processed/mpra/'
mpra_tissues = c('Colon', 'GM', 'COLO', 'MC', 'KCD6', 'A431')

nl_tissues = c('Colon', 'GM',  'MC', 'KCD6')
scc_tissues = c('KCD6','A431')
mm_tissues = c('MC','COLO')

# get cooperative values
normlog_vocab_df_coop = read.csv('../../miseq/mpraanalyze_results/dk_norm_cooperativity/normlog_vocab_df_coop.csv',row.names=1, stringsAsFactor=F)%>%
    arrange(vocab,tissue)
head(normlog_vocab_df_coop)
# get AND vocabs
new_mpra_df = read.csv('../data/processed/mpra/results_df_piv_match_count.csv',row.names=1, stringsAsFactor=F)
# filter for AND VOCABs
and_vocab = unique(new_mpra_df[new_mpra_df$consensus_class=='AND',]$vocab)
# vocab to tissue annon
vocab_tissue = normlog_vocab_df_coop%>%
    select(vocab, tissue)%>%distinct()%>%column_to_rownames('vocab')
vocab_tissue%>%group_by(tissue)%>%tally()
coop_nl_df = normlog_vocab_df_coop%>% 
#     pivot_wider(names_from = orient, values_from = value)%>%
#     mutate(value=A_B_coop - A_B_add)%>%
    filter(vocab %in% and_vocab)%>%
    filter((tissue %in% nl_tissues)&(mpra_tissue%in% nl_tissues))%>%
    group_by(vocab, tissue, mpra_tissue)%>%
    summarise(value=mean(value))%>%
    arrange(tissue)
pheatmap(coop_nl_df%>%pivot_wider(id_cols='vocab',names_from='mpra_tissue',values_from=value)%>%column_to_rownames('vocab'),
        annotation_row=vocab_tissue,cluster_rows=F)

write.csv(coop_nl_df, paste0(save_prefix,'coop_nl_df.csv'))<jupyter_output><empty_output><jupyter_text>really hard to see differences very vocab specific effect<jupyter_code># try cooperative - additive effect
coop_add_nl_df = normlog_vocab_df_coop%>% 
    pivot_wider(names_from = orient, values_from = value)%>%
    mutate(value=A_B_coop - A_B_add)%>%
    filter(vocab %in% and_vocab)%>%
    filter((tissue %in% nl_tissues)&(mpra_tissue%in% nl_tissues))%>%
    group_by(vocab, tissue, mpra_tissue)%>%
    summarise(value=mean(value))%>%
    arrange(tissue)%>%
    group_by(vocab, tissue)%>%
    mutate(min_value=min(value))%>%
    mutate(norm_value = value-min_value)

pheatmap(coop_add_nl_df%>%pivot_wider(id_cols='vocab',names_from='mpra_tissue',values_from=value)%>%column_to_rownames('vocab'),
        annotation_row=vocab_tissue,cluster_rows=F)
write.csv(coop_add_nl_df, paste0(save_prefix,'coop_add_nl_df.csv'))

# try normalizing
# first by normalizing to the correct tissue
coop_nl_df_tissue_match = filter(coop_nl_df, tissue==mpra_tissue)%>%select(vocab,tissue,value)%>%rename(norm=value)
coop_nl_df_norm = coop_nl_df%>%
    left_join(coop_nl_df_tissue_match, by=c('vocab','tissue'))%>%
    mutate(norm_value = value-norm)
pheatmap(coop_nl_df_norm%>%pivot_wider(id_cols='vocab',names_from='mpra_tissue',values_from=norm_value)%>%column_to_rownames('vocab'),
        annotation_row=vocab_tissue,cluster_rows=F,
                color = colorRampPalette(rev(brewer.pal(n =11, name ="RdBu")))(255))



# then by normalize the lowest value
coop_nl_df_minus = coop_nl_df%>%
    group_by(vocab, tissue)%>%
    mutate(min_value=min(value))%>%
    mutate(norm_value = value-min_value)
pheatmap(coop_nl_df_minus%>%pivot_wider(id_cols='vocab',names_from='mpra_tissue',values_from=norm_value)%>%column_to_rownames('vocab'),
        annotation_row=vocab_tissue,cluster_rows=F,
               color = colorRampPalette(brewer.pal(n =11, name ="Reds"))(255))

# pheatmap(coop_nl_df%>%pivot_wider(id_cols='vocab',names_from='mpra_tissue',values_from=value)%>%column_to_rownames('vocab'),
#         annotation_row=vocab_tissue,clusters_row=F)<jupyter_output><empty_output><jupyter_text>USE THE LAST plot to figure out the best vignettes because the rest look like garbageFOXM1::KLF15 and EGR2::RORAalso the IRF8::SPIB,  EGR2::RORA<jupyter_code>coop_nl_df%>%select(vocab)%>%distinct()%>%dim()<jupyter_output>Adding missing grouping variables: `tissue`<jupyter_text>10 of the 33 synergistic vocab words displayed cell type specific synergy<jupyter_code>coop_nl_df_rank_all = coop_nl_df%>% 

#     pivot_wider(names_from = orient, values_from = value)%>%
#     mutate(value=A_B_coop - A_B_add)%>%
#     filter(tissue %in% nl_tissues)%>%
    group_by(vocab, tissue, mpra_tissue)%>%
    summarise(value=mean(value))%>%   
    arrange(desc(value)) %>% 
    group_by(vocab, tissue) %>% 
    mutate(rank=row_number())%>%
    filter(tissue==mpra_tissue)
write.csv(coop_nl_df_rank_all, paste0(save_prefix, 'coop_nl_df_rank_all.csv'))
coop_nl_rank = coop_nl_df%>% 

#     pivot_wider(names_from = orient, values_from = value)%>%
#     mutate(value=A_B_coop - A_B_add)%>%
#     filter(tissue %in% nl_tissues)%>%
    group_by(vocab, tissue, mpra_tissue)%>%
    summarise(value=mean(value))%>%   
    arrange(desc(value)) %>% 
    group_by(vocab, tissue) %>% 
    mutate(rank=row_number())%>%
    filter(tissue==mpra_tissue)%>%filter(rank<3)%>%
    rename(mean_value = value)#%>%

normlog_vocab_df_coop%>%
    filter(orient=='A_B_coop')%>%
    inner_join(coop_nl_rank,by=c('vocab','tissue','mpra_tissue'))%>%
    pivot_wider(names_from=mpra_tissue,values_from=value)%>%
    arrange(vocab)
normlog_vocab_df_coop%>% 

#     pivot_wider(names_from = orient, values_from = value)%>%
#     mutate(value=A_B_coop - A_B_add)%>%
    filter(tissue %in% nl_tissues)%>%
    group_by(vocab, tissue, mpra_tissue)%>%
    summarise(value=mean(value))%>%   
    arrange(desc(value)) %>% 
    group_by(vocab, tissue) %>% 
    mutate(rank=row_number())%>%
    filter(tissue==mpra_tissue)%>%filter(rank<2)
coop_nl_df%>%filter(vocab=='IRF8::SPIB')<jupyter_output><empty_output><jupyter_text>cell type specificity<jupyter_code>spec_df = read.csv('../data/processed/mpra/vocab_cell_type_sig_df_norm.csv')
spec_df = spec_df %>% 
    mutate(neglog10pval = -log10(pval))%>%
    arrange(tissue, -neglog10pval)

colnames(spec_df)

row_annon = spec_df%>%
    select(tissue,consensus_class, vocab)%>%
    column_to_rownames('vocab')

plot_df = spec_df%>%
    select(vocab, neglog10pval)%>%
    mutate(filler = 1)%>%
    column_to_rownames('vocab')
    
p = pheatmap(plot_df,
            cluster_row=F, cluster_col=F,
            annotation_row = row_annon,
         color = colorRampPalette(rev(brewer.pal(n =11, name ="PuOr")))(255))
save_pheatmap_pdf(p, '../data/processed/mpra/vocab_cell_type_sig_df_norm.pdf', height=20, width=5)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12638724275401358, 'fraction_numerical': 0.009589483891822003, 'mean_word_length': 5.947604790419161, 'pattern_counts': {'":': 0, '<': 26, '<?xml version=': 0, '>': 106, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '221554', 'license': 'MIT', 'n_tokens_mistral': 3551, 'n_tokens_neox': 3487, 'n_words': 727, 'path': 'notebooks/17BR_magn_coop_diff_celltype.ipynb', 'repo_name': 'mguo123/pan_omics', 'size': '9281'}