<jupyter_start><jupyter_text>Analysis of ClustersThe purpose of this analysis is to explore how states might be identified if different clustering techniques are used. The focus is on using the normalized data.<jupyter_code>import init
from common import constants as cn
from common.data_provider import DataProvider
from common.data_grouper import DataGrouper
from common import transform_data
from terms.term_matrix import TermMatrix
from plots import util_plots
from common_python.text.term_analyzer import TermAnalyzer
from common_python.text import util_text

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import os
import seaborn as sns
from sklearn.cluster import KMeans
# Initialize the data
provider = DataProvider()
provider.do()
provider.df_normalized.head()<jupyter_output><empty_output><jupyter_text>Visualization of the Normalized Data<jupyter_code>df = util_plots.plotClusteredHeatmap(provider=provider, ncluster=60)<jupyter_output><empty_output><jupyter_text>Perfect CorrelationsPerfectly correlated genes are ones that are differentially expressed in the same way at each interval.<jupyter_code>df = transform_data.thresholdData(provider.df_normalized, is_include_nan=False)
grouper = DataGrouper(df)
grouper.do(min_size=1)
MAX = 25
probs = []
grouper.do(min_size=0, is_include_zeros=True)
df = grouper.df_group
total = df[cn.CNT_GROUP].sum()
for n in range(0, MAX):
    count = sum([r[cn.CNT_GROUP] for _, r in df.iterrows() if r[cn.CNT_UP] + r[cn.CNT_DOWN] == n])
    probs.append(count)
probs = [1.0*p / total for p in probs]


grouper.do(min_size=0, is_include_zeros=True)
len(grouper.df_group)
# Hist for the number of genes timepoints with DE
_ = plt.bar(range(len(probs)), probs)
_ = plt.xlabel("Number DE")
_ = plt.ylabel("Fraction")
_ = plt.title("Genes")
# Histogram for number of groups with differential expression
bins = 10
for col in [cn.CNT_UP, cn.CNT_DOWN, "both"]:
    plt.figure()
    if col == "both":
        _ = plt.hist(grouper.df_group[cn.CNT_UP] + grouper.df_group[cn.CNT_DOWN], density=True, bins=bins)
    else:
        _ = plt.hist(grouper.df_group[col], density=True, bins=bins)
    _ = plt.xlabel("Number DE")
    _ = plt.ylabel("Fraction")
    _ = plt.title("Correlated Groups: %s" % col)
# CDF of group sizes
def plotGroupSizeCDF(min_size):
    grouper.do(min_size=min_size)
    cnts = grouper.df_group[cn.CNT_GROUP].tolist()
    cnts.sort()
    frac = [1.0*i/len(cnts) for i in range(len(cnts))]
    _ = plt.plot(cnts, frac)
    _ = plt.xlabel("Group Size")
    _ = plt.ylabel("Fraction")
# CDF of group sizes
def plotGroupSizeHist(min_size):
    grouper.do(min_size=min_size)
    cnts = grouper.df_group[cn.CNT_GROUP].tolist()
    plt.hist(cnts, bins=len(grouper.df_group))
    _ = plt.xlabel("Group Size")
    _ = plt.ylabel("# Groups")
    _ = plt.title("Min. Group Size: %d" % min_size)
plotGroupSizeHist(2)
grouper.do(min_size=1)
grouper.df_group[cn.CNT_GROUP].sum()
grouper.do(min_size=3)
1464 - grouper.df_group[cn.CNT_GROUP].sum()
2772-1464
for size in [1, 2, 3, 5]:
    plotGroupSizeCDF(size)
from scipy.stats import binom
binom.cdf(0, 10, 0.047)
len(provider.df_normalized)<jupyter_output><empty_output><jupyter_text>**Observations**1. What's going on with the block of genes that are solid blue? Should be 0s? Some genes remain over- or under- expressed across multiple state transitions1. Also, seem to be mini-states1. It looks like there's a solid band around 2300. This seems impossible since the high values must be compensated by low values relative to the mean.1. Note the juxtaposition of over-expression with under-expression. What are these genes? **Analyses**1. Validate against Mario's data1. Find perfectly correlated genes, plot statistics on groups.1. Create a spreadsheet with groups, genes, GO terms1. Fit frequency of groups with n DEs to a discrete exponential?1. Plot number or fraction of >1 in succession DEs over time with vertical lines before looking at statistical tests.1. Overlay the groups on time.  Gene Analysis<jupyter_code>util_plots.plotClustermap()
util_plots.plotClustermap(is_up_regulated=False)<jupyter_output><empty_output><jupyter_text>Term Analysis<jupyter_code>matrix = TermMatrix()
matrix.plotTimeAggregation(is_up_regulated=False)
matrix.plotTimeAggregation(is_up_regulated=True)
_, _, ser_cluster = matrix.calcClusters(max_distance=0, is_up_regulated=True)
ser_cluster.sort_values()
df = pd.DataFrame({'group': ser_cluster, 'term': ser_cluster.index})
dfg = df.groupby('group')
groups = dfg.groups
terms = ["\n".join(v.tolist()) for v in groups.values()]
sizes = [len(v.tolist()) for v in groups.values()]
df_g = pd.DataFrame({'group': list(groups.keys()), 'terms': terms, 'sizes': sizes})
df_g.to_csv("terms.csv")
len(groups.keys())
type(groups)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11173533083645443, 'fraction_numerical': 0.011860174781523096, 'mean_word_length': 5.53125, 'pattern_counts': {'":': 1, '<': 21, '<?xml version=': 0, '>': 22, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '486583', 'license': 'MIT', 'n_tokens_mistral': 1715, 'n_tokens_neox': 1651, 'n_words': 435, 'path': 'notebooks/Cluster Analysis.ipynb', 'repo_name': 'uwescience/expression_state', 'size': '4806'}