<jupyter_start><jupyter_text>Quantitative QC by CV calculationThe data used in this notebook is lymphocyte data for one patient's B cells and T cells. We use this data to show the proteome variation between the cell types. Here, we calculate CVs to show the quality of the data.After calculating CVs, we calculate Spearman correlation among replicates. First, we import our loader module. This brings the functions defined there (in our repository at ~/load_data.py) into scope so we can use them in this script. Then we can load our data and store it as data.Calling head shows the first several lines of the dataframe, which provides an idea of the type of data present and the structure of the dataframe.<jupyter_code>import load_data
#data_raw = load_data.load_FragPipe()
data_raw=load_data.load_max_quant()
data_raw.head()<jupyter_output><empty_output><jupyter_text>Now we normalize across runs. Note that following median normalization, we reverse the log2, leaving the data aligned between runs but allowing the most typical coefficient variation calculation.<jupyter_code>import data_utils
data_log2_normalized = data_utils.normalize(data_raw)
data_normalized = data_log2_normalized.apply(lambda series: 2**series)<jupyter_output><empty_output><jupyter_text>Next, we select the proteins that are measured in at least three samples from each group, allowing the calculations to proceed without imputed zero-handling.<jupyter_code>import data_utils    
cell_types = [' B_', ' T_']
indecies = data_normalized.apply(data_utils.check_three_of_each_type, axis=1,
                      cell_types = cell_types)
data = data_normalized[indecies]
data.head()<jupyter_output><empty_output><jupyter_text>Finally, we will calculate the coeffients of variation for each protein within each of the two cell types.<jupyter_code>from scipy.stats import variation
from statistics import mean
import pandas as pd
from numpy import isnan
CVs = {}
for population in cell_types:
    cells_in_population = list(s for s in data.columns.values.tolist() if population in s)
    data_by_type = data[cells_in_population]
    #now we have a dataframe with just one subpopulation
    #Call variation function
    var = data_by_type.apply(variation, axis=1, nan_policy='omit')
    CVs[population] = var
    
    #Here we report an overview
    print (population)
    print ('Mean CV:\t',mean(var))
    print ('Min CV: \t',min(var))
    print ('Max CV: \t',max(var))
    print ('nan:    \t',len([i for i in var if isnan(i)]))
    print ('Zero:    \t',len([i for i in var if i==0]))
    var_under_20 = len([i for i in var if i < .2])
    var_under_10 = len([i for i in var if i < .1])
    count = len(var)#len([i for i in var if i!=0])
    print ('Under 10%:\t',var_under_10,'\t{0:.2f}'.format(var_under_10/count))
    print ('Under 20%:\t',var_under_20,'\t{0:.2f}'.format(var_under_20/count))
    print (count)
    print ()
    

CVs = pd.DataFrame.from_dict(CVs)<jupyter_output>B_
Mean CV:	 0.423891976004037
Min CV: 	 0.0384642685934447
Max CV: 	 1.3386985052351754
nan:    	 0
Zero:    	 0
Under 10%:	 15 	0.03
Under 20%:	 85 	0.14
589

 T_
Mean CV:	 0.34056569996861485
Min CV: 	 0.045101501984886404
Max CV: 	 1.1559536845992306
nan:    	 0
Zero:    	 0
Under 10%:	 27 	0.05
Under 20%:	 152 	0.26
589<jupyter_text>Next, we will visualize the data, which shows that the majority of proteins have small coefficients of variation.<jupyter_code>import matplotlib.pyplot as plt
import seaborn as sns
sns.set(font_scale=1.5)
sns.set_style("white")
figure = sns.violinplot(data=CVs, width=.5)
figure.set_ylabel("Coefficient of Variation")
figure.set_xticklabels(['B cells', 'T cells'])
#plt.savefig("data/figures/CV.png", dpi=300)

plt.show()
import matplotlib.pyplot as plt
import seaborn as sns

sns.set(font_scale=1.5)

sns.set_style("white")
figure = sns.distplot(CVs[' B_'], hist = False, label='B cells')#, width=.5)
figure = sns.distplot(CVs[' T_'], hist = False, label='T cells')#, width=.5)
figure.set_ylabel("Relative Frequency")
figure.set_xlabel("Coefficient of Variation")

figure.legend(['B cells', 'T cells'])
plt.show()<jupyter_output><empty_output><jupyter_text>Here we summarize the CVs overall. Note that the CVs were calculated within types and so still represent technical variability, not variation between cell types.<jupyter_code>var = CVs.values.flatten()

print ('Mean CV:\t',mean(var))
print ('Min CV: \t',min(var))
print ('Max CV: \t',max(var))
print ('nan:    \t',len([i for i in var if isnan(i)]))
print ('Zero:    \t',len([i for i in var if i==0]))
    
var_under_20 = len([i for i in var if i < .2])
var_under_10 = len([i for i in var if i < .1])
count = len(var)#len([i for i in var if i!=0])
print ('Under 10%:\t',var_under_10,'\t{0:.2f}'.format(var_under_10/count))
print ('Under 20%:\t',var_under_20,'\t{0:.2f}'.format(var_under_20/count))<jupyter_output>Mean CV:	 0.3822288379863259
Min CV: 	 0.0384642685934447
Max CV: 	 1.3386985052351754
nan:    	 0
Zero:    	 0
Under 10%:	 42 	0.04
Under 20%:	 237 	0.20<jupyter_text>This shows accuracy in repeatedly characterizing each cell type.  Correlation coefficientNext, we show reproducibility of the replicates by Spearman correlation coefficient.<jupyter_code>correlations = data.corr(method="spearman")

labels=['B cells - C10','B cells - C11',
                    'B cells - C12','B cells - C13',
                    'B cells - C9  ','T cells - D10',
                    'T cells - D11','T cells - D12',
                    'T cells - D13','T cells - D9  ']

correlations.index=labels
correlations.columns=labels

import numpy as np
mask = np.zeros(correlations.shape, dtype=bool)
mask[np.triu_indices(len(mask))] = True

sns.heatmap(correlations, cmap = 'coolwarm', mask = mask)


plt.savefig("data/correlations_heatmap_FragPipe.png", dpi=300,
            bbox_inches='tight')
from numpy import nan
#drop self-correlations of 1
for sample in correlations.columns:
        correlations[sample][sample]=nan
        
correlations<jupyter_output><empty_output><jupyter_text>Here we split the dataset by cell type and perform the same correlation test. We then take the average correlation between replicates.<jupyter_code>from numpy import nan

corr_type = {}
corr_summary={}
for cell_type in cell_types:
    
    cells_of_type = list(s for s in data.columns.values.tolist() if cell_type in s)
    data_by_type = data[cells_of_type]
    corr_of_type = data_by_type.corr(method='spearman')
    
    #drop self-correlations of 1
    for sample in corr_of_type.columns:
        corr_of_type[sample][sample]=nan
    
    corr_type[cell_type] = corr_of_type
    
    #take the average of the correlations between a sample and  
    #    the other samples of the same type
    summary = corr_of_type.apply(lambda x: mean(x.dropna()))
    corr_summary[cell_type] = mean(summary)
    
    print (cell_type,"average correlation:\t",mean(summary))<jupyter_output>B_ average correlation:	 0.8711005899668742
 T_ average correlation:	 0.9000559082660787<jupyter_text>With the average correlation between each set of replicates, we now average them to get the overall average.<jupyter_code>mean(corr_summary.values())<jupyter_output><empty_output><jupyter_text>Now, we calculate the average correlation when comparing a B cell replicate to a T cell replicate. We expect this to be lower than either of the two above numbers comparing replicates within a cell type.<jupyter_code>B_cells = list(s for s in correlations.index if 'B cells' in s)
T_cells = list(s for s in correlations.index if 'T cells' in s)

corr_cross_types = []

for B in B_cells:
    for T in T_cells:
        c = correlations[B][T]
        corr_cross_types.append(c)
print("Mean cross-type correlations:")        
mean(corr_cross_types)<jupyter_output>Mean cross-type correlations:
{'fraction_non_alphanumeric': 0.10454079014879425, 'fraction_numerical': 0.04258594150846588, 'mean_word_length': 4.322184300341297, 'pattern_counts': {'":': 0, '<': 41, '<?xml version=': 0, '>': 37, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '62334', 'license': 'RSA-MD', 'n_tokens_mistral': 2893, 'n_tokens_neox': 2571, 'n_words': 815, 'path': 'Quantitative_QC.ipynb', 'repo_name': 'PayneLab/Lymphocytes1', 'size': '7796'}