<jupyter_start><jupyter_text>The Kruskal-Wallis H-test tests the null hypothesis that the population median of all of the groups are equal. It is a non-parametric version of ANOVA. The test works on 2 or more independent samples, which may have different sizes. Note that rejecting the null hypothesis does not indicate which of the groups differs. Post-hoc comparisons between groups are required to determine which groups are different. The Kruskal-Wallis H and Friedman tests for comparing more than two data samples: the nonparametric version of the ANOVA and repeated measures ANOVA tests.<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import glob
import scipy
import scikit_posthocs as sp
from scipy import stats
%matplotlib inline
fn=glob.glob("*xlsx")
print(fn)
FA=pd.read_excel(fn[8])
#FA=pd.read_excel(fn[0],header=False)
COLS=list(FA.columns.values)
print(COLS)
FA
GTYPES=list(set(FA[COLS[0]]))
FA.groupby(COLS[0]).size()
values_per_group = {col_name:col for col_name, col in FA.groupby(COLS[0])[COLS[1]]}
#print(values_per_group.values())
stat,p = stats.kruskal(*values_per_group.values())
print('Statistics=%.3f, p=%.20f' % (stat, p))
print(p)
# interpret
alpha = 0.05
if p > alpha:
    print('Same distributions (fail to reject H0)')
else:
    print('Different distributions (reject H0)')<jupyter_output>Statistics=18.002, p=0.00294394479229011284
0.002943944792290113
Different distributions (reject H0)<jupyter_text>P value tells us we may reject the null hypothesis that the population medians of all of the groups are equal. To learn what groups (species) differ in their medians we need to run post hoc tests. scikit-posthocs provides a lot of non-parametric tests mentioned above. Let's choose Conover's test.<jupyter_code>pc=sp.posthoc_conover(FA, val_col=COLS[1], group_col=COLS[0], p_adjust = 'holm')
print(pc)
fig, ax = plt.subplots(nrows=1, ncols=1, figsize=(20,10))
heatmap_args = {'linewidths': 0.25, 'linecolor': '0.5', 'clip_on': False, 'square': True, 'cbar_ax_bbox': [0.80, 0.35, 0.04, 0.3]}
sp.sign_plot(pc, **heatmap_args)

pc2=sp.posthoc_dunn(FA, val_col=COLS[1], group_col=COLS[0], p_adjust = 'holm')
print(pc2)<jupyter_output>stg3      stg4  stg4_cryoSEM      stg5  stg5_AFM  \
stg3         -1.000000  1.000000      0.019448  1.000000  0.590607   
stg4          1.000000 -1.000000      0.060891  0.793623  0.793623   
stg4_cryoSEM  0.019448  0.060891     -1.000000  0.005788  1.000000   
stg5          1.000000  0.793623      0.005788 -1.000000  0.441064   
stg5_AFM      0.590607  0.793623      1.000000  0.441064 -1.000000   
stg5_cryoSEM  1.000000  1.000000      0.793623  0.961726  1.000000   

              stg5_cryoSEM  
stg3              1.000000  
stg4              1.000000  
stg4_cryoSEM      0.793623  
stg5              0.961726  
stg5_AFM          1.000000  
stg5_cryoSEM     -1.000000<jupyter_text>Post hoc pairwise test for multiple comparisons of mean rank sums (Dunn’s test). May be used after Kruskal-Wallis one-way analysis of variance by ranks to do pairwise comparisons.<jupyter_code>fig, ax = plt.subplots(nrows=1, ncols=1, figsize=(20,10))
heatmap_args = {'linewidths': 0.25, 'linecolor': '0.5', 'clip_on': False, 'square': True, 'cbar_ax_bbox': [0.80, 0.35, 0.04, 0.3]}
sp.sign_plot(pc2, **heatmap_args)

import seaborn as sns
fig, ax = plt.subplots(nrows=1, ncols=1, figsize=(20,10))
bplot=sns.boxplot(y=COLS[1],x=COLS[0],data=FA)

bplot=sns.swarmplot(y=COLS[1],x=COLS[0],data=FA,color='black',alpha=0.75)
plt.yticks(fontsize=24)
plt.xticks(fontsize=12, rotation="vertical")
plt.ylabel(r"$\lambda$ [nm]", fontsize=34)
plt.xlabel(" ", fontsize=34)
plt.savefig("boxplotlambda_alltechniques.jpeg",dpi=400,bbox_inches ="tight")
fig, ax  = plt.subplots(1, 1, figsize=(20, 10))

plt.yticks(size=24)
plt.xticks(fontsize=12, rotation="vertical")
bplot=sns.violinplot(y=COLS[1],x=COLS[0],
                 data=FA, 
                 width=0.90,
                 alpha=0.17,
                 inner=None,
                 palette="colorblind")
bplot=sns.swarmplot(y=COLS[1],x=COLS[0],data=FA,color='black',alpha=0.75)
plt.savefig("violijnplot_lambda all techniques.png",dpi=400,bbox_inches ="tight")
FA
Sn=list(set(FA['sample']))
mu=pd.DataFrame()
A=[]
B=[]
C=[]
for k in Sn:
    fj=FA.loc[FA['sample']==k]
    u=fj['lambda'].describe()
    #print(k)
    #print(u)
    #print('median:', fj['lambda'].median())
    A.append(k)
    B.append(fj['lambda'].median())
    C.append(fj['lambda'].mean())
    #print('*******************')
mu['sample']=A
mu['median-lambda']=B
mu['mean-lamdba']=C
print(mu)<jupyter_output>sample  median-lambda  mean-lamdba
0  stg5_cryoSEM      1279.6740  1269.972101
1  stg4_cryoSEM      1149.0215  1207.788603
2          stg4      1266.1600  1289.285833
3      stg5_AFM      1112.0000  1192.300000
4          stg5      1305.4015  1313.780725
5          stg3      1269.6845  1308.788827
{'fraction_non_alphanumeric': 0.13655848132271892, 'fraction_numerical': 0.11267605633802817, 'mean_word_length': 3.9295774647887325, 'pattern_counts': {'":': 0, '<': 10, '<?xml version=': 0, '>': 11, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '175350', 'license': 'MIT', 'n_tokens_mistral': 2239, 'n_tokens_neox': 1890, 'n_words': 433, 'path': 'UPFIGS/.ipynb_checkpoints/Lambda measurement-alltechniques-checkpoint.ipynb', 'repo_name': 'calugo/ROBS', 'size': '4899'}