<jupyter_start><jupyter_text>DAE 10 Non parametric testsAuthor:- |    <NAME>, <EMAIL>\    Dept. Architecture, Design and Media Technology, Aalborg University Copenhagen--- Learning goalsAfter working your way through this notebook, you should be able to..- Explain when and why we use non-parametric tests- Set up a contingency table and perform Chi square tests for categorical data- Perform common non parametric tests:    - Mann-Whitney test    - Wilcoxon rank-sum test    - Wilcoxon signed-rank test    - Kruskal–Wallis test    - Friedman’s ANOVA Table of Contents- [1. Nominal data](Nominal)    - [1.2. Degrees of freedom for contingency tables](Contingency)    - [1.3. Chi-square test](Chisquare)    - [1.4. Degrees of freedom for contingency tables](dof)- [2. Interval data: Comparing two groups of scores](Interval)    - [2.1 Independent samples: Mann-Whitney test](MannWhitney)    - [2.2 Related samples: Wilcoxon signed-rank test](Wilcoxon)- [3. More than two conditions](Moresamples)    - [3.1 Independent groups: Kruskal-Wallis Test](Kruskal)    - [3.2 Repeated measures: Friedman’s Test](Friedmans)<jupyter_code>%matplotlib inline
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import scipy.stats as stats
#if you want to import data in colab
from google.colab import files 

uploaded = files.upload()<jupyter_output><empty_output><jupyter_text>1. Nominal dataReturning once more to measurement scales, you might have a selection task where people are asked to make a choice between two or more alternatives. For instance, you may be interested in testing which of two sound designs the participants prefer. In psychology, the "N-alternative-forced-choice" paradigm is often used, where "N" is typically 2,3, or some other number of choices that the participants have to select from. For instance in a listening test where the research question is whether the original version of a sound can be identified (or is preferred) from a set of modified sounds. This is nominal data with categories, and, as such, cannot be analyzed using parametric methods. But that does not mean there are no alternatives. As a start, we will look at how to summarize the results.    1.2 Contingency TableAssume a perception experiment related to audio compression, where listeners were asked to select the original, uncompressed sound in a 2-alternative forced choice (2AFC). For nominal data, we typically get it in a form similar to:|Sound File |Original||-----|--------||1  |Yes       ||2  |No     ||3  |No     ||4  |No     ||5  |No     ||6  |Yes    ||7  |Yes   ||8  |No     ||:  |:     |and so on. For analysis, we instead want a *contingency table* where we summarize the observed frequencies. In the above example, we can imagine that for a total of 144 sound clips, 80 of them were correctly identified as uncompressed. |Correct |Incorrect|Total||-----|--------|-----||80   |64      |144  |In other cases, we have more variables resulting in several rows, giving us a R x C table. For instance, we might have two groups of instrumentalists that evaluate and select their preferences from among three interfaces for a digital musical instrument.<jupyter_code># The data for the different participants and their preferences could look something like this: 
df = pd.DataFrame({'Participant': [1, 2, 3, 4, 5, 6, 7, 8, 9, 10,
                            11, 12, 13, 14, 15, 16, 17, 18, 19, 20],
                   'Instrument': ['Drum', 'Violin', 'Violin', 'Drum', 'Drum', 'Violin',
                               'Violin', 'Violin', 'Violin', 'Violin', 'Drum', 'Drum', 'Drum',
                               'Drum', 'Drum', 'Drum', 'Drum', 'Drum', 'Violin', 'Drum'],
                   'Interface': ['A', 'A', 'A', 'A', 'B', 'B', 'B', 'B', 'B', 'B', 'B',
                               'B', 'C', 'C', 'C', 'C', 'C', 'C', 'C', 'C']})
# Print out the first four participants
df.head(4)<jupyter_output><empty_output><jupyter_text>This can now be summarized in a contingency table with Instrument in the rows, and choice of Interface in columns.<jupyter_code># Create contingency table with Instrument as rows, and choice of Interface in columns

Interfacetable = pd.crosstab(df.Instrument, [df.Interface], margins = False)
# Margins = True adds the row and column sums

Interfacetable<jupyter_output><empty_output><jupyter_text>The table should be interpreted as- Among the 12 drummers (first row): 2 chose interface A, 3 interface B, and 7 chose interface C - Among the Violinists (second row): 2 chose interface A, 5 interface B, and 1 chose interface CAlternatively, if we choose to focus on the columns:- Interface A was selected by a total of 4 participants, while interface B and C were both preferred by 8. From the table, it seems like drummers and violinists had different preferences in their choice of interface to control the musical instrument. Q. Are these differences purely random, or are interfaces B and C actually preferred over A?The reason that we summarize nominal data in this way is, of course, because we can also use these tables for something. Specifically, we can apply a chi-square test to answer the above question.    1.3 Chi-Square TestIn the case of nominal data we cannot use parametric tests, but we can still test whether one condition is selected more than the others. In our example, we can do this by comparing the number of times the uncompressed sound was selected compared to the alternatives, with what would have been expected *if the selection had been random*. That is, if participants had just randomly picked one of the sounds without even bothering to listen.The chi-square test can be used for non numerical and non-normally distributed data where there are categories or different options. The chi-square statistic is calculated as the squared difference between *observed* frequency  $f_o$ and *expected* frequency $f_e$ , divided by the latter:$$\chi^2= \sum{{(f_o - f_e)^2}\over{f_e}}$$In the listening test earlier, we had a contingency table with 144 sound clips, either correctly or incorrectly identified as uncompressed:|Correct |Incorrect|Total||-----|--------|-----||80   |64      |144  |The expected frequency $f_e$, if people were randomly selecting one of the two sounds, would have been equally distributed:|Correct |Incorrect|Total||-----|--------|-----||72   |72      |144  |  Try out 1:Calculate $\chi^2$ for the example listening test above.<jupyter_code># The task is almost as easily done by hand, but here are some starting hints for using Python.

# fo_correct=
# fo_incorrect=
# fe_correct=
# fe_incorrect=

# Square the difference between observed and expected
# (fo_correct - fe_correct) ^ 2
# (fo_incorrect - fe_incorrect) ^ 2

#<jupyter_output><empty_output><jupyter_text>Much like the parametric tests, one determines whether the calculated statistic falls in the *critical region*. That is, if the difference between the expected and observed value is so large that it is unlikely to occur by chance. Just as for parametric tests, the cut-off point for the critical region depends on the significance level (alpha, $\alpha$) and the *degrees of freedom* (df).     1.3 Degrees of Freedom for Contingency TablesThe degrees of freedom for a chi-square test relate to the number of choices or options in the contingency table: $$df=(R-1)(C-1)$$Note that this is different from  many other statistics, for instance sample variance, where $df$ is related to the sample size.For the special case of only one row, the first factor is not considered. For instance, in a test involving participants' selection of singers as being either male or female, there are two possible options to chose from. Just like the listening test example above, and is results in two categories (C) and   $df =$ $C-1=2-1=1$  Try out 2: Calculate the degrees of freedom for the instrument and interface contingency table (in Section 1.2).<jupyter_code># Get the rows and columns from the table
# Interfacetable.shape<jupyter_output><empty_output><jupyter_text>So with the $\chi^2$ and the degrees of freedom, we can look up the probability of achieving this statistic and its relation to the critical value. There are plenty of actual tables to do this, but, like most other statistical packages, Python has functions that does the looking-up-job for us.<jupyter_code># The scipy.stats function chi2_contingency give us the chisquare test from the observed contingency table
# based on the marginal sums (calculated totals) it calculates the expected frequencies 
# and returns the chi-square statistic, probability, as well as degrees of freedom
stat, p, dof, expected = stats.chi2_contingency(Interfacetable)

print('Statistics=%.3f, p=%.3f' % (stat, p))<jupyter_output>Statistics=4.375, p=0.112<jupyter_text>Try out 3: Print the degrees of freedom output from the chi2_contingency function and compare to what you calculated in Try out 2.<jupyter_code># print('Degrees of freedom=%i '  % (dof))<jupyter_output><empty_output><jupyter_text>2. Interval Data: Comparing Two Groups of ScoresIn many other cases, we have interval or ratio data, but other assumptions for parametric tests are violated. Perhaps the plan was to run a t-test, but the scores are skewed or variances too different between groups. Rather than relying on the mean and variance as we would for parametric data, the approach is then to *rank* it. Instead of using the actual scores in the data, we arrange them such that the lowest score receive the rank of 1, the next highest score 2, etc. The test statistic is then calculated on these ranks. This also means that we do not have to make any assumptions about the data (with respect to the shape of their distribution for example). Unfortunately we also lose information, and hence these tests are not as powerful at detecting effects as parametric tests.<jupyter_code># Generate two sets of univariate observations with 100 observations
N = 100
data1 = 5 * np.random.randn(N) + 50
data2 = 5 * np.random.randn(N) + 53
# Summarize
plt.boxplot([data1,data2])
plt.show()<jupyter_output><empty_output><jupyter_text>2.1 Independent Samples: Mann-Whitney Rank-Sum TestThe Mann-Whitney Rank-Sum Test compares two independent groups of scores and is the non-parametric equivalent of the independent t-test. The test compares the ranked positions of scores in the two groups. The function returns the probability that these two samples are from the same distribution (but, again, based on ranks and without assumptions on the shape).<jupyter_code>from scipy.stats import mannwhitneyu

# Mann-Whitney rank-sum test on data1 and data2, returning statistic and probability
Ustat, p = stats.mannwhitneyu(data1, data2)

# Print results
print('U=%.3f, p=%.3f' % (Ustat, p))<jupyter_output>U=2974.000, p=0.000<jupyter_text>The effect-size from the Mann-Whitney test can be calculated from the number of observations (N) and the z-score.$$r =\frac {Z} {\sqrt{N}}$$..but we do not get the z-score in the output from the stats-function for this test.<jupyter_code># help(stats.mannwhitneyu)<jupyter_output><empty_output><jupyter_text>Try out 4: Go back up to the generation of data and change N to 30. Redo the Mann-Whitney rank-sum test and compare.     2.2 Related samples: Wilcoxon signed-rank testIf we instead want to compares two dependent groups of scores we use the Wilcoxon signed rank test.Again, data is ranked and the test examines the difference between the ranks in the two groups. This can be either negative (the 2nd condition rank is smaller than that in the first), positive (the 2nd condition rank is larger than that in the first), or tied (no difference between the ranks in the two conditions). We return to the stroop-test data we discovered to be skewed in the parametric data chapter. The test has related samples because each participant had both congruent and incongruent stimuli to respond to.<jupyter_code>#load stroop test data
data = pd.read_csv('Stroopdata.txt', delimiter = '\t', index_col=0,header=0) 
data.describe()
plt.boxplot([data.Congruent,data.Incongruent])
plt.show()
# Perform Wilcoxon test (from scipy.stats) on the difference between conditions
Tstat_stroop, p_stroop =stats.wilcoxon(data.Congruent,data.Incongruent)
# Print results
print('T=%.3f, p=%.3f' % (Tstat_stroop, p))<jupyter_output>T=0.000, p=0.000<jupyter_text>The resulting p-value shows that it is highly unlikely that the two sets of scores would come from the same distribution. So the test confirms what might be guessed from the plot: it takes considerably longer to read the colour in the incongruent condition. Of course the probability is not perfectly zero, but rounded off. Like for the Mann-Whitney  test, the effect size $r$ could be calculated from the z-score, if you run this test using a program/function that reports it.    3 More than Two ConditionsAs in the parametric case, we should avoid multiple comparisons when we have more than two samples or conditions. So just as we use ANOVA to examine main effects for groups when we have more than two conditions, we use the non-parametric equivalents when the parametric assumptions are not fulfilled.<jupyter_code>data1 = np.random.randn(100)*5 + 50
data2 = np.random.randn(100)*5 + 50
data3 = np.random.randn(100)*5 + 54

plt.boxplot([data1,data2, data3])
plt.show()<jupyter_output><empty_output><jupyter_text>3.1  Independent Groups: Kruskal-Wallis TestIn case we have more than two independent samples, the Kruskal-Wallis is the non-parametric alternative to one-way independent  ANOVA and tests differences between independent groups of scores.Similar to the two-group examples above, the test is performed on ranked data. The mean rank data can be used to interpret effects (if you can get it).<jupyter_code># kruskal from scipy.stats 

Hstat,p=stats.kruskal(data1,data2,data3)
print('H=%.3f, p=%.3f' % (Hstat, p))<jupyter_output>H=35.648, p=0.000<jupyter_text>Just as for the ANOVA, the main effect does not say which of the groups/conditions is the cause of the effect. For this, one needs follow-up testing. Field & Hole suggest doing this using Mann-Whitney tests but with Bonferroni correction of the significance level, and being selective about the pairwise comparisons needed. As always, it is a good idea to check the details of the function beforehand. Specifically, you will notice a warning about using the test for very small sample sizes (must have at least 5 observations).<jupyter_code>#help(kruskal)<jupyter_output><empty_output><jupyter_text>3.2 Repeated measures: Friedman TestWhen we have used the same participants in all conditions, the observations are related. The Friedman test compares more than two paired samples and is the non-parametric correlate of repeated measures ANOVA.<jupyter_code># Generate three independent samples
Condition1 = 5 * np.random.randn(100) + 16
Condition2 = 5 * np.random.randn(100) + 20
Condition3 = 5 * np.random.randn(100) + 15

plt.boxplot([Condition1,Condition2, Condition3])
plt.show()<jupyter_output><empty_output><jupyter_text>The test statistics is an acquaintance from earlier up this notebook,  $\chi^2$<jupyter_code>Chistat, p = stats.friedmanchisquare(Condition1, Condition2, Condition3)
print('Chi-square=%.3f, p=%.3f' % (Chistat, p))<jupyter_output>Chi-square=34.880, p=0.000
{'fraction_non_alphanumeric': 0.08055446580358311, 'fraction_numerical': 0.01830783313717798, 'mean_word_length': 4.507742167806986, 'pattern_counts': {'":': 0, '<': 62, '<?xml version=': 0, '>': 62, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '414050', 'license': 'MIT', 'n_tokens_mistral': 4444, 'n_tokens_neox': 4094, 'n_words': 2049, 'path': 'DAE10_nonparametric.ipynb', 'repo_name': 'sofiadahl/Design_and_Analysis_of_Experiments', 'size': '15308'}