<jupyter_start><jupyter_text>Chi-square test for Independence___The chi-square test is a nonparametric test for testing the association between two variables. A non-parametric test is one that does not make any assumption about the distribution of the population from which the sample is drawn.The following are some of the characteristics of the chi-square test.+ The chi-square test of association is used to test if the frequency of occurrence of one categorical variable is significantly associated with that of another categorical variable.>  The chi-square test statistic is given by: > $$\chi^2 = \sum\frac {(f_o -f_e)^2}{f_e}$$    >   where, $f_o$ denotes the observed frequencies, $f_e$ denotes the expected frequencies, and $\chi$ is the test statistic.  >   Using the chi-square test of association, we can assess if the differences between the frequencies are statistically significant.+ A contingency table is a table with frequencies of the variable listed under separate columns. The formula for the degrees of freedom in the chi-square test is given by: *df=(r-1)(c-1)*, where *df* is the number of degrees of freedom, r is the number of rows in the contingency table, and c is the number of columns in the contingency table.+ The chi-square test compares the observed values of a set of variables with their expected values. It determines if the differences between the observed values and expected values are due to random chance (like a sampling error), or if these differences are statistically significant. If there are only small differences between the observed and expected values, it may be due to an error in sampling. If there are substantial differences between the two, it may indicate an association between the variables.![](data/csd.png)+ The shape of the chi-square distribution for different values of k (degrees of freedom) When the degrees of freedom are few, it looks like an F-distribution. It has only one tail (toward the right). As the degrees of freedom increase, it looks like a normal curve. Also, the increase in the degrees of freedom indicates that the difference between the observed values and expected values could be meaningful and not just due to a sampling error.         Example:Suppose there is a city of 1,000,000 residents with four neighborhoods: A, B, C, and D. A random sample of 650 residents of the city is taken and their occupation is recorded as "white collar", "blue collar", or "no collar". The null hypothesis is that each person's neighborhood of residence is independent of the person's occupational classification.The data are tabulated as:|   OBSERVED   | A   | B   | C   | D   | Row Total ||:------------:|-----|-----|-----|-----|-----------|| White Collar | 90  | 60  | 104 | 95  | 349       || Blue Collar  | 30  | 50  | 51  | 20  | 151       || No Collar    | 30  | 40  | 45  | 35  | 150       || Column Total | 150 | 150 | 200 | 150 | 650       |+ **Null hypothesis:** $H_0$: Occupation and Neighbourhood of Residence are not related.  + **Alternative hypothesis**: $H_1$: Occupation and Neighbourhood of Residence are related.  + **Number of variables:** Two categorical variables (Occupation and Neighbourhood)+ What we are testing: Testing for an association between Occupation and Neighbourhood.+ We conduct a chi-square test of association based on the preceding characteristics.+ Fix the level of significance: α=0.05Make an **expected** value table from the totalsFor each entry calcuate : $$\frac{(row\ total * column\ total)}{overall\ total}$$Example: For A neighbourhood 150 * (349/650) must be the expected White collar Job.|   EXPECTED   | A     | B     | C      | D     ||:------------:|-------|-------|--------|-------|| White Collar | 80.54 | 80.54 | 107.38 | 80.54 || Blue Collar  | 34.85 | 34.85 | 46.46  | 34.85 || No Collar    | 34.62 | 34.62 | 46.15  | 34.62 |Each of the value in the Expected Value table is 5 or higher. May proceed with Chi-Square test.Calculate: $$\chi^2 = \sum\frac {(f_o -f_e)^2}{f_e}$$$$\chi^2\  statistic\  \approx\  24.6$$Under the null hypothesis, this sum has approximately a chi-squared distribution whose number of degrees of freedom is*dof = (number of rows-1)(number of columns-1) = (3-1)(4-1) = 6*From chi square distribution table p value less than 0.0005<jupyter_code>import scipy.stats as stats
import numpy as np

observations = np.array([[90,60,104,95],[30,50,51,20],[30,40,45,35]])
chi2stat, pval, dof, expvalue = stats.chi2_contingency(observations)

print(f'Chi-Square Statistic: ', chi2stat)
print(f'p-value: ', pval)
print(f'degrees of freedom: ', dof)
print(f'Expected Value: \n', expvalue)<jupyter_output>Chi-Square Statistic:  24.5712028585826
p-value:  0.0004098425861096696
degrees of freedom:  6
Expected Value: 
 [[ 80.53846154  80.53846154 107.38461538  80.53846154]
 [ 34.84615385  34.84615385  46.46153846  34.84615385]
 [ 34.61538462  34.61538462  46.15384615  34.61538462]]
{'fraction_non_alphanumeric': 0.10174654752233957, 'fraction_numerical': 0.06701868399675061, 'mean_word_length': 3.7908560311284045, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 8, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '106532', 'license': 'MIT', 'n_tokens_mistral': 1600, 'n_tokens_neox': 1331, 'n_words': 653, 'path': '15. Chi-Square Test for Independence and Goodness of Fit.ipynb', 'repo_name': 'luckyrabbit85/Statistics-For-Data-Science', 'size': '4924'}