<jupyter_start><jupyter_text>OverviewIn this notebook you will be doing a t-test on rho, the population correlation coefficient, and a chi-square test for independence. A t-test on rho allows you to test the null hypothesis that a correlation coefficient between a predictor and an outcome variable at the populaton level is zero. The alternative hypothesis is that the population correlation coefficient is not zero. This type of test is suitable for situations where you are looking for a correlation between two variables and the data can be summarized by a scatterplot and fit with a line-of-best-fit. In order to do this a t-test on rho, the following must be true:* The data has interval or ratio measurement scales.* The residuals between the observed outcomes and the line-of-best fit must be normally distributed.Chi-square tests are non-parametric test for differences between frequency distributions of categorical/nominal data. This means it can be employed like an one-sample T-test, an independent samples T-test, or a Mann-Whitney test in situations where your data is nominal/categorical instead of ordinal, interval, or ratio. Chi-square goodness-of-fit tests allow you to test the null hypothesis that a the frequency distribution of observed values of nominal data are no different than a known or expected distribution. For example, let's say you surveyed 99 people asking their political affiliation, Democrat, Republican, or Independent. Your null hypotheis is that all three options are equally likely. You would then expect the frequency distribution to be 33 Democrats, 33 Republicans, and 33 Independents. The alternative hypothesis would be that the three party affiliations are not equally likely. Chi-square tests for homogeneity or independence allo you to test the null hypothesis that the frequency distribution of observed values of nominal data is independent of some other nominal dimansion of your data. For example, let's say that in your political affilation survey you also record the gender of those surveyed. Your null hypotheis could be that the frequency distribution of politial affiliation is independent of gender. The alternative would be that the distribution is dependend on gender, that is, there is a significant difference between the distribution of political affiliation between the genders. In order to do a chi-square test you should ensure the following:* The data set is sufficiently large that the expected number of indivduals assigned to each 'class'  of a categorical variable is greater than or equal to 5 at least 80% or the time.  Run the following cell (shift-enter) to load needed python packages and modules.<jupyter_code># RUN BUT DON'T MODIFY THIS CELL
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import scipy.stats as stats<jupyter_output><empty_output><jupyter_text>Data for 70+ Cereals * Load cereal.csv into a pandas dataframe.* If this was in your library you would use the path `.../library/filename.csv`.* Use the `.head()` method to print out the first 7 rows of the dataframe.* Get the `.shape` (no parentheses) property to see how many rows and columns are in the dataset.**Source:** dowloaded 12/18/2017 from the Kaggle public dataset repository. Credited to <NAME>**Description:** Nominal/categorical and ratio/interval data for 70+ different cereals.<jupyter_code># RUN BUT DON'T MODIFY THIS CELL
url = "https://raw.githubusercontent.com/prof-groff/evns462/master/data/cereal.csv"
cereal = pd.read_csv(url)
print(cereal.head())
print("shape: ", cereal.shape)<jupyter_output>name mfr type  calories  protein  fat  sodium  fiber  \
0                  100% Bran   N    C        70        4    1     130   10.0   
1          100% Natural Bran   Q    C       120        3    5      15    2.0   
2                   All-Bran   K    C        70        4    1     260    9.0   
3  All-Bran with Extra Fiber   K    C        50        4    0     140   14.0   
4             Almond Delight   R    C       110        2    2     200    1.0   

   carbo  sugars  potass  vitamins  shelf  weight  cups     rating  
0    5.0       6     280        25      3     1.0  0.33  68.402973  
1    8.0       8     135         0      3     1.0  1.00  33.983679  
2    7.0       5     320        25      3     1.0  0.33  59.425505  
3    8.0       0     330        25      3     1.0  0.50  93.704912  
4   14.0       8      -1        25      3     1.0  0.75  34.384843  
shape:  (77, 16)<jupyter_text>Is fat content a predictor of calories?In the following cell:* Extract fat and calories data columns and store as x and y, respectively.* Make a scatter plot of calories (y-axis) as a function of fat (x-axis).* Fit the data with a line of best fit. * Do a T-test on rho to test the following null hypothesis.H0: &rho; = 0 at &alpha; = 0.05HA: &rho; not equal to 0.<jupyter_code># import tools to build linear models and find a line-of-best-fit
from sklearn import linear_model

# create a linear regression object to use to build the linear model
regr = linear_model.LinearRegression()

# pull out the 'fat' column and store is as a variable called x
x = cereal['fat']
x = x.values.reshape(-1,1) # this is needed to reshape x so it works with LinearRegression()

# TODO: UNCOMMENT AND COMPLETE THE FOLLOWING LINES OF CODE TO PULL OUT THE 'CALORIES' COLUMN 
# AND STORE IT AS Y THEN, RESHAPE Y.

# y = 
# y = 

# TODO: UNCOMMENT AND COMPLETE THE FOLLOWING LINE OF CODE TO USE THE regr.fit() METHOD TO FIT A 
# LINE-OF-BEST-FIT TO THE X AND Y DATA.

# regr.fit()

# TO DO: UNCOMMENT AND COMPLETE THE FOLLOWING LINES OF CODE THAT TAKE TWO X VALUES, 0 AND 5, AND FINDS
# THE CORRESPONDING Y VALUES ACCORDING TO THE EQUAITON FOR THE LINE OF BEST FIT Y=MX+B USING
# THE regr.predict() METHOD. FINDING THESE TWO POINTS WILL ALLOW US TO DRAW THE LINE-OF-BEST-FIT BECAUSE
# ALL YOU NEED ARE TWO POINTS, (X1,Y1) AND (X2,Y2), TO DRAW A LINE.

# x_fit = 
# y_fit = 


# TO DO: UNCOMMENT AND COMPLETE THE FOLLOWING LINES OF CODE TO MAKE A SCATTER PLOT OF THE
# DATA AND SHOW THE LINE-OF-BEST-FIT TOO.

# plt.scatter()
# plt.xlabel()
# plt.ylabel()
# plt.plot()
# plt.show()

# TO DO: UNCOMMENT AND COMPLETE THE FOLLOWING LINES OF CODE TO CALCULATE THE COEFFICIENT OF 
# DETERMINATION (r^2) USING THE regr.score() METHOD AND THE CORRELATION COEFFICIENT (r) BY
# TAKING THE SQUARE ROOT OF THIS RESULT.
# PRINT BOTH r^2 AND r TO THE CONSOLE USING print() COMMANDS

# rsqr = 
# r = 

# TO DO: UNCOMMENT AND COMPLETE THE FOLLOWING TO USING THE BUILT IN PYTHONG FUNCTION
# stats.pearsonr() TO CALCULATE r AND ITS CORRESPONDING p-value. PRINT BOTH OF THESE VALUES
# TO THE CONSOLE USING A print() COMMAND

# r, pvalue =<jupyter_output><empty_output><jupyter_text>Questions:Is the null hypothsis accepted or rejected, that is, is the population correlation coefficient &rho; statistically equal to zero or is it non-zero meaning there is a coorelation between the x and y data?  Do certain brands of cereal get better shelf placement?Let's take a look at Kelloggs and General Mills cereals and see if one or the other gets better shelf placement. * H0: Shelf placement is independent of cereal brand.* HA: Shelf placement is not independent of cereal brand.<jupyter_code># The following line of code uses the .isin() method to pull out all rows of the cereal dataframe
# that have either K or G in the 'mfr' column 
k_gm = cereal[cereal['mfr'].isin(('K','G'))]

# TO DO: UNCOMMENT AND COMPLETE THE FOLLOWING TWO LINES OF CODE THAT ACCOMPLISH EXACTLY THE SAME 
# END AS THE ABOVE LINE OF CODE BUT USES THE .groupby AND .get_group() METHODS INSTEAD.

# cereal_by_mfr = cereal.groupby()
# K_GM = pd.concat([cereal_by_mfr.get_group(), cereal_by_mfr.get_group()])

# TO DO: UNCOMMENT AND COMPLETE THE FOLLOWING LINE OF CODE THAT USES THE pd.crosstab() FUNCTION
# TO CONTRUCT A CONTINGENCY TABLE WITH A ROW FOR EACH MANUFACTUERER (K AND G) AND A COLUMN FOR EACH
# SHELF PLACEMENT (1, 2, AND 3). THEN USE THE PRINT COMMAND TO PRINT THIS TABLE TO THE CONSOLE

# table = 

# TO DO: UNCOMMENT AND COMPLETE THE FOLLOWING LINE OF CODE THAT USES THE stats.chi2_contingency()
# FUNCTION TO TEST THE NULL HYPOTHESIS ABOVE. PRINT THE TEST STATISTICS AND THE COORESPONDING P-VALUE
# TO THE CONSOLE

# statistic, pvalue, dof, exp =<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.05910677741663657, 'fraction_numerical': 0.026965210063801613, 'mean_word_length': 3.398094229751191, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '517650', 'license': 'MIT', 'n_tokens_mistral': 2679, 'n_tokens_neox': 2436, 'n_words': 1215, 'path': 'notebooks/assignments/assignment-5-rho-t-test-chi-square.ipynb', 'repo_name': 'prof-groff/evns-462', 'size': '8315'}