<jupyter_start><jupyter_text>Framingham  The Framingham Heart Study is a long-term, ongoing cardiovascular cohort study of residents of the city of Framingham, Massachusetts. The study began in 1948 with 5,209 adult subjects from Framingham, and is now on its fourth generation of participants. Prior to the study almost nothing was known about the epidemiology of hypertensive or arteriosclerotic cardiovascular disease. Much of the now-common knowledge concerning heart disease, such as the effects of diet, exercise, and common medications such as aspirin, is based on this longitudinal study. It is a project of the National Heart, Lung, and Blood Institute, in collaboration with (since 1971) Boston University. Various health professionals from the hospitals and universities of Greater Boston staff the project. Data source: https://www.kaggle.com/amanajmera1/framingham-heart-study-dataset Further reading: https://en.wikipedia.org/wiki/Framingham_Heart_Study  Diabetes and the population<jupyter_code>framingham = pd.read_csv("framingham.csv")
framingham.head()<jupyter_output><empty_output><jupyter_text>Propose Null and Alternative hypotheses According to the CDC, the prevalence of diagnosed diabetes (i.e., the percentage of the population who have it) in the U.S. around this time was 0.93%. We are going to conduct a hypothesis test with the following null and alternative hypotheses: 1. **Null hypothesis**: The probability that a participant within the Framingham Study has diabetes is equivalent to the prevalence of diagnosed diabetes within the population, any difference is due to chance.2. **Alternative hypothesis**: The probability that a participant within the Framingham Study has diabetes is different than the prevalence of diagnosed diabetes within the population.  Propose a test statistic **Test Statistic**: the absolute distance between the observed prevalence and the true population prevalence as our test statistic.  The observed value of the statistic in the sample<jupyter_code>prop_diabetes = np.abs(framingham[framingham["diabetes"] == 1].shape[0] / framingham.shape[0] - 0.0093)
prop_diabetes<jupyter_output><empty_output><jupyter_text>The distribution of the test statistic under the null<jupyter_code>def one_statistic():
    n = framingham.shape[0]
    random_sample = np.random.binomial(n, 0.0093, 1) / n
    return np.abs(random_sample - 0.0093)
def bootstrap(n):
    """Return an array of n bootstrap samples."""
    test_statistics = np.array([])
    for i in range(n):
        statistic = one_statistic()
        test_statistics = np.append(statistic, test_statistics)
    return test_statistics
statistics = bootstrap(10000)
statistics
plt.hist(statistics, density=True, bins=20, edgecolor="black")
plt.scatter(prop_diabetes, 0, color="red")
plt.xlabel("Proportion of people with diabetes")
plt.ylabel("Density")
print("The P-value is: {}".format(np.sum(statistics >= prop_diabetes) / len(statistics)))<jupyter_output>The P-value is: 0.0<jupyter_text>Conclusion of the test Based on P-value, we reject the null hypothesis.  Possible explanations 1. Diabetes was under-diagnosed in the population (i.e., there were a lot of people in the population who had diabetes but weren't diagnosed). By contrast, the Framingham participants were less likely to go undiagnosed because they had regular medical examinations as part of the study.2. The relatively wealthy population in Framingham ate a luxurious diet high in sugar (high-sugar diets are a known cause of diabetes).3. The Framingham Study subjects were older on average than the general population, and therefore more likely to have diabetes.  Cholesterol and Heart Disease  Propose Null and Alternative Hypotheses In the remainder of this question, we are going to examine one of the main findings of the Framingham study: an association between serum cholesterol (i.e., how much cholesterol is in someone's blood) and whether or not that person develops heart disease.We'll use the following null and alternative hypotheses:**Null Hypothesis**: In the population, the distribution of cholesterol levels among those who get heart disease is the same as the distribution of cholesterol levels among those who do not.**Alternative Hypothesis**: The cholesterol levels of people in the population who get heart disease are higher, on average, than the cholesterol level of people who do not.  Choose a test statistic **Test statistic**: the average cholesterol among the population who get heart disease - the average cholesterol level among the population who do not.   The distribution of the test statistic under the null<jupyter_code>def cho_test_statistic(df):
    """Compute one test statistic give the dataframe df.
    
    Args:
        df -- dataframe like framingham 
    
    Return:
        the average of cholesterol level among TenYearCHD=1 
        - the average of cholesterol level among TenYearCHD=0
    """
    series = df.groupby("TenYearCHD")["totChol"].agg(np.mean)
    return series[1] - series[0] 
framingham.groupby("TenYearCHD")["totChol"].agg(np.mean)
observed = cho_test_statistic(framingham)
observed
def simulate_once(df):
    """Shuffle the TenYearCHD label of df and return one test statistic."""
    permuted = np.random.permutation(df["TenYearCHD"])
    copied = df.copy()
    copied["TenYearCHD"] = permuted 
    return cho_test_statistic(copied)
def simulate_n(df, n):
    results = np.array([])
    for i in range(n):
        statistic = simulate_once(df)
        results = np.append(results, statistic)
    return results
chos = simulate_n(framingham, 10000)
chos
plt.hist(chos, bins=20, density=True, edgecolor="black")
plt.xlabel("Test statistic")
plt.ylabel("Density")
plt.title("Density of bootstrapped test statistics under the null")
plt.scatter(observed, 0, color="red")
print("The P-value is: {}".format(np.sum(chos >= observed) / len(chos)))<jupyter_output>The P-value is: 0.0
{'fraction_non_alphanumeric': 0.07663740715732613, 'fraction_numerical': 0.010972316002700877, 'mean_word_length': 4.984848484848484, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 17, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '340173', 'license': 'MIT', 'n_tokens_mistral': 1709, 'n_tokens_neox': 1539, 'n_words': 683, 'path': 'Testing_hypotheses/Testing_hypothesis.ipynb', 'repo_name': 'Hegelim/Enlightment', 'size': '5924'}