<jupyter_start><jupyter_text>**Download** (right-click, save target as ...) this page as a jupyterlab notebook from: [Lab21](http://54.243.252.9/engr-1330-webroot/8-Labs/Lab21/Lab21.ipynb)___  Laboratory 21: "Towards Hypothesis Testing"LAST NAME, FIRST NAMER00000000ENGR 1330 Laboratory 21 - In-Lab  Hypothesis TestingHypothesis tests are methods to quantify if two groups of data are similar or different.In this lab we will just get started using mostly exploratory data analysis and histograms, and will explore the concept in increasing detail over the next few labs.First import some necessary packages:<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>Next lets get a database to work with, in this case the database is simply two collections of numerical values.<jupyter_code>######### CODE TO AUTOMATICALLY DOWNLOAD THE DATABASE ################
#! pip install requests #install packages into local environment
import requests # import needed modules to interact with the internet
# make the connection to the remote file (actually its implementing "bash curl -O http://fqdn/path ...")
remote_url = 'http://54.243.252.9/engr-1330-webroot/8-Labs/Lab21/Lab21_data.csv' # a csv file
response = requests.get(remote_url) # Gets the file contents puts into an object
output = open('Lab21_data.csv', 'wb') # Prepare a destination, local
output.write(response.content) # write contents of object to named local file
output.close() # close the connection
mydata = pd.read_csv("Lab21_data.csv") 
mydata<jupyter_output><empty_output><jupyter_text>Question 1What are the names of the two series in "mydata"?<jupyter_code># put your answer here<jupyter_output><empty_output><jupyter_text>Question 2Describe the two data series, which has a larger mean value, which has a larger variance?<jupyter_code># put your answer here<jupyter_output><empty_output><jupyter_text>Now lets prepare histograms of the two data series, an easy way to generate two histoprgams on the same plot is listed below<jupyter_code>fig, ax = plt.subplots()
mydata.plot.hist(density=False, ax=ax, title='Histogram: Set1 vs. Set2', bins=40)
ax.set_ylabel('Count')
ax.grid(axis='y')<jupyter_output><empty_output><jupyter_text>Question 3: **Are the two data series similar or not?**Describe (using words, and sentences, not the method) how the series are different.<jupyter_code># put your answer here<jupyter_output><empty_output><jupyter_text>Now lets generate two more series using the descriptive statistics from "Set1" and "Set2"<jupyter_code>set1_s = np.random.normal(np.array(mydata['Set1']).mean(), np.array(mydata['Set1']).std(), 100) # random sample from a normal distribution function
set2_s = np.random.normal(np.array(mydata['Set2']).mean(), np.array(mydata['Set2']).std(), 100) # random sample from a normal distribution function<jupyter_output><empty_output><jupyter_text>Put these into a new dataframe<jupyter_code>mydata_d = pd.DataFrame({'Set1s':set1_s,'Set2s':set2_s}) # make into a dataframe _d == derived<jupyter_output><empty_output><jupyter_text>Now lets prepare histograms of the two data series, an easy way to generate two histograms on the same plot is listed below<jupyter_code>fig, ax = plt.subplots()
mydata_d.plot.hist(density=False, ax=ax, title='Histogram: Set1 samples vs. Set2 samples', bins=40)
ax.set_ylabel('Count')
ax.grid(axis='y')<jupyter_output><empty_output><jupyter_text>Question 4: **Are the two new data series similar or not?**Describe (using words, and sentences, not the method) how the series are different.<jupyter_code># put your answer here<jupyter_output><empty_output><jupyter_text>Now lets examine all 4 data series. First a histogram of all 4 on the same graph<jupyter_code>fig, ax = plt.subplots()
mydata_d.plot.hist(density=False, ax=ax, title='Histogram: Set1 and Set1 samples vs. Set2 and Set2 samples', bins=40,alpha=0.5)
mydata.plot.hist(density=False, ax=ax, bins=40,alpha=0.5)

ax.set_ylabel('Count')
ax.grid(axis='y')<jupyter_output><empty_output><jupyter_text>Question 5:Are the series "Set1" and "Set1s" the same or different?  How do they compare? What about series "Set2" and "Set2s"?<jupyter_code># put your answer here<jupyter_output><empty_output><jupyter_text>Another graphical tool is a set of boxplots<jupyter_code>fig = plt.figure(figsize =(10, 7)) 
plt.boxplot ([set1, set1_s, set2, set2_s],1, '')
plt.xticks([1, 2, 3, 4], ['Set1', 'Set1_s', 'Set2', 'Set2_s'])
plt.show()<jupyter_output><empty_output><jupyter_text>Question 6Interpret the results of the boxplot.  Are the Set2 "collections" different from the Set1 "collections"?<jupyter_code># put your answer here<jupyter_output><empty_output><jupyter_text>Question 7Suppose we are comparing the arithmetic means of the 4 collections.  Are the mean values of "Set1" and "Set2" far apart?  How many "Set1" standard deviations is the "Set1" mean value from "Set2"?  How about the converse (Set2 deviations)?<jupyter_code># put your answer here<jupyter_output><empty_output><jupyter_text>Question 8Visit [SixSigma](https://en.wikipedia.org/wiki/Six_Sigma) and after reading the Wiki, decide if the Set1 and Set2 collections are far enough apart to be considered "statistically" different.Repeat with Set1 and Set1s - are they far apart?<jupyter_code># put your answer here<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10873207968720909, 'fraction_numerical': 0.025879724446099425, 'mean_word_length': 6.6852646638054365, 'pattern_counts': {'":': 0, '<': 65, '<?xml version=': 0, '>': 65, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 2, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '398948', 'license': 'CC0-1.0', 'n_tokens_mistral': 1849, 'n_tokens_neox': 1723, 'n_words': 558, 'path': '8-Labs/Lab21/Lab21.ipynb', 'repo_name': 'dustykat/engr-1330-psuedo-course', 'size': '5371'}