<jupyter_start><jupyter_text>Data and Meaning Lab PH 211 COCC Bruce Emerson 1/12/2021This notebook is meant to provide tools and discussion to support data analysis and presentation as you generate your lab reports. [Data/Meaning Lab](http://coccweb.cocc.edu/bemerson/PhysicsGlobal/Courses/PH211/PH211Materials/PH211Labs/PH211Labdatamean21.html) and [Data/Meaning Lab Discussion](http://coccweb.cocc.edu/bemerson/PhysicsGlobal/Courses/PH211/PH211Materials/PH211Labs/PH211LabDdatamean.html)In this lab we are gathering some data, entering the data into the notebook, extracting useful statistical measures, plotting histograms of the data, and finally seeking to demonstrate our understanding of the meaning of all that went before.For the formal lab report you will want to create your own description of what you understand the process and intended outcome of the lab is. Please don't just copy the purpose statement from the lab page.  DependenciesAs we talked about previously there are a number of what are called libraries that allow us to add tools to our notebooks. These libraries only need to be accessed once in a given notebook. The general approach is to load any needed tools at the begining of the notebook. These libraries themselve use other libraries which are called dependencies. This is how we load the various libraries of python tools that are needed for the particular work we are undertaking. To access commands from these libraries we have to type the entire name of the library which is often aggravating and time consuming. More typically we give the imported library an "alias' which is a shorthand for the long library name. The language to bring the library onboard and to give it an alias is ```import [library] as [alias]```.  ```numpy``` is a numerical tools library - often imported as np. ```numpy``` also contains the statistical tools that we will use in this lab. There are other libraries dedicated to statistical tools but 'numpy' has everything we need.  ```matplotlib```is a 'MATLAB like' library. ```matplotlib.pyplot``` is often imported as ```plt``` to make it easier to use. ```matplotlib``` has the plotting tools that we need for this lab.  The following code cell will need to be run first before any other code cells. This sometimes takes a while because a lot of code is being loaded during this process.<jupyter_code>import numpy as np
import matplotlib.mlab as mlab
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>*** Lab Deliverable:For your lab notebook you will include the usual 'header' information we talked about last week in the first cell of the lab (a markdown cell for sure). In the next cell please describe how you determined the HR for you. Include a description of your data gathering process and the math that supports your results. I expect to see a few lines of math illustrating that calculation. After that include the code cell above that has the dependencies that we need for this week's lab. Remember that you need to execute (shift-return or shift-enter) the code cell before executing any other code cells.***  Lists (Vectors)We usually talk about our data as a list of data points. In spreadsheets we enter data in a set of successive cells in a column or a row. There is a creature in python called a list which is a way of accomplishing a similar task.  [python documentation: list](https://docs.python.org/3.7/tutorial/datastructures.html) This link to the python documentation for lists will bury you in features and uses of lists. It is easy and common to overwhelmed by the deluge of information which is why these notebooks are created. I include this reference link so you could explore more deeply if you have time to do so but what you actually need to understand for this lab is contained within this notebook.Our lists need names so we can work with them later on. As we have disucssed it is best practice to use names that have relevant meaning. Note the square brackets and the comma as a separator between data points. The data here is made up so you will surely replace it with your data! Explore:Things to explore include putting an extra decimal point somewhere, shortening up one of the lists. Change names of things to see what is allowed and what isn't. *** Lab Deliverable:Once we collect the data on our HR (height ratios) from the discussion folder on Bb you will enter the data for the two groups here. Depending on how we divide the class you may well want to change the labels on the two data sets to something more appropriate. For now this means typing the data points in by hand and making sure that they match the information you were given. You should expect that both data sets will have **different** numbers of data points.In the markdown cell describing the data include a description of the distinction between the two sets of data that we are exploring. This will have been discussed in class and will have some sort of rationale for why we made this choice.***<jupyter_code>data1 = [.21,.22,.23,.4,.5,.6,.77,.8,.9,.10,.31,.2,.33,.34,.35,.36,.37,.18,.49,.50,.100]
data2 = [.41,.52,.43,.54,.65,.6,.67,.58,.9,.50,.1,.72,.3,.84,.5,.6,.4,.5,.69,.50,.100]
# 2 ways to print out and check your data
print(data1)
print("Second data set:",data2)<jupyter_output>[0.21, 0.22, 0.23, 0.4, 0.5, 0.6, 0.77, 0.8, 0.9, 0.1, 0.31, 0.2, 0.33, 0.34, 0.35, 0.36, 0.37, 0.18, 0.49, 0.5, 0.1]
Second data set: [0.41, 0.52, 0.43, 0.54, 0.65, 0.6, 0.67, 0.58, 0.9, 0.5, 0.1, 0.72, 0.3, 0.84, 0.5, 0.6, 0.4, 0.5, 0.69, 0.5, 0.1]<jupyter_text>Check data pointsThere are various ways you might choose to check your data points. You can look through the list above or you can ask python to count how many data points for you as shown below. Check that the number of data points matches your expectation. In the long run we will use this technique to be sure we have the same number of x data points as y data points when we make plots.<jupyter_code>data1length = len(data1)
data2length = len(data2)
# length counts how many 'data points' in the list
print("number of data points (1):", data1length)
print("number of data points (2):", data2length)<jupyter_output>number of data points (1): 21
number of data points (2): 21<jupyter_text>Statistics[numpy statistics](https://docs.scipy.org/doc/numpy/reference/routines.statistics.html)The ```numpy``` library is a project of [SciPy.org](https://www.scipy.org/) that has created almost all of the tools that we will use. We need mean, mediam, median, min, max, and standard deviation. Here's how you get them....```np.mean(data1)``````np.``` tells the notebook that you want to use a tool from the numpy library which we imported as np in the dependencies cell.```np.mean()``` tells the notebook that we want to use the tool called ```mean()```. Within the parentheses go any information that the tool wants in order to perform it's task. These bits of information are called 'arguments' that being 'passed' to the tool. Sometimes there are many arguments and sometimes only one. The arguments have to have the correct form or numpy and python will complain. The first time I show you a command I will try to remember to provide a link to the documention for that command. This documentation is often very difficult to read and understand for novices but I include it for completeness. You do **NOT** need to read the documentation to successfullly complete the lab.[np.mean documentation](https://numpy.org/doc/stable/reference/generated/numpy.mean.html)  [np.median documentation](https://numpy.org/doc/stable/reference/generated/numpy.median.html)  [np.std documentation](https://numpy.org/doc/stable/reference/generated/numpy.std.html)  [np.min documentation](https://numpy.org/doc/stable/reference/generated/numpy.min.html)  [np.max documentation](https://numpy.org/doc/stable/reference/generated/numpy.max.html)<jupyter_code>data1mean = np.mean(data1)
data1median = np.median(data1)
data1stddev = np.std(data1)
data1min = np.min(data1)
data1max = np.max(data1)
print("Data set 1 Mean",data1mean)
print("Data set 1 Median",data1median)
print("Data set 1 Std Dev",data1stddev)
print("Data set 1 min",data1min)
print("Data set 1 max",data1max)
data2mean = np.mean(data2)
data2median = np.median(data2)
data2stddev = np.std(data2)
data2min = np.min(data2)
data2max = np.max(data2)
print("Data set 2 Mean",data2mean)
print("Data set 2 Median",data2median)
print("Data set 2 Std Dev",data2stddev)
print("Data set 2 min",data2min)
print("Data set 2 max",data2max)<jupyter_output>Data set 2 Mean 0.5261904761904761
Data set 2 Median 0.52
Data set 2 Std Dev 0.19621359087135223
Data set 2 min 0.1
Data set 2 max 0.9<jupyter_text>Basic HistogramFrom a structural point of view the plt.hist command creates the histogram which is displayed using the plt.show() command. THis is how many coding languages work. The code creates objects, in this case the histogram, and then there is a command that displays the object that is created. I don't want you to get down in the weeds about how that works this week. We will slowly move in that direction. As the plt.hist command generates the plot it also returns three bits of information - the heights of each bin (array), the edges of each bin (array), and the patch objects that are the bars that are plotted. For now we can ignore these information items that are returned from the ```plt.hist``` command. [matplotlib.pyplot.hist](https://matplotlib.org/3.1.1/api/_as_gen/matplotlib.pyplot.hist.html)    Histogram StylesIn the ```plt.hist``` command there are a number of arguments that can be provided to control the features of the histogram that is produced. In the code I have broken each argument out onto an individual line to make them more clear. ```height, bins, patches = plt.hist(data1,                                 num_bins,                                 fullrange,                                 histtype = "barstacked",                                 color="tab:blue",                                 alpha= .7                                )``````data1``` is the name of the data list (vector) I want to generate the data from.  ```num_bins``` is an integer that tells the command how many different divisions I want across the possible value of my histogram. You will explore different values for num_bins to understand the impact of this argument on the how the histogram looks.```fullrange``` is a sort vector [x_low,x_high] that defines the upper and lower bounds of your histogram. Notice you can leave space around the histogram or end up packing everything into the edges. ```histtype = "barstacked"``` tells the command what styleThe other options for 'histtype' are "barstacked", "step", and "stepfilled". The "stepfilled" style solves some of the consideration discussed above.```color="tab:blue"``` allows us to choose the color for the histogram. In case you want to play here are the available colors--[plot colors](https://matplotlib.org/3.1.0/gallery/color/named_colors.html)```alpha= .7``` sets the density of the color plot. 0 is totally transparent and 1. is totally opaque.  *** Lab Deliverable:Play with the arguments for the histogram until you feel like you have some comfort with them. Be sure to do your playing in a copy of the lab because of the possibility that you will make some seemingly insignificant change that will keep it from working. Being able to go back to the original and copy-paste to get back to the original code seems important. We won't use histograms a lot in this class but this is a good time to experiment with getting comfortable 'messing around' in the python code. Try to generate a sense of confidence that you can recover if something goes horribly wrong.***<jupyter_code># choose how many bins you want to display
# ...also set the histogram type in the arguments
# Just data1 in this histogram
num_bins = 12
fullrange = [0,1]
height, bins, patches = plt.hist(data1,
                                 num_bins,
                                 fullrange,
                                 histtype = "barstacked",
                                 color="tab:blue",
                                 alpha= .6
                                )
plt.show()
# choose how many bins you want to display
# ...also set the histogram type in the arguments
# Just data2 in this histogram, also changed type of histogram
num_bins = 12
fullrange = [0,1]
height, bins, patches = plt.hist(data2,
                                 num_bins,
                                 fullrange,
                                 histtype = "step",
                                 color="red",
                                 alpha= .4
                                )
plt.show()
# choose how many bins you want to display
# ...also set the histogram type in the arguments
# Just both data1 and data2 in this histogram, also changed type of histogram
num_bins = 12
fullrange = [0,1]
height, bins, patches = plt.hist([data1,data2],
                                 num_bins,
                                 fullrange,
                                 histtype = "barstacked",
                                 color=["tab:blue","red"],
                                 alpha= .57
                                )
plt.show()<jupyter_output><empty_output><jupyter_text>*** Lab Deliverable:Now produce 3 histograms of our actual data (practice with this data but in the end it's our class data) that demonstrate your understanding of the 'Goldilocks Triplet'. One with too few bins to be meaningful, one with too many, and one that is 'just right'. Describe, in your own words, how you understand this triplet.*** *** Lab Deliverable: Analysis IWhere is your personal data point in the histogram of the data (don't worry about plotting it just describe). Is 'your' data point consistent with other members of your group? Could you be easily confused with members of the other group? ***  In Case You're Curious....**NOT REQUIRED!**Many times I will include in the lab notebook examples of enhanced tools that go way beyond what I expect for the lab. If you have a coding background or you're just curious you can look through these examples to get a sense of some additional options. To be clear, there are truly remarkable things you can do in python plots with pyplot and the other tools in ```matplotlib```. The link below takes you to the ```matplotlib``` example gallery which is overwhelming.[Matplotlib](https://matplotlib.org/3.1.1/gallery/index.html) This Example:**Assign a color name:** As I make the plot more complex I will want to use the same color for everything associated with data set 1 and a different color for data set 2. In each plot element I will have to tell the plotting code what color to use. If I then change the color I have to go manually change it in multiple lines of code. That's aggravating. It's much easier to create a variable that 'holds' the color that I want. For those who are new to coding this sort of variable is called a ```string``` variable because it contains a string of characters. To define a string variable (simplest way) set the variable equal to some set of characters within quotes. Python will automatically assume that you intend for the variable to be a string of characters. ```data1Color = "tab:blue"data2Color = "tab:red"```**Create a Figure with option for multiple plots**```figHist, axHist = plt.subplots()```This command does two things. It creates a overall object called a figure that contains all the plots I want to make. Within that figure are a number of axes objects which are the individual plots within the figure. If I have just one plot I create just one set of axes. In this case the figure is called figHist and the only 'plot' is called axHist.**Create a vertical line to indicate the mean:**We could certainly create a vertical line as a function but it's a common enough need that pyplot has a built in function to do this called, remarkably, ```vlines```. Here is the format and the meaning of the arguments.```axHist.vlines(data1mean,              0,              10,              color = data1Color,              linestyle = '-',              linewidth = 2.,              label = "data1 mean")axHist.vlines(data2mean,```argument 1: data1mean -- 'x' value where I want the vertical line placed -- the data1mean in this casearg2, arg3: minimum and maximum y values for the vertical line. In this case from 0 to 10. color, linestyle, linewidth, and label are standard style arguments that are similar in all pyplot commands.**Size of the Figure:**```figHist.set_size_inches(10, 9)```This command to the figure sets the physical size of the plot in inches. Amusingly or irritatingly, there is no direct metric equivalent such as set_size_cm. This allows me to control the size of the plot for visibility.**Show Grid Lines**```axHist.grid()```For each plot object you can choose to show the grid associated with your axis variables this can make it easier to see what the values are on the plot. I'm sure that are some interesting effects that I can get with the arguments for the grid command but I haven't explored them yet.**Legend**```plt.legend(loc=1)```The legend is a helpful tool for reminding the viewer which color or line represents what feature of your data. There are many options here but the only one I am using at the moment is the loc (location) argument. This tells pyplot where I want the legend to be placed on the plot. Mostly pyplot does pretty good putting it out of the way by default but if you need to control the location try changing the 1 (1 = top right) to other values. loc must be an integer from 1 to 9.<jupyter_code># Choose my number of bins and the range of the histogram again.
num_bins = 10
fullrange = [0,1]

# Assign particular colors to each data set
data1Color = "tab:blue"
data2Color = "tab:red"

# Explicitly create a figure with a particular plot 'axHist' contained within the figure
figHist, axHist = plt.subplots()
height, bins, patches = axHist.hist([data1,data2],
                                 num_bins,
                                 fullrange,
                                 histtype = "bar",
                                 color=[data1Color,data2Color],
                                 alpha= .5,
                                 label=['data1','data2'])

# Reminder -- we have already determined data1mean and data2mean
# see previous cell for description of arguments.

axHist.vlines(data1mean,
              0,
              10,
              color = data1Color,
              linestyle = '-',
              linewidth = 2.,
              label = "data1 mean")
axHist.vlines(data2mean,
              0,
              10,
              color = data2Color,
              linestyle = '-',
              linewidth = 2.,
              label = "data2 mean")

# Set the size of my plot for better visibility
figHist.set_size_inches(10, 9)

# turn on grid lines to help interpretation
axHist.grid()

# Label the figure
plt.xlabel('height ratio')
plt.ylabel('frequency')
plt.title('Histogram of HR (Height Ratios)')

# show a legend for what the different parts of the plot mean. The legend grabs the 'lable'
# from each plot command to create this
plt.legend(loc=1)
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07369021091361351, 'fraction_numerical': 0.0219723273047624, 'mean_word_length': 3.367051369088029, 'pattern_counts': {'":': 0, '<': 22, '<?xml version=': 0, '>': 22, 'https://': 11, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 29, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '584783', 'license': 'MIT', 'n_tokens_mistral': 5252, 'n_tokens_neox': 4875, 'n_words': 2644, 'path': 'Other Labs/DataMeaningLab21.ipynb', 'repo_name': 'JNichols-19/PH211', 'size': '19297'}