<jupyter_start><jupyter_text>Module 2 Sample Mean and Sample Standard Deviation<jupyter_code>import numpy as np
from math import sqrt<jupyter_output><empty_output><jupyter_text>Pb 6.1  Setup<jupyter_code>X = [500, 2000, 2500, 5000, 10000]
n = [96, 107, 130, 89, 78]
n_groups = len(n)
n_people = sum(n)
print("n_groups=", n_groups, " n_people=", n_people)<jupyter_output>n_groups= 5  n_people= 500<jupyter_text>Sample Mean<jupyter_code>x_bar = sum([Xi*ni for Xi, ni in zip(X, n)])/n_people
print("x_bar=", x_bar)<jupyter_output>x_bar= 3624.0<jupyter_text>Equation 6.3 from Montgomery probably looks tempting, right? It turns out you can't use this directly on the table of values given. Why? Because group data is given, whereas Equation 6.3 sums over individuals.  Sample Standard Deviation (Long Way) A simple approach is to convert the group data to data of individuals. This is not the best approach, but it illustrates use of Eq 6.3.<jupyter_code>X[0]
[X[0]]*5
print([X[0]]*n[0])
print([[xsub]*nsub for xsub, nsub in zip(X, n)])<jupyter_output>[[500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500], [2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, [...]<jupyter_text>This gives us a (ragged) list of lists; however, a flattened version will be easier to deal with. From a quick Google search "flatten ragged list of lists", we find an example of how to do this and some sources:https://stackabuse.com/python-how-to-flatten-list-of-lists/https://stackoverflow.com/questions/9057379/correct-and-efficient-way-to-flatten-array-in-numpy-in-pythonhttps://stackoverflow.com/questions/2158395/flatten-an-irregular-list-of-lists<jupyter_code>def flatten(regular_list):
    return [item for sublist in regular_list for item in sublist]
print(flatten(individuals))<jupyter_output>[500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 200[...]<jupyter_text>All together, this would look like:<jupyter_code>def expand_groups(X, n):
    persons = flatten([[xsub]*nsub for xsub, nsub in zip(X, n)])
    return persons
x = expand_groups(X, n)
print("length=",len(x))
print("mean=",np.mean(x))<jupyter_output>length= 500
mean= 3624.0<jupyter_text>Now that we have a flat list of all 500 people and their corresponding deposits, we can plug this directly into Eq 6.3 from the book: ![image.png](attachment:image.png)<jupyter_code>std = sqrt(sum([(xi-x_bar)**2 for xi in x])/(n_people-1))
print("sample standard deviation=", std)<jupyter_output>sample standard deviation= 3076.451382065451<jupyter_text>Sample Standard Deviation (Better Way)  Analytical Formula If you keep reading in Section 6.1 of the textbook, however, it turns there is an easier way to calculate this (Eq 6.4). ![image.png](attachment:image.png) Now that the $x_i$ no longer appear directly next to $\bar{x}$, and because for a single group with m people, we can write: $\sum_{i=1}^{m} x^{2}=m x^{2}$ we can rewrite the following in terms of "groups": $\sum_{i=1}^{n_{people}} x_{i}^{2} = \sum_{i=1}^{n_{groups}} n_i X_{i}^{2}$ and $\sum_{i=1}^{n_{people}} x_{i} = \sum_{i=1}^{n_{groups}} n_i X_{i}$ Substituting this back in, we get: $\frac{\sum_{i=1}^{n_{people}} x_{i}^{2}-\frac{\left(\sum_{i=1}^{n_{people}} x_{i}\right)^{2}}{n_{people}}}{n_{people}-1} = \frac{\sum_{i=1}^{n_{groups}} n_i X_{i}^{2}-\frac{\left(\sum_{i=1}^{n_{groups}} (X_{i}n_i)\right)^{2}}{n_{people}}}{n_{people}-1}$  Coded Version And finally, to convert the RHS of this equation to code:<jupyter_code>def frequency_sample_std_dev(X, n):
    """Sample standard deviation for X and n,
    where X[i] is the quantity each person in group i has,
    and n[i] is the number of people in group i."""
    n_groups = len(n)
    n_people = sum(n)
    lhs_numerator = sum([ni*Xi**2 for Xi, ni in zip(X, n)])
    rhs_numerator = sum([Xi*ni for Xi, ni in zip(X,n)])**2/n_people
    denominator = n_people-1
    var = (lhs_numerator - rhs_numerator) / denominator
    std = sqrt(var)
    return std
std = frequency_sample_std_dev(X, n)
print(std)<jupyter_output>3076.451382065451<jupyter_text>Example with More Data If we had an entirely different set of values or even a different problem with the same format, we can compute this simply as:<jupyter_code>X2 = [500, 2000, 2500, 5000, 10000, 12000, 16000, 20000, 22500]
n2 = [96, 107, 130, 89, 78, 48, 158, 62, 95]
std2 = frequency_sample_std_dev(X2, n2)
print("sample standard deviation=",std2)<jupyter_output>sample standard deviation= 7755.100282255846<jupyter_text>Other Code Sources This technique is simple enough that it's likely someone has already written some code and made it publicly available. Using a few different search queries: "sample standard deviation python", "sample standard deviation with repeats python", and finally settling on the following webpage:  [Weighted standard deviation in NumPy](https://stackoverflow.com/questions/2413522/weighted-standard-deviation-in-numpy). Having the right keywords is very important (a search for weighted standard deviation would quickly pull up what we're looking for), and we also need to make sure that the code we're using is appropriate for the use-case. Here are a few examples from the linked StackOverflow post:  Population Average and Standard Deviation<jupyter_code>def weighted_avg_and_std(values, weights):
    """
    Return the weighted average and standard deviation.

    values, weights -- Numpy ndarrays with the same shape.
    """
    average = np.average(values, weights=weights)
    # Fast and numerically precise:
    variance = np.average((values-average)**2, weights=weights)
    return (average, sqrt(variance))

print(weighted_avg_and_std(X, n))<jupyter_output>(3624.0, 3073.3733909175435)<jupyter_text>Sample Average and Standard Deviation Note that this gave a "population" standard deviation rather than a "sample" standard deviation. This can be corrected as follows:<jupyter_code>def weighted_sample_avg_std(values, weights):
    """
    Return the weighted average and weighted sample standard deviation.

    values, weights -- Numpy ndarrays with the same shape.
    
    Assumes that weights contains only integers (e.g. how many samples in each group).
    
    See also https://en.wikipedia.org/wiki/Weighted_arithmetic_mean#Frequency_weights
    """
    average = np.average(values, weights=weights)
    variance = np.average((values-average)**2, weights=weights)
    variance = variance*sum(weights)/(sum(weights)-1)
    return (average, sqrt(variance))

print(weighted_sample_avg_std(X, n))<jupyter_output>(3624.0, 3076.451382065451)
{'fraction_non_alphanumeric': 0.13744757339724387, 'fraction_numerical': 0.18514080287597365, 'mean_word_length': 5.15033161385409, 'pattern_counts': {'":': 1, '<': 35, '<?xml version=': 0, '>': 35, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 20, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '179541', 'license': 'MIT', 'n_tokens_mistral': 4308, 'n_tokens_neox': 2780, 'n_words': 1029, 'path': 'python/module-2-sample-deviation.ipynb', 'repo_name': 'sparks-baird/mete-3070', 'size': '8345'}