<jupyter_start><jupyter_text>Coding exercisesExercises 1-3 are thought exercises that don't require coding. If you need a Python crash-course/refresher, work through the [`python_101.ipynb`](./python_101.ipynb) notebook in chapter 1. Exercise 4: Generate the data by running this cellThis will give you a list of numbers to work with in the remaining exercises.<jupyter_code>import random
random.seed(0)
salaries = [round(random.random() * 1000000, -3) for _ in range(100)]
print(salaries)<jupyter_output>[844000.0, 758000.0, 421000.0, 259000.0, 511000.0, 405000.0, 784000.0, 303000.0, 477000.0, 583000.0, 908000.0, 505000.0, 282000.0, 756000.0, 618000.0, 251000.0, 910000.0, 983000.0, 810000.0, 902000.0, 310000.0, 730000.0, 899000.0, 684000.0, 472000.0, 101000.0, 434000.0, 611000.0, 913000.0, 967000.0, 477000.0, 865000.0, 260000.0, 805000.0, 549000.0, 14000.0, 720000.0, 399000.0, 825000.0, 668000.0, 1000.0, 494000.0, 868000.0, 244000.0, 325000.0, 870000.0, 191000.0, 568000.0, 239000.0, 968000.0, 803000.0, 448000.0, 80000.0, 320000.0, 508000.0, 933000.0, 109000.0, 551000.0, 707000.0, 547000.0, 814000.0, 540000.0, 964000.0, 603000.0, 588000.0, 445000.0, 596000.0, 385000.0, 576000.0, 290000.0, 189000.0, 187000.0, 613000.0, 657000.0, 477000.0, 90000.0, 758000.0, 877000.0, 923000.0, 842000.0, 898000.0, 923000.0, 541000.0, 391000.0, 705000.0, 276000.0, 812000.0, 849000.0, 895000.0, 590000.0, 950000.0, 580000.0, 451000.0, 660000.0, 996000.0, 917000.0, 793000.0, 82000.0, 613000.0, 486000.0]<jupyter_text>Exercise 5: Calculating statistics and verifying mean<jupyter_code>salaries_mean = sum(salaries) / len(salaries)
print(f"mean: {salaries_mean}")

import statistics
salaries_mean_stats = statistics.mean(salaries)
print(f"mean: {salaries_mean_stats}")<jupyter_output>mean: 585690.0
mean: 585690.0<jupyter_text>median<jupyter_code>salaries_sorted = sorted(salaries)
num_entries_is_even = len(salaries) % 2 == 0
if num_entries_is_even:
    salaries_mid_index = int(len(salaries) / 2)
    salaries_median = (salaries_sorted[salaries_mid_index-1] + salaries_sorted[salaries_mid_index]) / 2
else:
    salaries_mid_index = int((len(salaries) - 1) / 2)
    salaries_median = salaries_sorted[salaries_mid_index]
print(f"median: {salaries_median}")

import statistics
salaries_median_stats = statistics.median(salaries)
print(f"median: {salaries_median_stats}")<jupyter_output>median: 589000.0
median: 589000.0<jupyter_text>mode<jupyter_code>import collections
salaries_mode = collections.Counter(salaries).most_common(1)
print(salaries_mode)
print(f"mode: {salaries_mode[0][0]}")

import statistics
salaries_mode_stats = statistics.mode(salaries)
print(f"mode: {salaries_mode_stats}")<jupyter_output>[(477000.0, 3)]
mode: 477000.0
mode: 477000.0<jupyter_text>sample varianceRemember to use Bessel's correction.<jupyter_code>salaries_mean = sum(salaries) / len(salaries)
print(f"mean: {salaries_mean}")
sv_sum = 0
for n in salaries:
    sv_sum += (n - salaries_mean) ** 2
sv = sv_sum / (len(salaries) - 1)
print(f"sample variance: {sv}")

sv = sum([(n - (sum(salaries) / len(salaries))) **2 for n in salaries]) / (len(salaries) - 1)
print(f"sample variance: {sv}")

import statistics
sv_stats = statistics.variance(salaries)
print(f"sample variance: {sv_stats}")<jupyter_output>mean: 585690.0
sample variance: 70664054444.44444
sample variance: 70664054444.44444
sample variance: 70664054444.44444<jupyter_text>sample standard deviationRemember to use Bessel's correction.<jupyter_code>import math
sdev = math.sqrt(sv)
print(f"standard deviation: {sdev}")

import statistics
sdev_stats = statistics.stdev(salaries)
print(f"standard deviation: {sdev_stats}")<jupyter_output>standard deviation: 265827.11382484
standard deviation: 265827.11382484<jupyter_text>Exercise 6: Calculating more statistics range<jupyter_code>range = max(salaries) - min(salaries)
print(f"range: {range}")<jupyter_output>range: 995000.0<jupyter_text>coefficient of variationMake sure to use the sample standard deviation.<jupyter_code>import statistics
cv = statistics.stdev(salaries) / statistics.mean(salaries)
print(f"coefficient of variation: {cv}")<jupyter_output>coefficient of variation: 0.45386998894439035<jupyter_text>interquartile range<jupyter_code>import statistics
quantiles = statistics.quantiles(salaries)
print(f"quantiles: {quantiles}")

# IQR = Q3 - Q1
q3 = quantiles[2]
q1 = quantiles[0]
iqr = q3 - q1
print(f"interquartile range: {iqr}")<jupyter_output>quantiles: [400500.0, 589000.0, 822250.0]
interquartile range: 421750.0<jupyter_text>quartile coefficent of dispersion<jupyter_code>import statistics
quantiles = statistics.quantiles(salaries)
print(f"quantiles: {quantiles}")

# QCD = (Q3 - Q1) / (Q3 + Q1)
q3 = quantiles[2]
q1 = quantiles[0]
qcd = (q3 - q1) / (q3 + q1)
print(f"quartile coefficent of dispersion: {qcd}")<jupyter_output>quantiles: [400500.0, 589000.0, 822250.0]
quartile coefficent of dispersion: 0.34491923941934166<jupyter_text>Exercise 7: Scaling data min-max scaling (normalizing)<jupyter_code>min_salary = min(salaries)
max_salary = max(salaries)
salary_range = max_salary - min_salary
print(f"salary_range: {salary_range}")

salaries_scaled = [(n - min_salary) / salary_range for n in salaries]
print(f"min-max scaling: {salaries_scaled}")

print()

from sklearn.preprocessing import minmax_scale
salaries_scaled_scikit = minmax_scale(salaries)
print(f"min-max scaling: {salaries_scaled}")<jupyter_output>salary_range: 995000.0
min-max scaling: [0.8472361809045226, 0.7608040201005025, 0.4221105527638191, 0.2592964824120603, 0.5125628140703518, 0.40603015075376886, 0.7869346733668342, 0.3035175879396985, 0.47839195979899496, 0.5849246231155779, 0.9115577889447236, 0.5065326633165829, 0.28241206030150756, 0.7587939698492462, 0.6201005025125628, 0.25125628140703515, 0.91356783919598, 0.9869346733668342, 0.8130653266331658, 0.9055276381909547, 0.31055276381909547, 0.7326633165829146, 0.9025125628140703, 0.6864321608040201, 0.4733668341708543, 0.10050251256281408, 0.43517587939698493, 0.6130653266331658, 0.9165829145728643, 0.9708542713567839, 0.47839195979899496, 0.8683417085427135, 0.26030150753768844, 0.8080402010050252, 0.5507537688442211, 0.01306532663316583, 0.7226130653266332, 0.4, 0.828140703517588, 0.6703517587939698, 0.0, 0.49547738693467336, 0.871356783919598, 0.2442211055276382, 0.3256281407035176, 0.8733668341708543, 0.19095477386934673, 0.5698492462311557, 0.23919597989949748, [...]<jupyter_text>standardizing<jupyter_code>import statistics
salaries_mean = statistics.mean(salaries)
salaries_stdev = statistics.stdev(salaries)
salaries_standardized = [((n - salaries_mean) / salaries_stdev) for n in salaries]
print(f"standardized: {salaries_standardized}")<jupyter_output>standardized: [0.9717217942267801, 0.6482032533127501, -0.6195380058503674, -1.228956652688424, -0.28097209094033604, -0.6797275018343729, 0.7460111842867592, -1.0634355387324086, -0.40887476990634786, -0.010119359012310937, 1.2124797781628023, -0.3035431519343381, -1.142434252211416, 0.6406795663147493, 0.12154516345270126, -1.2590514006804265, 1.220003465160803, 1.4946180405878284, 0.8438191152607681, 1.1899087171688003, -1.037102634239406, 0.5428716353407403, 1.178623186671799, 0.36982683438672426, -0.4276839874013496, -1.8233279255304788, -0.5706340403633628, 0.09521225895969881, 1.2312889956578041, 1.434428544603823, -0.40887476990634786, 1.0507205077057873, -1.2251948091894236, 0.8250098977657664, -0.1380220379783228, -2.150608309943509, 0.5052532003507368, -0.7022985628283751, 0.9002467677457734, 0.3096373384027187, -2.199512275430514, -0.34492343042334195, 1.0620060382027885, -1.285384305173429, -0.9806749817544008, 1.0695297252007891, -1.4847620106204475, -0.06654701149731616,[...]<jupyter_text>Exercise 8: Calculating covariance and correlation covariance between standardized and normalized data<jupyter_code>from sklearn.preprocessing import minmax_scale
salaries_normalized = minmax_scale(salaries)

import statistics
salaries_mean = statistics.mean(salaries)
salaries_stdev = statistics.stdev(salaries)
salaries_standardized = [((n - salaries_mean) / salaries_stdev) for n in salaries]

running_total = [
    (x - statistics.mean(salaries_normalized)) * (y - statistics.mean(salaries_standardized))
    for x, y in zip(salaries_normalized, salaries_standardized)
]
salaries_cov = statistics.mean(running_total)
print(f"salaries_cov: {salaries_cov}")

import numpy as np
salaries_cov_matrix = np.cov(salaries_normalized, salaries_standardized)
print(f"salaries_cov_matrix: {salaries_cov_matrix}")
salaries_cov = salaries_cov_matrix[0][1]
print(f"salaries_cov: {salaries_cov}")<jupyter_output>salaries_cov: 0.26449129918250414
salaries_cov_matrix: [[0.07137603 0.26716293]
 [0.26716293 1.        ]]
salaries_cov: 0.267162928467176<jupyter_text>Pearson correlation coefficient ($\rho$)<jupyter_code>import statistics
salaries_rho = salaries_cov / (statistics.stdev(salaries_normalized) * statistics.stdev(salaries_standardized))
print(f"correlation coefficient: {salaries_rho}")<jupyter_output>correlation coefficient: 1.0000000000000004
{'fraction_non_alphanumeric': 0.12314295146913172, 'fraction_numerical': 0.2967976229778805, 'mean_word_length': 7.216998191681736, 'pattern_counts': {'":': 0, '<': 43, '<?xml version=': 0, '>': 43, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '483282', 'license': 'MIT', 'n_tokens_mistral': 5373, 'n_tokens_neox': 3686, 'n_words': 625, 'path': 'ch_01/exercises_sk.ipynb', 'repo_name': 'samkramer/Hands-On-Data-Analysis-with-Pandas-2nd-edition', 'size': '9087'}