<jupyter_start><jupyter_text>Problem Set 3<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import math<jupyter_output><empty_output><jupyter_text>1. Explore Distributions 1.<jupyter_code>S = 10_000 #samples size is 10k<jupyter_output><empty_output><jupyter_text>2.<jupyter_code>LN1 = np.random.lognormal(0, 0.2, size = S) 
LN2 = np.random.lognormal(0, 0.5, size = S)
LN3 = np.random.lognormal(0, 1.68, size = S)
plt.hist(LN1, bins = 25, edgecolor = "black")
plt.title("Linear-Linear of (0,0.2)")
plt.show()

plt.hist(np.log(LN1), bins = 25, edgecolor = "black")
plt.title("Log-Linear of (0,0.2)")
plt.show()
plt.hist(LN2, bins = 25, edgecolor = "black")
plt.title("Linear-Linear of (0,0.5)")
plt.show()

plt.hist(np.log(LN2), bins = 25, edgecolor = "black")
plt.title("Log-Linear of (0,0.2)")
plt.show()
plt.hist(LN3, bins = 30, edgecolor = "black")
plt.title("Linear-Linear of (0,1.68)")
plt.show()

plt.hist(np.log(LN3), bins = 25, edgecolor = "black")
plt.title("Log-Linear of (0,1.68)")
plt.show()<jupyter_output><empty_output><jupyter_text>The linear-linear versions of the graphs all seem to be somewhat skewed right, as most values are on the left. The log-linear versions seem to be much more normal.  3.For linear-linear graphs, (all of the log-linear graphs look relatively equal) I believe that the (0,1.68) distribution looks to be the most unequal, whereas the (0,0.2) graph looks to be the most equal.  4.<jupyter_code># Computing sample means
LN1m = np.mean(LN1)
LN2m = np.mean(LN2)
LN3m = np.mean(LN3)
# Computing variance
LN1v = np.var(LN1)
LN2v = np.var(LN2)
LN3v = np.var(LN3)
print("LN1 mean: ", LN1m, " LN2 mean: ", LN2m, " LN3 mean: ", LN3m)
print("LN1 variance: ", LN1v, " LN2 variance: ", LN2v, " LN3 variance: ", LN3v)
# Computing expected value via theoretical formulas:
LN1ex = 1 + 0.5 * 0.2**2
LN2ex = 1 + 0.5 * 0.5**2
LN3ex = 1 + 0.5 * 1.68**2
print("LN1 ev: ", LN1ex, " LN2 ev: ", LN2ex, " LN3 ev: ", LN3ex)
# Computing variance via theoretical formulas
LN1var = (math.e**(0.2**2)) * (math.e**(0.2**2) - 1)
LN2var = (math.e**(0.5**2)) * (math.e**(0.5**2) - 1)
LN3var = (math.e**(1.68**2)) * (math.e**(1.68**2) - 1)
print("LN1 theoretical variance: ", LN1var, " LN2 theoretical variance: ", LN2var,
      " LN3 theoretical vairance: ", LN3var)<jupyter_output>LN1 ev:  1.02  LN2 ev:  1.125  LN3 ev:  2.4112
LN1 theoretical variance:  0.04247629348257031  LN2 theoretical variance:  0.3646958540123865  LN3 theoretical vairance:  265.99982369887033<jupyter_text>All of the theoretical vs. automatically calculated values are relatively similar except for the variance of the (0, 1.68) range sample.  5.The numbers printed below shows what percent of the values is owned by what percent of the distribution. The top number represents the 50th percentile, and the bottom represents the 1st percentile.<jupyter_code># 20/80 for LN1
LN1_sum = LN1.sum() # sum of all values
for i in range(50,100): # ranges through percentiles 1%-50%
    thresh = np.percentile(LN1, i) # gets the percentile, creates a threshold
    top_10 = LN1[LN1 > thresh].sum() # sums all numbers above said threshold
    print(top_10/LN1_sum)
# 20/80 for LN2
LN2_sum = LN2.sum() # sum of all values
for i in range(50,100): # ranges through percentiles 1%-50%
    thresh = np.percentile(LN2, i) # gets the percentile, creates a threshold
    top_10 = LN2[LN2 > thresh].sum() # sums all numbers above said threshold
    print(top_10/LN2_sum)
# 20/80 for LN1
LN3_sum = LN3.sum() # sum of all values
for i in range(50,100): # ranges through percentiles 1%-50%
    thresh = np.percentile(LN3, i) # gets the percentile, creates a threshold
    top_10 = LN3[LN3 > thresh].sum() # sums all numbers above said threshold
    print(top_10/LN3_sum)<jupyter_output>0.9530369773693516
0.9505190941556811
0.9478937965040982
0.9451520231864633
0.9422668688140078
0.9392491890358681
0.9361171307326928
0.9328420932532197
0.9293985867065119
0.9257777945947043
0.9219907396364283
0.9180465864413963
0.913934170534137
0.9096337271855907
0.9051354807343784
0.9004266629362444
0.8955261048681405
0.8904000253700182
0.8849808864830346
0.879277711265337
0.8733265068769904
0.8670864403608729
0.8605233041858317
0.8536603064166602
0.8464636770033239
0.8389208636045846
0.831000988999406
0.8226336863219931
0.813779336694813
0.8044609774787469
0.7945949270250113
0.7841231293134144
0.7729644700494941
0.7611041462375308
0.7484559914600976
0.7349094649859526
0.7201795863966665
0.7042668395935902
0.6871076903933924
0.6684634312881244
0.6481984329962815
0.625977619914587
0.6010816065536835
0.5731042335211568
0.541207881554473
0.5048276658067287
0.461784872577333
0.40859783558882223
0.34156312181981047
0.24564221751330223<jupyter_text>6.<jupyter_code>labor = pd.read_csv("../Data/treatment.csv.bz2", sep = "\t")
labor = labor['re78'].dropna() # makes an re78 only data frame, drops na's
citation = pd.read_csv("../Data/mag-30k-citations.csv.bz2", sep = "\t")
citation = citation['citations'].dropna() # makes an citations only data frame, drops na's
lunar = pd.read_csv("../Data/lunar-impact-craters_v08-2015-09.csv.bz2", sep = "\t")
lunar = lunar['7. Radius [m]'].dropna() # makes a 7. Radius [m] only data frame, drops na's
plt.hist(labor, bins = 30, edgecolor = "black")
plt.title("Labor histogram")
plt.show()

plt.hist(citation, bins = 30, edgecolor = "black")
plt.title("Citations histogram")
plt.show()

plt.hist(lunar, bins = 30, edgecolor = "black")
plt.title("Lunar impacts histogram")
plt.show()<jupyter_output><empty_output><jupyter_text>These most definitely look more like linear-linear graphs, as they are all considerably right skewed.  2. Explore Central Limit Theorem<jupyter_code># Function that calculates the expected value of user inputted list using probability
def expected_value(values, weights):
    values = np.asarray(values)
    weights = np.asarray(weights)
    return (values * weights).sum() / weights.sum()
first = np.random.randint(0,2, size = 10)*2 - 1 # Generates the Bernoulli-distributed numbers
first_prob = [0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5] # The probability of the random numbers

expected = (expected_value(first, first_prob))
print("Expected value is: ", expected)
# Function that calculates the variance of user inputted list
def variance(user):
    avg = sum(user) / len(user)
    var1 = sum((x-avg)**2 for x in user) / len(user)
    return var1
print("Variance is : ", variance(first))<jupyter_output>Expected value is:  0.6
Variance is :  0.6400000000000002<jupyter_text>2.<jupyter_code>R = 1_000 # Number of repititions<jupyter_output><empty_output><jupyter_text>3.  **Not sure if I am supposed to do random of the array that was given (-1,1) or something else<jupyter_code>S = 1
x = np.random.randint(0,2, size = (R,S))*2 - 1 
plt.hist(x, bins = 100, edgecolor = "black")
plt.title("1000 repititions -1,1")
plt.show()<jupyter_output><empty_output><jupyter_text>4.<jupyter_code>print("Mean is: ", np.mean(x), "Variance is: ", np.var(x))<jupyter_output>Mean is:  0.022 Variance is:  0.999516<jupyter_text>This is interesting, the mean is much less than the theoretical expected value, however the sample variance is not too far off the theoretical variance.   5.<jupyter_code>S = 2
y = np.random.randint(0,2, size = (R,S))*2 - 1 
y_means = y.mean(axis=1)
plt.hist(y_means, bins = 'auto', edgecolor = "black")
plt.title("Sample size 2 means")
plt.show()<jupyter_output><empty_output><jupyter_text>This graph looks very normal  6.<jupyter_code>mean_means = np.mean(y_means)
var_means = np.var(y_means)
print("Sample mean of means: ", mean_means, "Sample variance of means: ", var_means)<jupyter_output>Sample mean of means:  -0.001 Sample variance of means:  0.530999<jupyter_text>7.<jupyter_code>bruh = np.repeat(0.5, R)
print("Theoretical EX: ", expected_value(y_means, bruh))
print("Theoretical variance: ", variance(y_means))<jupyter_output>Theoretical EX:  -0.001
Theoretical variance:  0.5309989999999978<jupyter_text>The theoretical and sample calculated values are the same.  8.<jupyter_code>S = 5
q = np.random.randint(0,101,size = (R,S))
# Theoretical mean:
bruh1 = np.full(shape = (R,S), fill_value = 0.01)
print("Theoretical EX: ",expected_value(q, bruh1))

# Sample mean:
print("Sample mean: ",np.mean(q))

#Theoretical variance:
print("Theoretical variance: ", np.mean(variance(q)))

# Sample variance:
print("Sample variance: ", np.var(q))

# Histogram:
q_means = q.mean(axis=1)
plt.hist(y_means, bins = 100, edgecolor = 'black')
plt.title("5-tuple plot")
plt.show()<jupyter_output>Theoretical EX:  49.3392
Sample mean:  49.3392
Theoretical variance:  864.1939867999997
Sample variance:  864.8009433600001<jupyter_text>9.<jupyter_code>S = 25
q = np.random.randint(0,101,size = (R,S))
# Theoretical mean:
bruh1 = np.full(shape = (R,S), fill_value = 0.01)
print("Theoretical EX: ",expected_value(q, bruh1))

# Sample mean:
print("Sample mean: ",np.mean(q))

#Theoretical variance:
print("Theoretical variance: ", np.mean(variance(q)))

# Sample variance:
print("Sample variance: ", np.var(q))

# Histogram:
q_means = q.mean(axis=1)
plt.hist(q_means, bins = 100, edgecolor = 'black')
plt.title("25-tuple plot")
plt.show()<jupyter_output>Theoretical EX:  50.02176
Sample mean:  50.02176
Theoretical variance:  846.3010076800002
Sample variance:  847.1136865024<jupyter_text>10.<jupyter_code>S = 1000
q = np.random.randint(0,101,size = (R,S))
# Theoretical mean:
bruh1 = np.full(shape = (R,S), fill_value = 0.01)
print("Theoretical EX: ",expected_value(q, bruh1))

# Sample mean:
print("Sample mean: ",np.mean(q))

#Theoretical variance:
print("Theoretical variance: ", np.mean(variance(q)))

# Sample variance:
print("Sample variance: ", np.var(q))

# Histogram:
q_means = q.mean(axis=1)
plt.hist(q_means, bins = 100, edgecolor = 'black')
plt.title("100-tuple plot")
plt.show()<jupyter_output>Theoretical EX:  50.012483999999965
Sample mean:  50.012484
Theoretical variance:  849.058388486
Sample variance:  849.9098981497438
{'fraction_non_alphanumeric': 0.13874450219912035, 'fraction_numerical': 0.14964014394242303, 'mean_word_length': 5.264871634314339, 'pattern_counts': {'":': 0, '<': 56, '<?xml version=': 0, '>': 59, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 3, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '418889', 'license': 'MIT', 'n_tokens_mistral': 4834, 'n_tokens_neox': 3855, 'n_words': 948, 'path': 'PS3 (1).ipynb', 'repo_name': 'Jdondy/370-PS3', 'size': '10004'}