<jupyter_start><jupyter_text>Collection of Simple Random Sample from Population of Websites  Simple Random SampleBelow is a program that creates a simple random sample of websites according to Alexa Analytics' Top One Million sites CSV. The function `create_sample` creates a random integer between 0 and 999,999. That number acts as the index for the site we will get from the population CSV (e.g. The site at index 0 is google.com). Once we have selected a site for our sample, we need to resolve its IP address (note: we will be using only IPv4 addresses as Alexa only offers those in their dataset and all websites should handle IPv4 and IPv6). If we are unable to resolve the address than we will randomly select another site to replace it.When sampling is completed, the sample dataset (domains and IPv4 addresses) is exported to a CSV file.<jupyter_code>from random import randint
import pandas as pd
import socket

# List of top one million sites according to Alexa Analytics/Website Ranking
# https://s3.amazonaws.com/alexa-static/top-1m.csv.zip
top_sites = pd.read_csv('top-1m.csv', header=None)[1]

# n is sample size
n=1000

# Dictionary used for stored sample data
sample = {
    'Website Domain' : [],
    'IPv4 Address' : []
}

def create_sample(n): 
    i = 0
    while i < n:
        i += 1
        # Get random number between 0 and 999,999
        random_index = randint(0, len(top_sites) - 1)

        # If the site has not already been selected, add it to our data set
        if not top_sites[random_index] in sample['Website Domain']:
            try:
                # print("\033[0mGetting IPv4 Address for %s..." % top_sites[random_index])
                ipv4 = socket.gethostbyname(top_sites[random_index])
            # If we can't resolve the IP from the host name, replace it with a different host name
            except:
                # print("\033[1mFailed. Selecting new site for sample.")
                i -= 1
                continue
            sample['Website Domain'].append(top_sites[random_index])
            sample['IPv4 Address'].append(ipv4)

create_sample(n)

# Save sample to a CSV file
dataset = pd.DataFrame.from_dict(sample)
dataset.to_csv('website_sample.csv')

dataset<jupyter_output><empty_output><jupyter_text>Use Dataset Instead of Creating New SampleEssentially importing `website_sample.csv` for our dataset so we don't have to create a new sample.<jupyter_code>n = 1000
dataset = pd.DataFrame.from_csv('website_sample.csv')
dataset<jupyter_output>/home/ari/.anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:2: FutureWarning: from_csv is deprecated. Please use read_csv(...) instead. Note that some of the default arguments are different, so please refer to the documentation for from_csv when changing your function calls<jupyter_text>Determining Proportion of Websites Running AWS  Proportion of IPv4 addresses owned by AWSThe program takes a list of all IPv4 addresses owned by AWS and compares them to the list of addresses in our sample. AWS does not give a list of IPv4 address but instead gives a subnet of their addresses, this means they've purchased addresses in bulk so they're grouped together. In order to properly compare an IPv4 address to a subnet, python offers a library called `ipaddress` that breaks up subnets and ip addresses into a data format that can easily be compared.If an address appears in Amazon's IPv4 range (their owned addresses) than the domain associated with the IP address is appended to a list. The list of websites is then exported as a CSV file.<jupyter_code>import json, requests, ipaddress 

# List of IP Ranges (IPv4 and IPv6) owned by Amazon and used for AWS
# https://ip-ranges.amazonaws.com/ip-ranges.json
aws_ip_ranges = json.loads(requests.get('https://ip-ranges.amazonaws.com/ip-ranges.json').text)

# Determine if given IP address (ip_input) shows uo in AWS IPv4 Range
def check_aws(ip_input):
    # Compare given IP to all AWS IP addresses within AWS IPv4 Subnetwork
    for i in range(len(aws_ip_ranges['prefixes'])):
        # Parse IPv4 address for comparison
        site_ip = ipaddress.ip_address(ip_input)
        
        # Parse AWS IPv4 Subnet
        aws_subnet = ipaddress.ip_network(aws_ip_ranges['prefixes'][i]['ip_prefix'])
        
        # If IP is within the AWS IPv4 Range, the website is run on AWS
        if site_ip in aws_subnet:
            return True
    # If the website is not within the range, the 
    # website operates independnetly of AWS    
    return False

# List of websites using AWS
websites_using_aws = []

def get_aws_domains():

    # Check every IP within our sample against AWS IPv4 Range
    for i in range(len(dataset)):
        if check_aws(dataset['IPv4 Address'][i]):
            websites_using_aws.append(dataset['Website Domain'][i])

get_aws_domains()
            
# Save dataset of AWS websites to a CSV file
aws_df = pd.DataFrame({'AWS Websites':websites_using_aws})
aws_df.to_csv('websites_using_aws.csv')

aws_df<jupyter_output><empty_output><jupyter_text>1-Proportion Z-Test for Proportion of AWS to non-AWS Websites  Testing in PythonThis is a procedure for completing a one-proportion z-test given the sample dataset and proportion of websites using AWS. First we declare `p` as the claimed market share percentage (The Forbes article claimed AWS has a 31% market share). We calculate `q` and proceed with our success failure condition. The `assert` lines basically test s/f - if np or qp is < 10 than the program stops and an exception (error) is raised. Because the index values when sampling were random, than we can assume our sample is random.Next we calculate our z-score value with a one-proportion z-test:$$\begin{align}z = \frac{\hat{p} - p}{\sigma} && \sigma = \sqrt{\frac{pq}{n}}\end{align}$$The code for calculating z-score is `z = ((len(websites_using_aws)/n) - p)/sd`. For standard deviation, it's `sd = math.sqrt((p*q)/2)`.Once we calculate our z-score, finally we can get our p-value. The library SciPy offers statistics which will allow us to calculate p-value similar to how a calculator would. Once complete, we can do hypothesis testing using a significance level of 5%.<jupyter_code>import math
import scipy.stats as st

# Creating initial values from datatset/claim
claimed_marketshare = 0.31

p = claimed_marketshare
q = 1-claimed_marketshare

# Success/Failure Condition, exception raised if np or nq is less than 10
assert n*p >= 10, True
assert n*q >= 10, True

# Calculate Z-Score & P-Value
sd = math.sqrt((p*q)/n)
z = ((len(websites_using_aws)/n) - p)/sd

p_value = st.norm.cdf(z)
  
print('P: %f\tQ: %f\nNP: %f\tNQ: %f\n\nP-Hat: %f\n\nZ-Score: %f\nP-Value: %f\n' 
      % (p, q, n*p, n*q, (len(websites_using_aws)/n), z, p_value))

# Hypothesis Testing
significants_level = 0.05

if p_value <= (significants_level): print('\033[1mReject H0')
else: print('\033[1mFail to reject H0')<jupyter_output>P: 0.310000	Q: 0.690000
NP: 310.000000	NQ: 690.000000

P-Hat: 0.061000

Z-Score: -17.025268
P-Value: 0.000000

[1mReject H0<jupyter_text>InterpretationWe've rejected the null-hypothesis as our P-Value is approximately zero. The claim made within the Forbes article is invalid according to this observational study as we can support the alternate hypothesis that Amazon does not have a 31% market share in cloud computing/hosting.  Confidence Interval for 1-Proportion Sample  Confidence Interval Based on Z-TestThis is a standard method to produce a confidence interval given the z-score, sample size, claimed population proportions and sample proportions produced by the one-proportion z-test.We use the following equations to calculate the standard deviation and standard error for our confidence interval:$$\begin{align}\sigma_p = \sqrt{\frac{pq}{n}} && SE_p = \sqrt{\frac{\hat{p}\hat{q}}{n}}\end{align}$$Both $\hat{p}$ and $\hat{q}$ are simply $p$ and $q$ for the sample proportion. Using the z-score we calculated during our hypothesis test as our critical value, we can multipy it by our standard error to produce our margin of error, like so:$$ME = Z_c\sqrt{\frac{\hat{p}\hat{q}}{n}}$$<jupyter_code># Sample p (statistic) and q values
aws_p = len(websites_using_aws)/n
aws_q = 1 - aws_p

# Standard error for the sample proportion
se = math.sqrt((aws_p*aws_q)/n)

# Margin of error
me = z*se

print('Interval: (%f, %f)' % (aws_p + me, aws_p - me))<jupyter_output>Interval: (-0.067852, 0.189852)
{'fraction_non_alphanumeric': 0.0872563005230623, 'fraction_numerical': 0.020328102710413694, 'mean_word_length': 3.951736315479694, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 20, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '274184', 'license': 'MIT', 'n_tokens_mistral': 2688, 'n_tokens_neox': 2466, 'n_words': 1022, 'path': 'AWS-Market Share Analysis.ipynb', 'repo_name': 'ceiphr/AWS-Market-Share-Analysis', 'size': '8412'}