<jupyter_start><jupyter_text>Hypothesis Testing **Background of the study**  After a quick look at the dataset that contains data on the usage of Autolib's cars between 1st January 2018 and 19th June 2018, Autolib's manager claims that the average number of Bluecars taken from station with postal code 75015 is the same as the average number of Bluecars taken from station with postal code 75016 on the weekdays. This study seeks to establish whether this claim is true or not.**Hypothesis**  Null hypothesis: There is no difference between the average number of Bluecars taken from station with postal code 75015 and the average number of Bluecars taken from station with postal code 75016 on the weekdays   Alternative hypothesis: There is a difference between the average number of Bluecars taken from station with postal code 75015 and the average number of Bluecars taken from station with postal code 75016 on the weekdays*Keys*  $H_{0}$ = Null hypothesis  $H_{1}$ = Alternative hypothesis  $mu_{1}$ = Average number of Bluecars taken from postal code 75015 on the weekdays  $mu_{2}$ = Average number of Bluecars taken from postal code 75016 on the weekends  $H_{0}$: $mu_{1}$ = $mu_{2}$  $H_{1}$: $mu_{1}$ ≠ $mu_{2}$  **Process**  We will load the required dataset for the study, clean it, obtain sample units from the two postal codes, perform univariate analysis, then conduct a hypothesis test so as to determine whether we should reject or fail to reject the null hypothesis.   Importing Our Libraries<jupyter_code># importing pandas
import pandas as pd

# importing numpy
import numpy as np

# importing matplotlib.pyplot
import matplotlib.pyplot as plt

# importing scipy.stats
import scipy.stats as stats

# importing seaborn
import seaborn as sns<jupyter_output><empty_output><jupyter_text>Loading Our Dataset<jupyter_code># Load csv dataset with the required data from given url 'http://bit.ly/DSCoreAutolibDataset'
autolib = pd.read_csv('http://bit.ly/DSCoreAutolibDataset')<jupyter_output><empty_output><jupyter_text>Reading the Dataset<jupyter_code># Preview the top 5 entries in our dataset
autolib.head()
# Preview the bottom 5 entries in our dataset
autolib.tail()
# Check the general information about our dataset
# The information includes the number of entries and columns, name of each column, number of non-null values in every column, the data type of each column, and the distribution of each data type
autolib.info()
# Change "Postal code" and "date" columns' data types
autolib['date'] = pd.to_datetime(autolib['date'])

autolib['Postal code'] = autolib['Postal code'].astype(str)

# Verify data type change
autolib.dtypes<jupyter_output><empty_output><jupyter_text>Cleaning the Data  i. Validity<jupyter_code># Select the relevant columns for the analysis
# Create dataframe with "Postal code", "BlueCars_taken_sum", and "date" columns (date column is retained to identify duplicates in data)
# Select postal codes "75015" and "75016" in "Postal code" column and specify "weekday" as the day type
postal_data = autolib[['Postal code', 'date','BlueCars_taken_sum']][((autolib['Postal code']=='75015')|(autolib['Postal code']=='75016'))&(autolib['day_type']=='weekday')].reset_index()

# Drop 'index' column
cols = [0] #specify the index of the column to be dropped
postal_data.drop(postal_data.columns[cols],axis=1,inplace=True) #drop the column by the index

# Rename BlueCars_taken_sum to bluecars_taken
postal_data.rename(columns = {'BlueCars_taken_sum':'bluecars_taken'}, inplace = True)

# Display the some records of the dataframe
postal_data<jupyter_output><empty_output><jupyter_text>ii. Uniformity<jupyter_code>#Standardize the column names by removing white spaces using strip()function, lowering the cases using lower() function, and replacing middle spaces with underscore 
postal_data.columns = postal_data.columns.str.strip().str.lower().str.replace(' ', '_')

#display the standardized column names  
postal_data.columns<jupyter_output><empty_output><jupyter_text>iii. Completeness<jupyter_code># Check for duplicates in the dataset
postal_data.duplicated().sum()<jupyter_output><empty_output><jupyter_text>There are no duplicates to be dropped  iv. Consistency<jupyter_code># Check for null values
postal_data.isnull().sum().sum()<jupyter_output><empty_output><jupyter_text>There are no null values to be dropped or replaced  v. Anomalies<jupyter_code># Upon the investigation of the main dataset it was discovered that records on dates '2018-06-19' and '2018-02-24' were significantly lower than the usual data entries due to unidentified reasons. 
# As much as they show the accurate data on the car usage, on the specific days, they do not represent the usual car usage in the different postal codes.
# If included in a random sample, they are likely to change the results or assumptions of the analysis; therefore, they will be dropped.

# Drop rows with '2018-06-19' or '2018-02-24' in the 'date' column
postal_data = postal_data.drop(postal_data[(postal_data.date=='2018-06-19')|(postal_data.date=='2018-02-24')].index)<jupyter_output><empty_output><jupyter_text>v. Outliers Instead of finding outliers within the general dataframe, we will divide the dataframe per postal code and find outliers within the different subsets<jupyter_code># Create a dataset with 75015's postal code data
post1 =  postal_data[['postal_code', 'bluecars_taken']][(postal_data['postal_code']=='75015')]

# Plot a boxplot to identify outliers
post1.boxplot(column=['bluecars_taken'], grid=False)

# Giving the box plot a title
plt.title('Distribution of Bluecars Taken on Weekdays From Postal Code 75015', fontsize=15, y=1.015)
# labelling the y axis
plt.ylabel('Number or Bluecars Taken', fontsize=14, labelpad=15)
# displaying the histogram
plt.show()<jupyter_output><empty_output><jupyter_text>From the boxplot, there are no outliers in postal code 75015 data.<jupyter_code># Create a dataframe with 75016's postal code data
post2 =  postal_data[['postal_code', 'bluecars_taken']][(postal_data['postal_code']=='75016')]

# Plot a boxplot to identify outliers
post2.boxplot(column=['bluecars_taken'], grid=False)

# Giving the box plot a title
plt.title('Distribution of Bluecars Taken on Weekdays From Postal Code 75016', fontsize=15, y=1.015)
# labelling the y axis
plt.ylabel('Number or Bluecars Taken', fontsize=14, labelpad=15)
# displaying the histogram
plt.show()<jupyter_output><empty_output><jupyter_text>There is no justifiable reason to remove the outliers as they accurately reflect the usage of cars in the different postal codes; therefore, we will keep the outliers.<jupyter_code># View number of records of the cleaned dataset
postal_data.shape<jupyter_output><empty_output><jupyter_text>Exporting the Cleaned Dataset<jupyter_code># Exporting the cleaned dataset as a csv file
postal_data.to_csv('postal_data_75015_76016.csv')<jupyter_output><empty_output><jupyter_text>Sampling<jupyter_code># Create a stratified random sample by grouping the dataset by postal code, then extracting sample units from each group
# The sample size is calculated using the Yamane formula with 0.05 as the margin of error, and 95% as the confidence level 
postal_sample = postal_data.groupby('postal_code', group_keys=False).apply(lambda x : x.sample(replace='False', frac=((222/(1+222*(0.05**2)))/222)))

# Print the number of sample units from each postal code
print(postal_sample['postal_code'].value_counts())

# Display some of the sample's data 
postal_sample
# Exporting the sample dataset as a csv file
postal_sample.to_csv('postal_sample_75015_75016.csv')<jupyter_output><empty_output><jupyter_text>Univariate Analysis For us to conduct univariate analysis that is meaningful to our objective, we are required to split the sample data by the postal code in order to analyse each group independently. The univariate analysis will make us understand our sample data better.<jupyter_code># Creating a subset with 75015's postal code sample data
post1_sample =  postal_sample[['postal_code', 'bluecars_taken']][(postal_sample['postal_code']=='75015')]

# Creating a subset with 75016's postal code sample data
post2_sample =  postal_sample[['postal_code', 'bluecars_taken']][(postal_sample['postal_code']=='75016')]<jupyter_output><empty_output><jupyter_text>i. Measures of central tendancy  a. Postal code 75015  Mean<jupyter_code>post1_sample['bluecars_taken'].mean()<jupyter_output><empty_output><jupyter_text>Median<jupyter_code>post1_sample['bluecars_taken'].median()<jupyter_output><empty_output><jupyter_text>Mode<jupyter_code>post1_sample['bluecars_taken'].mode()<jupyter_output><empty_output><jupyter_text>b. Postal code 75016 Mean<jupyter_code>post2_sample['bluecars_taken'].mean()<jupyter_output><empty_output><jupyter_text>Median<jupyter_code>post2_sample['bluecars_taken'].median()<jupyter_output><empty_output><jupyter_text>Mode<jupyter_code>post2_sample['bluecars_taken'].mode()<jupyter_output><empty_output><jupyter_text>ii. Measures of dispersion  a. Postal code 75015 Variance<jupyter_code>post1_sample['bluecars_taken'].var()<jupyter_output><empty_output><jupyter_text>Standard deviation<jupyter_code>post1_sample['bluecars_taken'].std()<jupyter_output><empty_output><jupyter_text>Range<jupyter_code>max_taken = post1_sample['bluecars_taken'].max()
min_taken = post1_sample['bluecars_taken'].min()

max_taken - min_taken<jupyter_output><empty_output><jupyter_text>Quartiles<jupyter_code># Calculating the quantiles 
post1_sample['bluecars_taken'].quantile([0.25,0.5,0.75])<jupyter_output><empty_output><jupyter_text>Interquartile range<jupyter_code># Calculating the interquatile range
Q3 = post1_sample['bluecars_taken'].quantile(0.75)
Q1 = post1_sample['bluecars_taken'].quantile(0.25)

Q3-Q1<jupyter_output><empty_output><jupyter_text>Skewness<jupyter_code>post1_sample['bluecars_taken'].skew()<jupyter_output><empty_output><jupyter_text>Kurtosis<jupyter_code>post1_sample['bluecars_taken'].kurt()<jupyter_output><empty_output><jupyter_text>b. Postal code 75016 Variance<jupyter_code>post2_sample['bluecars_taken'].var()<jupyter_output><empty_output><jupyter_text>Standard deviation<jupyter_code>post2_sample['bluecars_taken'].std()<jupyter_output><empty_output><jupyter_text>Range<jupyter_code>max_taken = post2_sample['bluecars_taken'].max()
min_taken = post2_sample['bluecars_taken'].min()

max_taken - min_taken<jupyter_output><empty_output><jupyter_text>Quartiles<jupyter_code># Calculating the quantiles 
post2_sample['bluecars_taken'].quantile([0.25,0.5,0.75])<jupyter_output><empty_output><jupyter_text>Interquartile range<jupyter_code># Calculating the interquatile range
Q3 = post2_sample['bluecars_taken'].quantile(0.75)
Q1 = post2_sample['bluecars_taken'].quantile(0.25)

Q3-Q1<jupyter_output><empty_output><jupyter_text>Skewness<jupyter_code>post2_sample['bluecars_taken'].skew()<jupyter_output><empty_output><jupyter_text>Kurtosis<jupyter_code>post2_sample['bluecars_taken'].kurt()<jupyter_output><empty_output><jupyter_text>iii. Visualizations  a. Postal code 75015<jupyter_code># Specifying the size of the graph
fig, ax = plt.subplots(figsize=(8,5))

# Ploting density plot and histogram of the number of cars taken from station with postal code 75015
sns.histplot(post1_sample['bluecars_taken'], kde=True, stat="density",color='darkblue', bins=10)

# Label the graph
plt.title('Density Plot and Histogram for the Number of Bluecars Taken from Postal Code 75015 Station', fontsize=14, y=1.015)

plt.show()<jupyter_output><empty_output><jupyter_text>b. Postal code 75016<jupyter_code># Specifying the size of the graph
fig, ax = plt.subplots(figsize=(8,5))

# Ploting density plot and histogram of the number of cars taken from station with postal code 75016
sns.histplot(post2_sample['bluecars_taken'], kde=True, stat="density",color='darkblue', bins=10)

# Label the graph
plt.title('Density Plot and Histogram for the Number of Bluecars Taken from Postal Code 75016 Station', fontsize=14, y=1.015)

plt.show()<jupyter_output><empty_output><jupyter_text>Hypothesis Testing Since we do not know the population's standard deviation, the most appropriate hypothesis test for this study will be a t-test but before the t-test is conducted, the assumptions of the test need to be checked to see if the t-test results can be trusted.    The following are assumptions of a t-test1. The samples are independently and randomly drawn2. The distribution of the residuals between the two groups follow the normal distribution3. The variances between the two groups are equal, incase of unequal variances we would conduct a Welch’s t-test instead of a Student t-test  If any of these assumptions are violated then another test should be used.    Checking the equality of the variances<jupyter_code>## Check the equality of the variances between the two groups using Levene's test

stat,p = stats.levene(post1_sample['bluecars_taken'], post2_sample['bluecars_taken'], center='mean') # we use center = mean because our samples are fairly/moderately skewed

print('Statistic=%.4f, pvalue=%.4f' % (stat, p))

# Interpretation of the Levene's test results

alpha = 0.05
if p <= alpha:
	print('The two variances are significantly different')
else:
	print('The two variances are approximately equal')<jupyter_output>Statistic=5.5953, pvalue=0.0194
The two variances are significantly different<jupyter_text>Checking the distribution of the residuals between the two groups<jupyter_code># Check the distribution of the residuals between the two groups using the Shapiro-Wilk test
from scipy.stats import shapiro

diff = post1_sample['bluecars_taken'] - post2_sample['bluecars_taken']

stat, p = shapiro(diff)

print('Statistic=%.4f, pvalue=%.4f' % (stat, p))

# Interpretation of the Shapiro-Wilk test results
alpha = 0.05
if p <= alpha:
	print('Sample does not look Gaussian')
else:
	print('Sample looks Gaussian')<jupyter_output>Statistic=nan, pvalue=1.0000
Sample looks Gaussian<jupyter_text>Since the samples are independently and randomly drawn and the distribution of the residuals between the two groups follow the normal distribution, but the variances between the two groups are significantly different, we will conduct the Welch's t-test.  Conducting the t-test<jupyter_code># Conduct the independent Welch's t-test using scipy.stats, use the stats.ttest_ind() method
stat,p = stats.ttest_ind(post1_sample['bluecars_taken'], post2_sample['bluecars_taken'], equal_var=False)

print('Statistic=%.14f, pvalue=%.14f' % (stat, p))

# Interpretation of the t-test results with 0.05 as the significance level
alpha = 0.025 #since the test is two tailed
if p <= alpha:
	print('We reject the null hypothesis')
else:
	print('We fail to reject the null hypothesis')<jupyter_output>Statistic=4.80747103604465, pvalue=0.00000418073786
We reject the null hypothesis
{'fraction_non_alphanumeric': 0.0955048596112311, 'fraction_numerical': 0.031047516198704102, 'mean_word_length': 7.22253052164262, 'pattern_counts': {'":': 0, '<': 157, '<?xml version=': 0, '>': 154, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '39', 'dir': 'jupyter-structured-clean-dedup', 'id': '538561', 'license': 'MIT', 'n_tokens_mistral': 5152, 'n_tokens_neox': 4599, 'n_words': 1440, 'path': 'Moringa_Data_science_Core_W4_Independen_Project_2021_08_Wambui_Kahuthu.ipynb', 'repo_name': 'WambuiKahuthu/MORINGA_CORE_WEEK4_IP_HYPOTHESIS_TESTING', 'size': '14816'}