<jupyter_start><jupyter_text>Importing Libraries<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import pandas.util.testing as tm

sns.set_style()<jupyter_output>/usr/local/lib/python3.6/dist-packages/statsmodels/tools/_testing.py:19: FutureWarning: pandas.util.testing is deprecated. Use the functions in the public API at pandas.testing instead.
  import pandas.util.testing as tm<jupyter_text>The data used for this analysis is from Autolib electric car sharing company and can be found on the following [link](http://bit.ly/DSCoreAutolibDataset) together with its [description](http://bit.ly/DSCoreAutolibDatasetGlossary) here.  Reading the data from csv file<jupyter_code>df = pd.read_csv("autolib_daily_events_postal_code.csv")<jupyter_output><empty_output><jupyter_text>Checking the data<jupyter_code># number of rows and columns of the data
df.shape
# preview first 5 rows(the top of the dataset)
df.head()
# previewing the bottom of the dataset
df.tail()
# checking the datatype of each column
df.dtypes<jupyter_output><empty_output><jupyter_text>Data Cleaning  **Null Values**<jupyter_code># check for null values
df.isnull().sum()

# there aren't any null values in the data<jupyter_output><empty_output><jupyter_text>**Duplicate entries**<jupyter_code># check for any duplicated entries
df.duplicated().sum()

# none is found<jupyter_output><empty_output><jupyter_text>**Uniformity**<jupyter_code># fixing messy column names and ensuring they are in same format using strip(), lower(), replace() functions
df.columns = df.columns.str.strip().str.lower().str.replace(' ', '_').str.replace('(', '').str.replace(')', '')

# preview column names to see changes made
print(df.columns)<jupyter_output>Index(['postal_code', 'date', 'n_daily_data_points', 'dayofweek', 'day_type',
       'bluecars_taken_sum', 'bluecars_returned_sum', 'utilib_taken_sum',
       'utilib_returned_sum', 'utilib_14_taken_sum', 'utilib_14_returned_sum',
       'slots_freed_sum', 'slots_taken_sum'],
      dtype='object')<jupyter_text>**Ouliers**<jupyter_code># check for outliers

# set the size of the figure to be plotted
plt.figure(figsize=(20,10))
# plot a box and whisker plot to look at present outliers
df.boxplot()

plt.xticks(rotation=45)

# print the size of dataset with outliers present in the data 
print(f"Size of dataset with outliers: {df.shape}")

# # dealing with outliers

Q1 = df.quantile(0.25)
Q3 = df.quantile(0.75)
IQR = Q3 - Q1
df = df[~((df < (Q1 - 1.5 * IQR)) | (df > (Q3 + 1.5 * IQR))).any(axis=1)]

print("...................")
# print size of the data after removal of outliers
print(f"Size of dataset after removal of outliers: {df.shape}")<jupyter_output>Size of dataset with outliers: (16085, 13)
...................
Size of dataset after removal of outliers: (9783, 13)<jupyter_text>**Anomalies**<jupyter_code># checking for anomalies

# find quartile ranges for the column sum of bluecars_taken
q1_bluecars_taken = df['bluecars_taken_sum'].quantile(.25)
q3_bluecars_taken = df['bluecars_taken_sum'].quantile(.75)

# calculating inter-quartile range using values from above
iqr_bluecars_taken = q3_bluecars_taken - q1_bluecars_taken

# another way is to find quantiles using the percentiles from the numpy library
q1_b_taken, q3_b_taken = np.percentile(df['bluecars_taken_sum'], [25, 75]) 
# IQR
iqr_b_taken = q3_b_taken - q1_b_taken

# compare if the two values are similar
print(iqr_bluecars_taken, iqr_b_taken)<jupyter_output>50.0 50.0<jupyter_text>Data-type conversion<jupyter_code># change datatypes of the date column

df['date'] = pd.to_datetime(df['date'])

# check the changes made on the datatype
df.dtypes<jupyter_output><empty_output><jupyter_text>**Exporting the clean dataset as a csv file**<jupyter_code># export the cleaned data onto a new csv file
df.to_csv("autolib.csv")

# read the new csv file using pandas library
df1 = pd.read_csv("autolib.csv")

# preview first 5 rows
print(df1.head())

# remove the first column-not important
autolib = df1.drop(df1.columns[0], axis=1)
autolib.head()<jupyter_output>Unnamed: 0  postal_code  ... slots_freed_sum  slots_taken_sum
0        3899        91330  ...               4                2
1        3900        91330  ...               0                0
2        3901        91330  ...               1                1
3        3903        91330  ...               1                1
4        3905        91330  ...               4                2

[5 rows x 14 columns]<jupyter_text>EXPLORATORY DATA ANALYSIS  Univariate analysis<jupyter_code># analysis on postal code

# plot a histogram plot for the postal code
autolib['postal_code'].hist()

# formatting and labelling axes
plt.title('Distribution of the postal code')
plt.xlabel('Postal Code')
plt.ylabel('Frequency/count')

# analysis on bluecars

# box plot of the distribution of the sum of bluecars returned
sns.boxplot(autolib['bluecars_returned_sum'], showmeans=True)

print(f"The mean is: {autolib['bluecars_returned_sum'].mean()}")
print(f"The median is: {autolib['bluecars_returned_sum'].median()}")
# boxplot of sum of bluecars taken
sns.boxplot(autolib['bluecars_taken_sum'], showmeans=True)

print(f"The mean is: {autolib['bluecars_taken_sum'].mean()}")
print(f"The median is: {autolib['bluecars_taken_sum'].median()}")
# boxplot of sum of utilib14 cars returned
sns.boxplot(autolib['utilib_14_returned_sum'], showmeans=True)

print(f"The mean is: {autolib['utilib_14_returned_sum'].mean()}")
print(f"The median is: {autolib['utilib_14_returned_sum'].median()}")
# boxplot of the sum of utilib14 cars taken
sns.boxplot(autolib['utilib_14_taken_sum'], showmeans=True)

print(f"The mean is: {autolib['utilib_14_taken_sum'].mean()}")
print(f"The median is: {autolib['utilib_14_taken_sum'].median()}")
# boxplot of utilib cares returned
sns.boxplot(autolib['utilib_returned_sum'], showmeans=True)

print(f"The mean is: {autolib['utilib_returned_sum'].mean()}")
print(f"The median is: {autolib['utilib_returned_sum'].median()}")
# boxplot of utilib cars taken
sns.boxplot(autolib['utilib_taken_sum'], showmeans=True)

print(f"The mean is: {autolib['utilib_taken_sum'].mean()}")
print(f"The median is: {autolib['utilib_taken_sum'].median()}")
# from the plot most of the data falls within 0 and 2 cars taken in a day
ax = sns.distplot(autolib['slots_taken_sum'])
# sns.boxplot(autolib['slots_taken_sum'], showmeans=True)

print(f"The mean is: {autolib['slots_taken_sum'].mean()}")
print(f"The median is: {autolib['slots_taken_sum'].median()}")
ax = sns.distplot(autolib['slots_freed_sum'])
# sns.boxplot(autolib['slots_freed_sum'], showmeans=True)

print(f"The mean is: {autolib['slots_freed_sum'].mean()}")
print(f"The median is: {autolib['slots_freed_sum'].median()}")<jupyter_output>The mean is: 0.6974343248492283
The median is: 0.0<jupyter_text>**Findings from univariate analysis** The number of BlueCars taken and returned falls between 10 and 60 per day with means of 49 cars and median 34The number of Utilib14 cars taken and returned falls between 1 and 5 cars per day with a mean approximating 4 cars and median 2The number of Utilib cars taken and returned falls between 0 and 2 cars per day with a mean approximating 1 car and median 1- The most used cars are the BlueCars  Bivariate Analysis<jupyter_code># drop columns not needed in the plot
autolib_data = autolib.drop(['n_daily_data_points', 'date', 'day_type', 'dayofweek', 'postal_code'], axis=1)
# plot pairplot
sns.pairplot(autolib_data)


# plot of correlation between the different fields present in the data
plt.figure(figsize=(20,10))
sns.heatmap(autolib.corr(), annot=True)
autolib.describe()<jupyter_output><empty_output><jupyter_text>**Findings from the bivariate analysis** There is a high correlation between the BlueCars, Utilib14 and Utilib cars both taken and returned with their correlations being above .5  Hypothesis Testing  **Hypothesis testing I** I wish to assess whether there is a statistically significant difference in mean of the number of bluecars taken and returned in city X of area code 92380 during weekdays and using a 5% level of significance.**Hypothesis statements:** - Null: mean of bluecars taken at area 92380 during weekdays is equal to the mean of bluecars returned- Alternate: mean of bluecars taken at area 92380 during weekdays is not equal to the mean of bluecars returned  Sampling Simple random sampling is used in this case.<jupyter_code># first step is to select the specific rows of the postal area code I am investigating
df = autolib[(autolib['postal_code']==92380)&(autolib['day_type']=='weekday')]

# preview the top of the data
df.head()

# extract the specific columns I wish to investigate onto new and different dataframes
bluecars_taken = df['bluecars_taken_sum']
print(bluecars_taken.head())

print("...............")

bluecars_returned = df['bluecars_returned_sum']
print(bluecars_returned.head())
# next step is to choose 2 random sample groups of elements from the data 
# this is done using simple random sampling procedure

sample1 = bluecars_taken.sample(n =40, replace="False")

sample2 = bluecars_returned.sample(n=40, replace="False")<jupyter_output><empty_output><jupyter_text>**Normality test for the two samples chosen**<jupyter_code># import the libraries to plot a Q-Q plot
from statsmodels.graphics.gofplots import qqplot
from matplotlib import pyplot

# plot for the first sample
qqplot(sample1, line='s')
pyplot.show()
# Q-Q plot for sample 2
qqplot(sample2, line='s')
pyplot.show()<jupyter_output><empty_output><jupyter_text>From the two Q-Q plots, the data from both samples appears to be normally distributed about the mean  Statistical Testing<jupyter_code># the next step is then to compute the mean and standard deviation in each sample of Bluecars taken and returned
m1 = sample1.mean() #mean of the first sample
s1 = sample1.std() #standard deviation of first sample

print(f"The mean of bluecars taken (m1) is: {m1}")
print(f"The standard deviation (s1) is: {s1}")
print(f"The variance (var1) is: {sample1.var()}")

print("..............................")

# Bluecars returned
m2 = sample2.mean() #mean of bluecars returned
s2 = sample2.std() #standard deviation of second sample
print(f"The mean of bluecars returned(m2) is: {m2}")
print(f"The standard deviation (s2) is: {s2}")
print(f"The variance (var2) is: {sample2.var()}")

# Before calculating the z-score, first check whether the assumption of equality of 
# population variances is reasonable. This is done by comparing the ratios from the two samples

sample_var_ratio = sample1.var()/sample2.var()
sample_var_ratio

# the sample variance ratio falls between 0.5 and 2.This means that the outcome is equally variable 
# in each of the comparison populations<jupyter_output><empty_output><jupyter_text>**Z-score** This is a two-tailed test, using a Z statistic(since sample size > 30) and a 5% level of significance. The critical region defined for a two-tailed z-test at 0.05 significance level is 1.960. Therefore, Reject the null hypothesis if Z  1.960.<jupyter_code># a z-test of two independent samples is used since conditions are satisfied
alpha = 0.05 #significance level of 5% will be used
n1 = sample1.count() 
n2 = sample2.count() 

# the formula of the z-score used is a ratio of the difference between the two means to the ratio of the standard deviation of both populations
# this is defined by the formula: z = (m1-m2)/(sp*(math.sqrt((1/n1+1/n2))))
# where (m1-m2) is the difference of the two sample means
# sp is the pooled estimate of the common standard deviation and is a number between the standard deviation of the two samples
# s1 and s2 are the standard deviations of sample 1 and sample 2 respectively while n1 and n2 are the sample sizes

# calculate for the pooled estimate of the common stdev
import math
sp = math.sqrt(((n1-1)*s1**2+(n2-1)*s2**2)/(n1+n2-2)) 
print(f"The pooled estimate of the commom stdev is: {round(sp,3)}, which is a value between {round(s1, 3)} and {round(s2,3)}")

# calculate the z-score
z = (m1-m2)/(sp*(math.sqrt((1/n1+1/n2))))
print(f"The z-score statistic is: {z}")<jupyter_output>The pooled estimate of the commom stdev is: 5.338, which is a value between 5.668 and 4.987
The z-score statistic is: -0.816824832768929<jupyter_text>Since z = -0.816824832768929, it is greater than -1.960.Therefore, we fail to reject the null hypothesisThere is not enough statistical significance evidence at α=0.05 to show that there is a difference between the mean number of bluecars taken and returned during weekdays.  **P-value**<jupyter_code># pvalue

import statistics
from scipy.stats import norm

p = 1 -norm.cdf(z)
print(f"p-value is: {p}")
alpha = 0.05
if p < alpha:
  print("Reject null hypothesis and accept that mean of bluecars taken at area 92380 is not equal to the mean of bluecars returned-test is statistically significant")
elif  p > alpha:
  print("Fail to reject null hypothesis")<jupyter_output>Fail to reject null hypothesis<jupyter_text>Here again we find that there is not enough statistical evidence to show that there is no difference between mean of the number of bluecars taken and returned in that particular city.There is not enough significant evidence to show that the alternate hypothesis (means are not equal) is true. We do not conclude that the null hypothesis (means are equal) is true.We could also be committing a Type II error by failing to reject the null hypothesis when it is actually false. Notice that there is a very small difference in the sample means (22.05-21.075 = 0.98 which is equal to 1 car) which indicates a slight and negligible difference  **Hypothesis tesing II** I also wish to assess whether there is a statistically significant difference in mean of the number of bluecars taken between two cities X and Y of area codes 92380 and 92390 respectively over the weekend using a 5% level of significance.**Hypothesis statements:** - Null: mean of bluecars taken at area 92380 during weekends is equal to the mean of bluecars taken at area 92390- Alternate: mean of bluecars taken at area 92380 during weekends is not equal to the mean of bluecars taken at area 92390I will also be finding out if it is greater in area X than in area Y?   Sampling<jupyter_code># simple random sampling is also used for this test
# first step is to select the specific rows of the postal area codes I am investigating
df2 = autolib[(autolib['postal_code']==92380)&(autolib['day_type']=='weekend')]
df3 = autolib[(autolib['postal_code']==92390)&(autolib['day_type']=='weekend')]

# preview the top of the data
print(df2.head())
print(df3.head())
# extract the specific columns I wish to investigate onto new and different dataframes
bluecars_taken_df2 = df2['bluecars_taken_sum']
print(bluecars_taken_df2.head())

print("...............")

bluecars_taken_df3 = df3['bluecars_taken_sum']
print(bluecars_taken_df3.head())
# choose 2 random sample groups of elements from the data 

sample_df2 = bluecars_taken_df2.sample(n =15, replace="False")

sample_df3 = bluecars_taken_df3.sample(n=15, replace="False")<jupyter_output><empty_output><jupyter_text>Normality test<jupyter_code># import the libraries to plot a Q-Q plot
from statsmodels.graphics.gofplots import qqplot
from matplotlib import pyplot

# plot for the first sample
qqplot(sample_df2, line='s')
pyplot.show()
# plot for the second sample
qqplot(sample_df3, line='s')
pyplot.show()<jupyter_output><empty_output><jupyter_text>The two samples are distributed normally about the mean  Statistical Testing<jupyter_code># compute the mean and standard deviation in each sample of Bluecars taken in each area code
# code 92380
m1_df2 = sample_df2.mean() #mean of the first sample
s1_df2 = sample_df2.std() #standard deviation of first sample

print(f"The mean of bluecars taken (m1) at code 92380 is: {m1_df2}")
print(f"The standard deviation (s1) is: {s1_df2}")
print(f"The variance (var1) is: {sample_df2.var()}")

print("..............................")

# Bluecars taken -92390
m2_df3 = sample_df3.mean() #mean of bluecars taken
s2_df3 = sample_df3.std() #standard deviation of second sample
print(f"The mean of bluecars taken(m2) at 92390 is: {m2_df3}")
print(f"The standard deviation (s2) is: {s2_df3}")
print(f"The variance (var2) is: {sample_df3.var()}")<jupyter_output>The mean of bluecars taken (m1) at code 92380 is: 31.0
The standard deviation (s1) is: 7.662524761087935
The variance (var1) is: 58.714285714285715
..............................
The mean of bluecars taken(m2) at 92390 is: 15.6
The standard deviation (s2) is: 3.699420804087349
The variance (var2) is: 13.685714285714287<jupyter_text>**T-test** This is a two-tailed test, using a t statistic(since sample size < 30) and a 5% level of significance. The critical region defined for a two-tailed t-test at 0.05 with degrees of freedom=28(n1+n2-2)is 2.048 Therefore, Reject the null hypothesis if t  2.048.<jupyter_code># a t-test of two independent samples is used since conditions are satisfied
alpha = 0.05 #significance level of 5% will be used
n1_df2 = sample_df2.count() 
n2_df3 = sample_df3.count() 

# the formula of the t-score used is a ratio of the difference between the two means to the ratio of the standard deviation of both populations
# this is defined by the formula: t = (m1-m2)/(sp*(math.sqrt((1/n1+1/n2)))) where df = n1+n2-2
# where (m1-m2) is the difference of the two sample means
# sp is the pooled estimate of the common standard deviation and is a number between the standard deviation of the two samples
# s1 and s2 are the standard deviations of sample 1 and sample 2 respectively while n1 and n2 are the sample sizes

# calculate for the pooled estimate of the common stdev
import math
sp1 = math.sqrt(((n1_df2-1)*s1_df2**2+(n2_df3-1)*s2_df3**2)/(n1_df2+n2_df3-2)) 
print(f"The pooled estimate of the commom stdev is: {round(sp1,3)}, which is a value between {round(s1_df2, 3)} and {round(s2_df3,3)}")

# calculate the t-statistic
t = (m1_df2-m2_df3)/(sp1*(math.sqrt((1/n1_df2+1/n2_df3))))
print(f"The t-statistic is: {t}")<jupyter_output>The pooled estimate of the commom stdev is: 6.017, which is a value between 7.663 and 3.699
The t-statistic is: 7.0096618403473725<jupyter_text>Since t = 7.0096618403473725, it is greater than 2.048.Therefore, we reject the null hypothesis and accept the alternate hypothesis that mean of bluecars taken at area 92380 during weekends is not equal to the mean of bluecars taken at area 92390. There is enough statistical significance evidence at α=0.05 to show that there is a difference between the mean number of bluecars taken in area 92380 and 92390.  **P-value**<jupyter_code>p = 1 -norm.cdf(t)
print(f"p-value is: {p}")
alpha = 0.05
if p < alpha:
  print("Reject null hypothesis and accept the alternate hypothesis")
elif  p > alpha:
  print("Fail to reject null hypothesis")<jupyter_output>Reject null hypothesis and accept the alternate hypothesis<jupyter_text>Here again we find that there is a statistically significant difference in mean number of bluecars taken between city X and city Y of area codes 92380 and 92390 repectively. Notice that there is a very small difference in the sample means (31.0-15.6 = 15.4 = 15 cars), but this difference is beyond what would be expected by chance **Is the mean of bluecars taken greater in area X than in area Y?**<jupyter_code>difference = m1_df2 - m2_df3
print(difference)
print(f"The mean of bluecars taken over the weekends in area X is greater than in area Y by: {difference}")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10456079353717149, 'fraction_numerical': 0.0394212087125473, 'mean_word_length': 4.975863122517568, 'pattern_counts': {'":': 0, '<': 86, '<?xml version=': 0, '>': 86, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '23', 'dir': 'jupyter-structured-clean-dedup', 'id': '332464', 'license': 'MIT', 'n_tokens_mistral': 6833, 'n_tokens_neox': 6226, 'n_words': 2282, 'path': "Statistical_EDA_&_Hypothesis_testing_on_Autolib's_cars.ipynb", 'repo_name': 'Jenn-mawia/Statistical-Inferential-Data-Analysis-and-Hypothesis-testing-on-Autolib-electric-car-sharing', 'size': '19558'}