<jupyter_start><jupyter_text>**AN HYPOTHESIS TESTING OF A PARIS-BASED CAR SHARING SERVICE**  **DEFINING THE QUESTION**  *  ***HO***= Null Hypothesis *  ***H1***= Alternative Hypothesis*  ***m1***= During weekends, the mean number of blue cars taken is equal to the mean number of cars returned.*   ***m2***= During weekends, the mean number of blue cars taken is not equal to the mean number of cars returned.* ***HO***: ***m1=m2***   ***H1***: ***m1!=m2***Our research problem is to figure out how the mean of blue cars taken compares to blue cars being returned over the weekends.   **LOADING OUR LIBRARIES AND DATASETS**<jupyter_code># IMPORT THE NECESSARY LIBRARIES

import numpy as np
import pandas as pd
import seaborn as sns
import scipy.stats as stats
import random
from math import sqrt
from matplotlib import pyplot as plt

# LOAD THE AUTOLIB DATASET

df = pd.read_csv('http://bit.ly/DSCoreAutolibDataset')
df.head()
# LOAD THE GLOSSARY 

glossary = pd.read_excel('/content/columns_explanation.xlsx')
glossary<jupyter_output><empty_output><jupyter_text>**READING, CHECKING AND CLEANING THE DATA**<jupyter_code>df.sample(5)
df.dtypes
df.columns
df.count()
df.shape
# lets check for duplicates and get rid if any
# then return a non duplicated dataset

def duplicates(data):
  out = df.duplicated().any()
  if out == False:
    return out
  else:
    pass

duplicates(df)
# LETS CHECK FOR MISSING DATA 

def missing(data):
  out = df.isnull().any()

  if out == False:
    return out
  else:
    pass
df.isnull().any()
# lets check for accuracy percentage by missing values check

miss = df.isnull().sum().sort_values(ascending=False)

percentage = (df.isnull().sum()/len(df)).sort_values(ascending=False)

# storing it in a dataframe
miss_values = pd.DataFrame({'MISSING': miss, 'Percent': percentage})
#return miss_values
miss_values<jupyter_output><empty_output><jupyter_text>There are no missing values Since our hypothesis is only about blue cars, we shall have to drop the unnecessary columns<jupyter_code># first lets check the columns we have
df.columns
# dropping unnecessary columns
df_new = df.drop(columns=[ 'n_daily_data_points','Utilib_taken_sum',
       'Utilib_returned_sum', 'Utilib_14_taken_sum', 'Utilib_14_returned_sum',
       'Slots_freed_sum', 'Slots_taken_sum'])<jupyter_output><empty_output><jupyter_text>**SAMPLING AND MORE CLEANING** We are only going to be working with the weekends so we shall drop all weekday rows. We shall do this by sampling the weekend day types.Lets name our sample autolib.<jupyter_code>df2= df_new[df_new['day_type'].str.contains("weekend")]
df2.sample(3)
code = np.sort(df2['Postal code'].unique())
code
# we want our data to have only two postal codes for the weekend
import random

#setting up a seed so that all the code can be run and give the same results

random.seed(0)
print ("Choosing 2 random Postal Codes from the dataset",random.sample(list(code), k=2))
PC1 = 92320
PC2 = 94450
autolib = df2.loc[(df2['Postal code'] == PC1) | (df2['Postal code'] == PC2)] 
autolib
autolib.shape
len(autolib)
# lets fix the messiness in the columns

autolib.rename(columns = {'Postal code': 'postal_code', 'dayOfWeek':'day',	'BlueCars_taken_sum':'bluecars_taken','BlueCars_returned_sum':'bluecars_returned'}, inplace=True)

#autolib.columns.str.strip().str.lower().str.replace(' ', '_').str.replace('(', '').str.replace(')', '')
#autolib.columns.str.strip().str.lower().str.replace('', '_').str.replace('(','').str.replace(')','')
autolib.sample(2)
# lets check for unique values in all of our columns
column = autolib.columns.to_list()

for column in column:
    print("\n")
    print("Column Name:", column)
    print(autolib[column].unique())<jupyter_output>Column Name: postal_code
[92320 94450]


Column Name: date
['1/6/2018' '1/7/2018' '1/13/2018' '1/14/2018' '1/20/2018' '1/21/2018'
 '1/27/2018' '1/28/2018' '2/3/2018' '2/4/2018' '2/10/2018' '2/11/2018'
 '2/17/2018' '2/18/2018' '2/24/2018' '3/3/2018' '3/4/2018' '3/10/2018'
 '3/11/2018' '3/17/2018' '3/18/2018' '3/24/2018' '3/25/2018' '3/31/2018'
 '4/1/2018' '4/7/2018' '4/8/2018' '4/14/2018' '4/15/2018' '4/21/2018'
 '4/22/2018' '4/28/2018' '4/29/2018' '5/5/2018' '5/6/2018' '5/20/2018'
 '5/26/2018' '5/27/2018' '6/2/2018' '6/3/2018' '6/9/2018' '6/10/2018'
 '6/16/2018' '6/17/2018']


Column Name: day
[5 6]


Column Name: day_type
['weekend']


Column Name: bluecars_taken
[83 79 82 80 74 88 73 97 71 76 77  4 69 90 86 75 67 54 59 68 64 72 55 57
 44 87 92 16 22 20 29 15 26 34 25 24 38  0 17 19 18 21 10 27 14 28 12 32]


Column Name: bluecars_returned
[ 82  85  93  78  87  95  74 103  72  71  81  91   7  64  86  90  80  83
  60  77  61  69  65  70  94  67  59  54  73  99  75 102  19  18  28  25[...]<jupyter_text>Looks like we have good values so we shall for outliers next.<jupyter_code>autolib.columns

fig, ((ax1, ax2)) = plt.subplots(1,2, figsize=(20, 14))
fig.suptitle('Boxplots')

sns.boxplot(autolib['bluecars_taken'], ax=ax1)
sns.boxplot(autolib['bluecars_returned'], ax=ax2)


plt.show()<jupyter_output>/usr/local/lib/python3.7/dist-packages/seaborn/_decorators.py:43: FutureWarning: Pass the following variable as a keyword arg: x. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  FutureWarning
/usr/local/lib/python3.7/dist-packages/seaborn/_decorators.py:43: FutureWarning: Pass the following variable as a keyword arg: x. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  FutureWarning<jupyter_text>Our sample has no outliers so We shall then check for anomalies<jupyter_code>Q1= autolib.quantile(0.25)
Q3 =autolib.quantile(0.75)
IQR = Q3-Q1
print(Q1)
print('----------------------------------------')
print(Q3)
print('----------------------------------------')
print(IQR)
autolib.isnull().any()<jupyter_output><empty_output><jupyter_text>**ANALYSIS**  **UNIVARIATE**<jupyter_code>whole =autolib[['bluecars_taken','bluecars_returned']]
whole.describe()
MO = autolib.mode()
MO.head(1)
cars = autolib[['bluecars_taken','bluecars_returned']]

QU = cars.quantile([0.25,0.5,0.75])
print(QU)
cars.skew()
cars.kurt()
cars.mean()<jupyter_output><empty_output><jupyter_text>The mean of cars taken and cars returned appears to be almost equal with a minute diiference.<jupyter_code>cars.median()
cars.var()
cars.std()

sns.distplot(autolib['bluecars_returned'])
sns.distplot(autolib['bluecars_taken'])
sns.distplot(autolib['day'])
days = autolib[['bluecars_taken','bluecars_returned','day']]
days

# Check frequency of day of the week
d=autolib['day'].value_counts()
d.plot.bar()
plt.xlabel('Day of the week')
plt.ylabel('Frequency')<jupyter_output><empty_output><jupyter_text>There is no big diffence between the number of cars taken and returned during the weekend.<jupyter_code># Check frequency by the postal codes

d=autolib['postal_code'].value_counts()
d.plot.bar()
plt.xlabel('Postal Code')
plt.ylabel('Frequency')
columnss = ['bluecars_taken', 'bluecars_returned']

for col in columnss:

  plt.hist(autolib[col],bins=10,color="pink")
  plt.title(col)

  plt.show()<jupyter_output><empty_output><jupyter_text>**BIVARIATE**<jupyter_code>autolib.sample(5)
# FINDING THE CORRELATION

corr=cars.corr()
print(corr)

sns.heatmap(corr, xticklabels=corr.columns, yticklabels=corr.columns, annot=True)<jupyter_output>bluecars_taken  bluecars_returned
bluecars_taken           1.000000           0.979978
bluecars_returned        0.979978           1.000000<jupyter_text>There's a high correlation<jupyter_code>p = sns.pairplot(autolib, diag_kind="kde", hue="day",vars=["bluecars_taken", "bluecars_returned"])
plt.show()
p = sns.pairplot(autolib, diag_kind="kde", hue="postal_code",vars=["bluecars_taken", "bluecars_returned"])
plt.show()<jupyter_output><empty_output><jupyter_text>**HYPOTHESIS TESTING**<jupyter_code># lets check for normality of our distribution
# importing the shapiro function
from scipy.stats import shapiro 

# test for normality
stat, r = shapiro(autolib['bluecars_taken'])

print('Statistics=%.3f, p=%.3f' % (stat, r))

# interpreting
alpha = 0.05
if r > alpha:
	print('Sample looks Gaussian')
else:
	print('Sample does not look Gaussian')
 
from statsmodels.graphics.gofplots import qqplot
from matplotlib import pyplot


from scipy.stats import shapiro 

# test for normality
stat, r = shapiro(autolib['bluecars_returned'])

print('Statistics=%.3f, p=%.3f' % (stat, r))

# interpreting
alpha = 0.05
if r > alpha:
	print('Sample looks Gaussian')
else:
	print('Sample does not look Gaussian')
 
from statsmodels.graphics.gofplots import qqplot
from matplotlib import pyplot
from statsmodels.graphics.gofplots import qqplot
from matplotlib import pyplot
# q-q plot
qqplot(autolib['bluecars_returned'], line='s')
pyplot.show()
from statsmodels.graphics.gofplots import qqplot
from matplotlib import pyplot
# q-q plot
qqplot(autolib['bluecars_taken'], line='s')
pyplot.show()<jupyter_output><empty_output><jupyter_text>Timeline of cars taken and returned. First lets convert date to date time and set it as an index<jupyter_code>#convert the date to datetime and making it an index
#set date as index

autolib.set_index(autolib.date, drop=True,inplace=True)
autolib.sample()


#convert the index to datetimeIndex
autolib.index = pd.to_datetime(autolib.index)
autolib.sample()


p1 = autolib[autolib['postal_code']== PC1]

plt.figure(figsize=(10,4))
p1['bluecars_taken'].plot()
plt.title('Timeline of Blue Cars Taken from 92320')
plt.xlabel('Date')
plt.xticks( rotation='horizontal')
plt.show()
p2 = autolib[autolib['postal_code']== PC2]

plt.figure(figsize=(10,4))
p1['bluecars_taken'].plot()
plt.title('Timeline of Blue Cars Taken from 94450')
plt.xlabel('Date')
plt.xticks( rotation='horizontal')
plt.show()<jupyter_output><empty_output><jupyter_text>Our data set shows that it does not resemble a normal distribution from our analysis several cells above hence we shall use non parametric tests. we are comparing the mean of cars taken and returned.<jupyter_code># lets perform a mann-whitney u test since we're comparing one pair

from numpy.random import seed
from numpy.random import randn
from scipy.stats import mannwhitneyu

# we already created our sample in autolib
# comparing samples
a = autolib['bluecars_taken']
b= autolib['bluecars_returned']

stat, p=mannwhitneyu(a,b)
print('Statistics=%.3f, p=%.3f' % (stat,p))
# interprete

alpha = 0.05
if p>alpha:
  print('Does not reject HO')
else:
  print('Rejects HO')<jupyter_output>Statistics=3766.000, p=0.377
Does not reject HO
{'fraction_non_alphanumeric': 0.14012917719741647, 'fraction_numerical': 0.05944023214452869, 'mean_word_length': 5.136703044227455, 'pattern_counts': {'":': 0, '<': 53, '<?xml version=': 0, '>': 56, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '287508', 'license': 'MIT', 'n_tokens_mistral': 4356, 'n_tokens_neox': 3811, 'n_words': 1055, 'path': 'Moringa_Data_Science_Prep_W4_Independent_Project_CORE_2021_09_Moreen_Mugambi_PythonNotebook.ipynb', 'repo_name': 'MoreenMarutaData/AUTOLIB-HYPOHESIS-PROJECT-CORE-IP-WEEK-4', 'size': '10683'}