<jupyter_start><jupyter_text>Autolib electric car sharing service analysis **Overview**We will work as a Data Scientist for the Autolib electric car-sharing service company to investigate a claim about the blue cars from the provided Autolib dataset. **Defining our research question** We will try to understand bluecar usage during the weekends by comparing two different areas indicated by their postal code Specific questions1.   Is the number of bluecars taken in area 95880 different that in area 75012?2.   Which day of the weekend were the bluecars taken most in the two areas?1.   is there a difference between the sum of bluecars taken and the sum of bluecars returned? Find more context and step by step analysis explanation here: https://docs.google.com/document/d/15bV_PE9YpPGUkWzPN-QwDwqQC7uTEisWcc5BoLuUxqM/edit?usp=sharing  **Processing our dataset to understand electric car usage over time**<jupyter_code>##Importing our libraries

import pandas as pd

import numpy as np

import seaborn as sns

import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>1.0 Reading the Dataset<jupyter_code>##Dataset url= http://bit.ly/DSCoreAutolibDataset


##autolib_df = pd.read_csv('autolib_daily_events_postal_code.csv')
##autolib_df

autolib_df= pd.read_csv('http://bit.ly/DSCoreAutolibDataset')

#Previewing the first 10 records of dataset

autolib_df.head(10)

#Previewing the last 10 records of dataset

autolib_df.tail(10)

#Data dimension
autolib_df.shape
##Finding uniques values
autolib_df.nunique()
#Confirming the datatypes
autolib_df.dtypes<jupyter_output><empty_output><jupyter_text>2.0 Data Cleaning<jupyter_code>#From our initial data investigation above there are some things about our variables we would like to change:

#Completeness~ We will check for missing values in our dataset
#Consistency~ We need to rename some of our column names to be uniform and make it easy to work with
##We will also check for duplicates##Our date columns will be changed to datetime 
#Validity ~ We will drop columns that are irrelevant to our analysis
#Accuracy~ We will eliminate outliers<jupyter_output><empty_output><jupyter_text>**Checking for missing values**<jupyter_code>autolib_df.isnull().sum()
#There are no missing values in our datasets<jupyter_output><empty_output><jupyter_text>Checking for duplicates<jupyter_code>autolib_df.duplicated().sum()
#There are no duplicates in our dataset<jupyter_output><empty_output><jupyter_text>**Renaming the column names with uniform names**<jupyter_code>#
autolib_df.columns
autolib_df.rename(columns={"Postal code" : "postal_code", "BlueCars_taken_sum" :"blue_cars_taken_sum", "BlueCars_returned_sum" :"blue_cars_returned_sum","Utilib_returned_sum":"utilib_returned_sum","Utilib_taken_sum":"utilib_taken_sum","Utilib_14_taken_sum":"utilib_14_sum", "Utilib_14_returned_sum":"utilib_14_returned_sum", "Slots_freed_sum":"slots_freed_sum", "Slots_taken_sum":"slots_taken_sum", "dayOfWeek" : "day_of_week"},inplace = True)
#Confirming change of columns names
autolib_df.columns
## We also noticed our date column listed as dtype object hence will convert it to datetime

##Importing the datetime library
from datetime import datetime

autolib_df['date'] = pd.to_datetime(autolib_df['date'], errors='coerce')

#Confirming the changed dtype
print(autolib_df.dtypes)<jupyter_output>postal_code                        int64
date                      datetime64[ns]
n_daily_data_points                int64
day_of_week                        int64
day_type                          object
blue_cars_taken_sum                int64
blue_cars_returned_sum             int64
utilib_taken_sum                   int64
utilib_returned_sum                int64
utilib_14_sum                      int64
utilib_14_returned_sum             int64
slots_freed_sum                    int64
slots_taken_sum                    int64
dtype: object<jupyter_text>**Dropping the columns and records that we do not need**<jupyter_code>##Since we are doing an analysis on a claim about blue cars, we will drop the following irrelevant columns
##utilib_taken_sum
##utilib_returned_sum
##utilib_14_sum
##utilib_14_returned_sum
##slots_freed_sum
##slots_taken_sum

#Remove all columns between column index 7 to 14

autolib_df.drop(autolib_df.iloc[:, 7:], inplace = True, axis = 1)


#Previewing our columns
autolib_df.columns<jupyter_output><empty_output><jupyter_text>Using filter conditions to create our new dataset<jupyter_code>#Since we are working with a specific day of the week type we need to create conditions that capture data in that frame
#Postal codes 95880 and 75012 and day of week = weekend
df1= autolib_df[(autolib_df.postal_code == 95880) & (autolib_df.day_type=='weekend')]
df2= autolib_df[(autolib_df.postal_code ==75012) & (autolib_df.day_type=='weekend')]
print(df1)
print(df2)<jupyter_output><empty_output><jupyter_text>**Outliers**<jupyter_code># Potstal Code 95880
plt.figure(figsize=(6,4))
plt.boxplot(df1['blue_cars_taken_sum'])
plt.show()

##Has outliers
#Cars returned
plt.figure(figsize=(6,4))
plt.boxplot(df1['blue_cars_returned_sum'])
plt.show()
##Postal code 75012
plt.figure(figsize=(6,4))
plt.boxplot(df2['blue_cars_taken_sum'])
plt.show()
plt.figure(figsize=(6,4))
plt.boxplot(df2['blue_cars_returned_sum'])
plt.show()<jupyter_output><empty_output><jupyter_text>Dealing with the outliers<jupyter_code>below_10=df1[df1.blue_cars_taken_sum < 10].index
df1.drop(below_10, inplace = True)
below_500=df2[df2.blue_cars_taken_sum < 500].index
df2.drop(below_500, inplace = True)
below_10=df1[df1.blue_cars_returned_sum > 50].index
df1.drop(below_10, inplace = True)
below_10=df2[df2.blue_cars_returned_sum < 500].index
df2.drop(below_10, inplace = True)
frames = [df1, df2]
df = pd.concat(frames, sort=False)
df= df.drop(['n_daily_data_points'], axis = 1)
df
df['postal_code']= df['postal_code'].astype('category')
df['day_of_week']= df['day_of_week'].astype('category')
df.duplicated().sum()<jupyter_output><empty_output><jupyter_text>3.0 Exploratory Data Analysis **Univariate Analysis**<jupyter_code>df.describe()<jupyter_output><empty_output><jupyter_text>Central Measure of Tendancy<jupyter_code>#From the data above we can deduce the following central measures of tendancies
#MEAN

print('The mean of bluecars taken is: ' +str(df.blue_cars_taken_sum.mean()))
print('The mean of bluecars returned is: ' +str(df.blue_cars_returned_sum.mean()))
##VARIANCE

print('The variance of bluecars taken is: ' +str(df.blue_cars_taken_sum.var()))
print('The variance of bluecars returned is: ' +str(df.blue_cars_returned_sum.var()))
#SKEWNESS

print('The skewness of bluecars taken is: ' +str(df.blue_cars_taken_sum.skew()))
print('The skewness of bluecars returned is: ' +str(df.blue_cars_returned_sum.skew()))

#The values close to zero indicate that the symmetry of our data is close to a bell-shaped curve(normal distribution)


#KURTOSIS

print('The kurtosis of bluecars taken is: ' +str(df.blue_cars_taken_sum.kurt()))
print('The kutrosis of bluecars returned is: ' +str(df.blue_cars_returned_sum.kurt()))
#RANGE 

print('The range of bluecars taken is: ' +str(df.blue_cars_taken_sum.max()-df.blue_cars_taken_sum.min()))
print('The range of bluecars returned is: ' +str(df.blue_cars_returned_sum.max()-df.blue_cars_returned_sum.min()))<jupyter_output>The range of bluecars taken is: 803
The range of bluecars returned is: 783<jupyter_text>**Bivariate Analysis** Categorical and Numerical<jupyter_code>#Let us compare which day of the weekend has the highest sum of bluecars taken

print(df.groupby('day_of_week')['blue_cars_taken_sum'].sum())
df.groupby('day_of_week')['blue_cars_taken_sum'].sum().plot.bar()

##We can see that day 5 has a slightly higher number of blue cars taken in both areas but not significantly high
##Comparing the blue cars taken and blue cars returned at each postal code.
fig, (ax1, ax2)= plt.subplots(1,2, figsize=(15, 7))
fig.suptitle("")
sns.barplot(x= df.postal_code, y=df.blue_cars_taken_sum, ax=ax1)
sns.barplot(x= df.postal_code, y=df.blue_cars_returned_sum, ax=ax2)
plt.show()<jupyter_output><empty_output><jupyter_text>We can deduce from the visualization above that the number of blue cars taken and those returned from postal code 75012 is way higher that those taken and returned in area postal code 95880. This is farther emphasised by the visualization below comparing the bluecars taken and blue cars returned to each postal address and day of the week<jupyter_code>#Comparing the sum of blue cars taken and returned to our subject postal areas to the day of week.
ax = sns.pointplot(x="postal_code", y="blue_cars_taken_sum", hue="day_of_week",
                   data=df, dodge=True)
ax = sns.pointplot(x="postal_code", y="blue_cars_returned_sum", hue="day_of_week",
                   data=df, dodge=True)<jupyter_output><empty_output><jupyter_text>General correlation overview<jupyter_code>#Correlation
correlation= df.corr()
correlation
#Using Heatmaps to find the correlation of our numerical variables
sns.heatmap(correlation, 
            xticklabels = correlation.columns.values,
            yticklabels = correlation.columns.values,
            annot = True);
#Finding correlation between our variables using scatterplots
plt.figure(figsize=(14, 14))

sns.pairplot(df, diag_kind='kde');<jupyter_output><empty_output><jupyter_text>There seems to be a strong direct relationship between the sum of blue cars taken and the sum of blue cars returned. 4.0 Challenging our solution We have found out from our basic analysis techniques above that:1.  The sum of blue cars taken from postal code 75012 are higher than those taken from 958802.  The sum of blue cars returned from postal code 75012 are higher than those returned from 958801.   There is a strong relationship between the sum of blue cars taken and the sum of blue cars returned in each of the stations  **To challenge our solution:(relationship between bluecars taken and those returned) we will embark on hypothesis testing**<jupyter_code>##Checking for normalcy in our distribution

df.blue_cars_taken_sum.hist()
df1= autolib_df[(autolib_df.postal_code == 95880) & (autolib_df.day_type=='weekend')]
df2= autolib_df[(autolib_df.postal_code ==75012) & (autolib_df.day_type=='weekend')]


##Choosing random saples from both independent groups to perform our hypothesis
import random

df1 = df1.sample(n = 25, replace="False")
df2 = df2.sample(n = 25, replace="False")
df1.blue_cars_taken_sum.hist()
df2.blue_cars_taken_sum.hist()
#From the normalcy testing above we notice that our data is not bell shaped 

##Step 1: Stating our hypothesis

#Null Hypothesis: There is no difference in the sum of bluecars taken and sum of bluecars returned in area 95880
#Alternate hypothesis: There is a difference in the sum of bluecars taken in areas 95880
Ho: Θ=Θo (The sum of bluecars taken and those returned in area 95880 are equal)
Ha: Θ ≠ Θo (The sum of bluecars taken and those returned in the postal code 95880 not equal )
##This means that it is going to be a two sided test
##Setting our significance level or alpha
#Our alpha is going to be 0.05

##In order to decide on the test, we need to check whether the two populations have equal variances or not. As a rule of thumb, we can assume the populations have equal variances if the ratio of the larger sample variance to the smaller sample variance is less than 4:1. 

print(np.var(df1['blue_cars_taken_sum']), np.var(df1['blue_cars_returned_sum']))
##we can assume that the population variances are not equal since the ratio is greater than 4:1
##We can then proceed to perform the Mann–Whitney U test (also called the Mann–Whitney–Wilcoxon (MWW), Wilcoxon rank-sum test (WRS), or Wilcoxon–Mann–Whitney test)

import scipy.stats as stats

stats.mannwhitneyu(df1['blue_cars_taken_sum'], df1['blue_cars_returned_sum'], alternative='two-sided')

##Doing the same for postal code 75012

stats.mannwhitneyu(df2['blue_cars_taken_sum'], df2['blue_cars_returned_sum'], alternative='two-sided')<jupyter_output><empty_output><jupyter_text>Interpreting our results<jupyter_code>##our pvalues are 0.28 (rounded off) and 0.95 (rounded off) which compared to our alpha 0.05 is greater.
#p>alpha 
#thus we fail to reject the null hypothesis. We do not have sufficient evidence to say that the mean is different between the two groups.<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08961616161616162, 'fraction_numerical': 0.023515151515151517, 'mean_word_length': 5.704225352112676, 'pattern_counts': {'":': 6, '<': 70, '<?xml version=': 0, '>': 69, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '552826', 'license': 'MIT', 'n_tokens_mistral': 4343, 'n_tokens_neox': 3966, 'n_words': 1255, 'path': 'Autolib_bluecar_usage_analysis.ipynb', 'repo_name': 'KevinaZeni/Autolib_electric_bluecar_usage_hypothesis_testing', 'size': '12375'}