<jupyter_start><jupyter_text><jupyter_code>#importing the libraries we will need.
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

#loading our autolib dataset
autolib = pd.read_csv('/content/autolib_daily_events_postal_code.csv')
autolib.head()

#loading our column explanation dataset
columns = pd.read_excel('/content/columns_explanation.xlsx')
columns.head()
#getting the information about our autolib dataset
#to understand our data for analysis
autolib.info()
#checking our column explanation dataset
columns.count
#counting our rows and columns in our autolib dataset
autolib.count
#checking  the shape of our autolib dataset
autolib.shape
#we have 13 columns and 16085 rows
autolib.describe()
autolib.dtypes<jupyter_output><empty_output><jupyter_text>DATA CLEANING<jupyter_code>#cleaning data using strip(),lower(), replace()
autolib.columns =autolib.columns.str.strip().str.lower().str.replace(' ', '_').str.replace('(', '').str.replace(')', '')
autolib.columns.to_list()

#checking for missing values
autolib.isnull().sum()
#our data has no null values
autolib['postal_code'].nunique()
# we have 104 unique postal codes.
#checking for duplicates in our data.
autolib.duplicated().sum()
#we have no duplicates in the dataset
# checking for outliers
plt.figure(figsize=(20,6))
sns.boxplot(data = autolib, showmeans = True)
#we have outliers in our dataset.
#  the cars were picked in different locations and many were returned. we are not going to get rid of the outliers<jupyter_output><empty_output><jupyter_text>UNIVARIATE ANALYSIS<jupyter_code># dataset with the day type as week day and as a weekend
autolib_weekday=autolib[autolib['day_type']=='weekday']
autolib_weekend=autolib[autolib['day_type']=='weekend']
#counting the weekend dataset
autolib_weekend.count()
#there are 4541 weekend days
#counting the weekday dataset
autolib_weekday.count()
# there are 11544 week days 


# from our analysis above, we have more week days working than the weekends
#comparing the different type of days in a graph
plt.figure(figsize=(10, 6))
sns.countplot(autolib.day_type,order = autolib.day_type.value_counts().index)
# just as mentioned above , the number of week days is higher than the weekends for the car usage
#Numerical analysis
my_numerical_analysis = ['postal_code', 'n_daily_data_points', 'dayofweek',
       'bluecars_taken_sum', 'bluecars_returned_sum', 'utilib_taken_sum',
       'utilib_returned_sum', 'utilib_14_taken_sum', 'utilib_14_returned_sum',
       'slots_freed_sum', 'slots_taken_sum']
for col in my_numerical_analysis:
  plt.hist(autolib[col],bins=15,color="blue")
  plt.title(col)
  plt.show()
autolib.describe(include='all')<jupyter_output><empty_output><jupyter_text>BIVARIATE ANALYSIS<jupyter_code># Correlation
plt.figure(dpi = 120,figsize= (5,4))
mask = np.triu(np.ones_like(autolib.corr(),dtype = bool))
sns.heatmap(autolib.corr(),mask = mask, fmt = ".2f",annot=True,lw=1,cmap = 'plasma')
plt.yticks(rotation = 0)
plt.xticks(rotation = 90)
plt.title('Correlation Heatmap')
plt.show()

df1 = autolib.corr()
sns.heatmap(df1)<jupyter_output><empty_output><jupyter_text>SAMPLING<jupyter_code># Simple Random Sampling 
weekend_sample =autolib_weekend.sample(n = 10, replace="False")
weekend_sample
#i would like to know more about the bluecars taken on the weekdays and the weekend.
autolib.groupby('day_type')['bluecars_taken_sum'].describe()
#the weekend mean is higher than the weekday mean, weekday = 116.028673 whereas the weekend = 151.090068
#standard deviation on the weekend is also higher than week day.
# on the quartile ranges, the weekend still has a higher value than weekday<jupyter_output><empty_output><jupyter_text>HYPOTHESIS TESTING.In my analysis, i would like to know if the number of bluecars taken in area X have any difference with bluecars taken in area Y over the weekend.H0 =  no of bluecars taken in area X >= the number of cars taken in area Y over the weekendHa = no of bluecars taken from area X < the no of bluecars taken in area Y over the weekend. our leveel of significance is 0.05<jupyter_code># we want to know about area Y and area X. our postal code is 75015 and 75017
# we filter our data to the postal code.
area_Y = autolib_weekend[autolib_weekend.postal_code == 75017]
area_Y.reset_index(inplace= True)
#we find the central tendencies of area_Y
area_Y.groupby('day_type')['bluecars_taken_sum'].describe()
#confidence interval
# finding the standard error for the mean interval
error = 831.022727-(1.96*159.843454)
print(error)
p_error=831.022727+(1.96*159.843454)
print(p_error)
#we are 95% confident the mean is between 517.72955716 and 1144.31589684
#our area_X
area_X = autolib_weekend[autolib_weekend.postal_code == 75015]
area_X.reset_index(inplace= True)
#we get their central tendencies
area_X.groupby('day_type')['bluecars_taken_sum'].describe()

# from the above analysis, area_X has a higher number of bluecars taken compared to area_Y
# it has a higher mean of 1072.772727 and area_Y = 831.022727<jupyter_output><empty_output><jupyter_text>T testthe sample i am working with is less than 30 , i can use t test<jupyter_code>x = (1072.772727-1000)/211.909136
y = (831.022727-1000)/159.843454<jupyter_output><empty_output><jupyter_text>P Value<jupyter_code>from scipy import stats
from scipy.stats import norm
#PValue
prob = stats.norm.cdf(x)
prob
# the P value is greater than the significance level, we fail to reject the null hypothesis
#P Value
prob = stats.norm.cdf(y)
prob
# the P V alue is greater than the significance level so we fail to reject the null hypothesis.<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09746138824782531, 'fraction_numerical': 0.03905556541807208, 'mean_word_length': 6.1046658259773015, 'pattern_counts': {'":': 0, '<': 34, '<?xml version=': 0, '>': 34, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '212326', 'license': 'MIT', 'n_tokens_mistral': 2101, 'n_tokens_neox': 1883, 'n_words': 591, 'path': 'week_4_ip.ipynb', 'repo_name': 'jane95924/week4_ip', 'size': '5633'}