<jupyter_start><jupyter_text>**Moringa Data Science Prep Independent Project: Week 4** In this week's independent project, you will be working as a data scientist working for an electric car-sharing service company. You have been tasked to process stations data to understand electric car usage over time by solving for the following research question;Research QuestionIdentify the most popular hour of the day for picking up a shared electric car (Bluecar) in the city of Paris over the month of April 2018.Bonus Questions (Optional)What is the most popular hour for returning cars?What station is the most popular?Overall?At the most popular picking hour?What postal code is the most popular for picking up Blue cars? Does the most popular station belong to that postal code?Overall?At the most popular picking hour?Do the results change if you consider Utilib and Utilib 1.4 instead of Blue cars?  **Datasets provided**---You can get the data and the dataset description for this Independent project here [http://bit.ly/autolib_dataset (Links to an external site.)] and here respectively [https://drive.google.com/a/moringaschool.com/file/d/13DXF2CFWQLeYxxHFekng8HJnH_jtbfpN/view?usp=sharing(Links to an external site.)].  The dataset contains data collected for a period of 9 days. The dataset may take a bit of some time to load [~ 10 minutes].  **1. Importing our Libraries**<jupyter_code># pandas
import pandas as pd

# numpy
import numpy as np<jupyter_output><empty_output><jupyter_text>**2. Load our dataset**<jupyter_code># get url
url = "http://bit.ly/autolib_dataset"

# load using csv
df = pd.read_csv(url)
df.head()<jupyter_output><empty_output><jupyter_text>**3. Preview our dataset**<jupyter_code># lets preview the first 10 rows of our dataset
df.head(10)<jupyter_output><empty_output><jupyter_text>**4. Accessing information about our dataset**<jupyter_code>print(df.info())<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 5000 entries, 0 to 4999
Data columns (total 25 columns):
 #   Column               Non-Null Count  Dtype 
---  ------               --------------  ----- 
 0   Address              5000 non-null   object
 1   Cars                 5000 non-null   int64 
 2   Bluecar counter      5000 non-null   int64 
 3   Utilib counter       5000 non-null   int64 
 4   Utilib 1.4 counter   5000 non-null   int64 
 5   Charge Slots         5000 non-null   int64 
 6   Charging Status      5000 non-null   object
 7   City                 5000 non-null   object
 8   Displayed comment    111 non-null    object
 9   ID                   5000 non-null   object
 10  Kind                 5000 non-null   object
 11  Geo point            5000 non-null   object
 12  Postal code          5000 non-null   int64 
 13  Public name          5000 non-null   object
 14  Rental status        5000 non-null   object
 15  Scheduled at         47 non-null     object
 16  Slots [...]<jupyter_text>We notice that some of the data types have wrong dtypes which we get to know from looking at the description document https://drive.google.com/a/moringaschool.com/file/d/13DXF2CFWQLeYxxHFekng8HJnH_jtbfpN/view?usp=sharing.<jupyter_code># check shape of the dataset
df.shape<jupyter_output><empty_output><jupyter_text>**5. Cleaning our dataset** Let us perform data cleaning procedures below providing a documentation for our actions and reasons. We will perform as many data cleaning procedures as we think suitable for the various dimensions of data. ***5.1 Validity***<jupyter_code># from the description document we note that cars is always redundant to the column bluecar counter thus we can drop it
df.drop(['Cars'], axis = 1, inplace = True)

df.columns<jupyter_output><empty_output><jupyter_text>***5.2 Accuracy***<jupyter_code># we don't check for this since we have no way of knowing whether the data provided is accurate or not 
# and we have no access to external information<jupyter_output><empty_output><jupyter_text>***5.3 Completness***<jupyter_code># check for null values
df.isnull().sum()<jupyter_output><empty_output><jupyter_text>Since the *'Displayed comment'* and *'Scheduled at'* columns have so much missing data we drop the two columns.<jupyter_code># Displayed comment column
df.drop('Displayed comment', axis = 1, inplace = True)
# Scheduled at 
df.drop('Scheduled at', axis = 1, inplace = True)
df.columns
# we then check for any more missing data
print(df.isnull().sum())
# here we want year, month, day, hour and minute to be in the same column 
df['Date'] = pd.to_datetime(df[['year', 'month', 'day', 'hour', 'minute']])
df.columns
# here we want hour and minute to be in the same column so we split it from Date column
# extracting date from timestamp
df['time'] = pd.to_datetime(df['Date'])
df['dates'] = df['time'].dt.date
df['times'] = df['time'].dt.time
df.head(1)
df.drop(['year', 'month', 'day', 'hour', 'minute', 'time'], axis = 1, inplace = True)
df.columns

df.head(1)
# check for any null values
df.isnull().sum()
# rename our columns
Columns = ['Address', 'Bluecar_Count', 'Utilib_Count', 'Utilib1.4_Count', 'Charge Slots', 'Charging Status', 'City', 'Station_ID', 'Kind', 'Geo_coord', 'Postal_Code', 'Public Name', 'Rental Status', 'Slots', 'Station Type', 'Status', 'Subscription', 'Datetime', 'Date', 'Time' ]
df.columns = Columns
df.columns<jupyter_output><empty_output><jupyter_text>***5.4 Consistency***<jupyter_code># Fixing inconsistency is not always possible: it requires a variety of strategies - e.g.,
# deciding which data were recorded more recently, which data source is likely to be most reliable
# (the latter knowledge may be specific to a given organization), or simply trying to find the truth by 
# testing both data items (e.g., calling up the customer). in this case its not possible<jupyter_output><empty_output><jupyter_text>***5.5 Uniformity***<jupyter_code># this is where we make sure that the data types correspond to the description document 

# checking data types
df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 5000 entries, 0 to 4999
Data columns (total 20 columns):
 #   Column           Non-Null Count  Dtype         
---  ------           --------------  -----         
 0   Address          5000 non-null   object        
 1   Bluecar_Count    5000 non-null   int64         
 2   Utilib_Count     5000 non-null   int64         
 3   Utilib1.4_Count  5000 non-null   int64         
 4   Charge Slots     5000 non-null   int64         
 5   Charging Status  5000 non-null   object        
 6   City             5000 non-null   object        
 7   Station_ID       5000 non-null   object        
 8   Kind             5000 non-null   object        
 9   Geo_coord        5000 non-null   object        
 10  Postal_Code      5000 non-null   int64         
 11  Public Name      5000 non-null   object        
 12  Rental Status    5000 non-null   object        
 13  Slots            5000 non-null   int64         
 14  Station Type     5000 non-null   object [...]<jupyter_text>Date and Time columns need to be converted to datetime<jupyter_code># convert Date column to datetime
df['Date'] = pd.to_datetime(df['Date'])
# convert Time column to datetime
#df['Time'] = (df['Time'].apply(pd.Timestamp))
df['Time'] =  pd.to_datetime(df['Time'], format='%H:%M:%S').dt.time
# inspect the Dtype column to confirm changes
df.dtypes<jupyter_output><empty_output><jupyter_text>**6. Data Analysis**<jupyter_code># Identify the most popular hour of the day for picking up a shared electric car (Bluecar)
# in the city of Paris over the month of April 2018.

# set a variable to just bluecar
a = df['Bluecar_Count']
# set a variable to rental status
b = df['Rental Status']
c = df['City'] == 'Paris'
#df.groupby(['Bluecar_Count', 'Time']).size()

df.groupby(['Time', 'Bluecar_Count', c])['Time'].count().sort_values(ascending = False)

# the most popular hour of the day is at 1600 hours 
# What station is the most popular?
df.groupby(['Station_ID', 'City'])['Station_ID'].count().sort_values(ascending = False)

# the most popular station is paris-portedemontrouge-8 overall
# What station is the most popular? most popular hour
a = df['Time'] == '16:00:00'
df.groupby(['Station_ID', a])['Station_ID'].count().sort_values(ascending=False)

# there are no stations popular at most popular hour<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10128251228574853, 'fraction_numerical': 0.03308162531463502, 'mean_word_length': 3.552100381887616, 'pattern_counts': {'":': 0, '<': 53, '<?xml version=': 0, '>': 53, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '105646', 'license': 'MIT', 'n_tokens_mistral': 2703, 'n_tokens_neox': 2463, 'n_words': 939, 'path': 'ip_W4.ipynb', 'repo_name': 'dote7ant/Moringa_W4', 'size': '8343'}