<jupyter_start><jupyter_text>1. Load Libraries<jupyter_code>import numpy as np
import pandas as pd

import matplotlib.pyplot as plt
import seaborn as sns

from mlp import pipeline

%matplotlib inline<jupyter_output><empty_output><jupyter_text>2. Load Dataset<jupyter_code># Extract and retrieve rentals data from Microsoft SQL server
# Refer to documentation within data module for technical and configuration details
df_rentals = pipeline.get_rentals()

df_rentals.head()<jupyter_output><empty_output><jupyter_text>3. Data Insights<jupyter_code>df_rentals.shape<jupyter_output><empty_output><jupyter_text>- Dataset contains 18,643 observations with 10 features.- There are 24 hours a day, 365 days a year. So over 2 years, there should be a maximum 17,520 (24 x 365 x 2) observations.- Given that there are more hourly observations than hours over a 2 year period, some of the observations may be duplicates or erroneous. - The problem statement is to predict the total number of active e-scooter users given the above dataset.- Each observation records the number of guest and registered users using rental e-scooters in a particular hour of a day.- I shall assume that the total number of active e-scooter users in a particular hour of a day is the sum of the guest and registered users i.e. active users = guest users + registered users.<jupyter_code>df_rentals.columns.values<jupyter_output><empty_output><jupyter_text>- Column labels of the rentals dataset<jupyter_code>df_rentals.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 18643 entries, 0 to 18642
Data columns (total 10 columns):
date                      18643 non-null object
hr                        18643 non-null int64
weather                   18643 non-null object
temperature               18643 non-null float64
feels_like_temperature    18643 non-null float64
relative_humidity         18643 non-null float64
windspeed                 18643 non-null float64
psi                       18643 non-null int64
guest_scooter             18643 non-null int64
registered_scooter        18643 non-null int64
dtypes: float64(4), int64(4), object(2)
memory usage: 1.4+ MB<jupyter_text>- No column with null/missing value.  4. Summary Statistics<jupyter_code>df_rentals.describe()<jupyter_output><empty_output><jupyter_text>- Large differnece in the 75th %tile and max values of columns windspeed, guest_scooter, registered_scooter- This observation suggests that there are extreme values or outliers in these columns. - There is sizable difference in the range of values across the independent variables. e.g. values for psi are always below 100, whereas values for registered_scooter could be in the thousands.- Some form of scaling needs to be done at the pre-processing stage.  5. Data Cleaning 5.1 date Column<jupyter_code># Check data type of the date column
df_rentals.dtypes['date']<jupyter_output><empty_output><jupyter_text>- Convert the date column from string to date data type.- Combine the date and hr columns to form a datetime column.- This is to facilitate the use of datetime/timeseries operations when doing exploration and feature engineering later.<jupyter_code># Rename date column to date_str to indicate string data type
df_rentals.rename(columns={'date': 'date_str'}, inplace=True)

# Convert date column from string to datetime data type
df_rentals['date'] = pd.to_datetime(df_rentals['date_str'])

# Verify date column data type
df_rentals.dtypes['date']
# Create datetime column by concatenating the date and hr columns
df_rentals['datetime'] = df_rentals.apply(lambda row: row.date_str + ' ' + str(row.hr), axis=1) + ':00'

# Convert datetime column from string to datetime data type
df_rentals.datetime = pd.to_datetime(df_rentals.datetime)

# Verify datetime column data type
df_rentals.dtypes['datetime']<jupyter_output><empty_output><jupyter_text>5.2 hr Column<jupyter_code># Check data type of the hr column
df_rentals.dtypes['hr']<jupyter_output><empty_output><jupyter_text>- The hour of the day when the rentals are made should be categorical in nature.- Convert the hr column from integer to string data type.<jupyter_code># Rename hr column to hr_str to indicate string data type
df_rentals.rename(columns={'hr': 'hr_str'}, inplace=True)

# Convert hr column from int to string data type
df_rentals.hr_str = df_rentals.hr_str.apply(str)

# Verify hr_str column data type
df_rentals.dtypes['hr_str']
# Check the number of unique hr values
unique_hrs = df_rentals.hr_str.unique()

unique_hrs
len(unique_hrs)<jupyter_output><empty_output><jupyter_text>- All 24 hours of the day are represented in the rentals dataset.<jupyter_code># Convert the hr column from string to categorical data type
df_rentals['hr'] = df_rentals.hr_str.astype('category')

df_rentals.dtypes['hr']<jupyter_output><empty_output><jupyter_text>5.3 weather Column<jupyter_code>df_rentals.weather.unique()<jupyter_output><empty_output><jupyter_text>- The weather column contains categorical data.- The weather data is 'dirty', clean up is neccessary. - Mixed cases i.e. clear and CLEAR..- Incorrect spelling e.g. lear, clar- Correct values 'lear' and 'clar' to be 'clear'.- Correct values 'cludy' and 'loudy' to be 'cloudy'.- Correct value 'liht snow/rain' to be 'light snow/rain'.<jupyter_code># Standardized weather column to lower case characters
df_rentals.weather = df_rentals.weather.str.lower()

dict_weather = {

    # Replace incorrect values 'lear' and 'clar' with 'clear'
    'lear': 'clear',
    'clar': 'clear',

    # Replace incorrect values 'cludy' and 'loudy' with 'cloudy'
    'cludy': 'cloudy',
    'loudy': 'cloudy',

    # Replace incorrect value 'liht snow/rain' with 'light snow/rain'
    'liht snow/rain': 'light snow/rain'
}

# Replace incorrect values in weather column
df_rentals.replace({'weather': dict_weather}, inplace=True)

# Verify that the incorrect values have been replaced
df_rentals.weather.unique()
# Convert the weather column from string to categorical data type
df_rentals['weather'] = df_rentals.weather.astype('category')

df_rentals.dtypes['weather']<jupyter_output><empty_output><jupyter_text>- The weather column contains 4 unique categorical values i.e. clear, cloudy, light snow/rain and heavy snow/rain.- One-hot encoding can be applied to the weather column later in feature engineering.  5.4 temperature, feels_like_temperature Columns<jupyter_code># Get the maximum and minimum temperature recorded
max(df_rentals.temperature), min(df_rentals.temperature)
# Get maximum and minimum feels_like_temperature recorded
max(df_rentals.feels_like_temperature), min(df_rentals.feels_like_temperature)
# Number of observations with temperatures above 120°F
len(df_rentals[df_rentals.temperature > 120])<jupyter_output><empty_output><jupyter_text>- I shall assume that values from the temperature and feels_like_temperature columns are in fahrenheit.- I shal assume that this dataset is gathered from a city/town since people are renting e-scooters and e-bikes.- The maximum value of the temperature column is 131°F which is pretty close to the [highest temperature ever recorded](https://en.wikipedia.org/wiki/List_of_weather_recordsHighest_temperatures_ever_recorded) of 134.1°F.- According to [TripSavvy](https://www.tripsavvy.com/the-worlds-hottest-cities-4070053), some of the highest temperatures recorded in a city include Phoenix 122°F, Marrakech 120°F, Mecca 121.6°F, Kuwait City 126°F, Ahvaz 129°F and Timbuktu 120°F.- There are 240 observations with temperatures above 120°F.  This dataset should be from a city known for its high temperatures. If otherwise, the temperatures in these observations need to be verified.- 'Feels like' temperature is also known as the [heat index](https://en.wikipedia.org/wiki/Heat_index).  In short, it is a temperature reading that factors in a component of relative humidity.- We can verify the values of the feels_like_temperature column using the heat index [formula](https://en.wikipedia.org/wiki/Heat_indexFormula).- Without any geographical information on this dataset given, I shall assume that all temperature readings are accurate.   5.5 relative_humidity Column<jupyter_code># Get the maximum and minimum values of relative humidity recorded
max(df_rentals.relative_humidity), min(df_rentals.relative_humidity)
# Number of observations with 0 relative humidity
len(df_rentals[df_rentals.relative_humidity==0])<jupyter_output><empty_output><jupyter_text>- [Relative humidity](https://en.wikipedia.org/wiki/Relative_humidity) (RH) is the actual amount of water vapor present in relation to the capacity that the air has at a particular temperature.  It is express as a percentage.- A relative humidity reading of 0 implies [air devoid of water vapor](https://www.chicagotribune.com/news/ct-xpm-2011-12-16-ct-wea-1216-asktom-20111216-story.html). This is quite impossible given the climate conditions of a city/town, where I assume this dataset is gathered.  Values of 0 in the relative_humidity column need to be verified.- Since there are only 25 observations with 0 relative humidity, I've decided to drop them.- A relative humidity reading of 100 means that the air is totally saturated with water vapor and cannot hold any more, creating the possibility of rain.  So values of 100 in the relative_humidity column are valid.<jupyter_code># Number of observations in dataset
len(df_rentals)
# Drop observations with relative humidity value of 0
df_rentals.drop(df_rentals[df_rentals.relative_humidity==0].index, inplace=True)

# Check number of observations left after dropping
len(df_rentals)<jupyter_output><empty_output><jupyter_text>5.6 windspeed Column<jupyter_code># Get the maximum and minimum values of the windspeed column
max(df_rentals.windspeed), min(df_rentals.windspeed)<jupyter_output><empty_output><jupyter_text>- No units were given for the windspeed column.- Apparently, wind speed can be measured using a variety of [units](https://en.wikipedia.org/wiki/Wind_speedUnits) e.g. beaufort, knots, m/s, km/h, mph, depending on purpose, region or target audience.- [Wind speed of 0](https://www.wral.com/weather/blogpost/1116592/) is possible and said to be calm.- I'm unable to gauge if the maximum wind speed of 57 is valid. 57 m/s implies a hurricane, but 57 km/h is just a near gale.  - As such, I shall assume that values in the windspeed column are valid.  5.7 psi Column<jupyter_code># Get the maximum and minimum values of the psi column
max(df_rentals.psi), min(df_rentals.psi)<jupyter_output><empty_output><jupyter_text>- The [Pollutant Standard Index (psi)](https://en.wikipedia.org/wiki/Pollutant_Standards_Index) is a measure of pollutants present in the air (0 to 400). - Values in the psi column are valid.  5.8 guest_scooter, registered_scooter Columns<jupyter_code># Get the maximum and minimum values of the guest_scooter column
max(df_rentals.guest_scooter), min(df_rentals.guest_scooter)
# Get the maximum and minimum values of the registered_scooter column
max(df_rentals.registered_scooter), min(df_rentals.registered_scooter)
# Number of observations with a negative value in either the guest_scooter or registered_scooter columns
len(df_rentals[(df_rentals.guest_scooter<0) | (df_rentals.registered_scooter<0)])<jupyter_output><empty_output><jupyter_text>- Values in the guest_scooter and registered_scooter columns indicate the number of guest and registered users renting e-scooters in a particular hour, of a particular date.- As such, the values in the guest_scooter and registered_scooter columns should not be negative.- There are 658 observations with a negative value in either the guest_scooter or registered_scooter columns.- As there is no way of verifying these erroneous values, I shall set all negative values in the guest_scooter or registered_scooter columns to 0.<jupyter_code># Set all negative values in the guest_scooter column to 0
df_rentals.loc[df_rentals.guest_scooter < 0, 'guest_scooter'] = 0

# Set all negative values in the registered_scooter column to 0
df_rentals.loc[df_rentals.registered_scooter < 0, 'registered_scooter'] = 0

# Verify that there all negative values in the guest_scooter and registered_scooter columns have been set to 0
len(df_rentals[(df_rentals.guest_scooter<0) | (df_rentals.registered_scooter<0)])<jupyter_output><empty_output><jupyter_text>5.9 Duplicate Observations<jupyter_code># Number of observations in dataset
len(df_rentals)<jupyter_output><empty_output><jupyter_text>- As mentioned in Section 3. [Data Insights](data_insights), there are more hourly observations than hours over 2 years from 2011 to 2012.- There are 18,618 hourly observations versus 17,520 (24 x 365 x 2) hours in the years 2011 and 2012.- Therefore, there are duplicate or erroneous observations in the dataset.<jupyter_code># Number of observations that are duplicates
len(df_rentals[df_rentals.duplicated()])<jupyter_output><empty_output><jupyter_text>- There are 1,609 duplicate observations in the dataset. - I shall drop these duplicated observations.<jupyter_code># Drop duplicate observations
df_rentals.drop_duplicates(inplace=True)

# Verify that the duplicate observations have been removed
len(df_rentals), any(df_rentals.duplicated())
# Verify that all 17,009 observations have unique datetime values
len(df_rentals.datetime.unique())<jupyter_output><empty_output><jupyter_text>6. Target/Dependent Variable- The target variable i.e. active e-scooter users, is numerical and discrete in nature.- As mentioned in Section 3. [Data Insights](data_insights), the target variable (active e-scooter users) will be the sum of the guest and registered e-scooter users.- The active_scooter column should be created AFTER data cleaning as both the guest_scooter and registered_scooter columns contain errors.- Creating the active_scooter column before data cleaning would have introduced those pre-existing errors into the target variable column.<jupyter_code># Create active_scooter column as target variable
df_rentals['active_scooter'] = df_rentals.guest_scooter + df_rentals.registered_scooter

# Verify target variable column has been created
df_rentals.columns<jupyter_output><empty_output><jupyter_text>7. Feature Engineering 7.1 Day of the Week - The day of the week will probably have an impact on the number of rentals.  There could be more rentals on work days (Mon-Fri) as people commute to work, and less on weekends (Sat-Sun) as people stay at home.- I will create a new feature/variable based on the day of week.- The day of the week is categorical in nature.<jupyter_code># Create day_of_wk column as independent variable
df_rentals['day_of_wk'] = df_rentals.apply(lambda row: row.datetime.strftime('%A'), axis=1)

# Verify day_of_wk variable column has been created
df_rentals.columns
# Convert the day_of_wk column from string to categorical data type
df_rentals['day_of_wk'] = df_rentals.day_of_wk.astype('category')

df_rentals.dtypes['day_of_wk']<jupyter_output><empty_output><jupyter_text>7.2 One-Hot Encoding- There are several independent variables that are categorical in nature .i.e. the [hr](hr_column) (Section 5.2), [weather](weather_column) (Section 5.3)  and [day_of_wk](day_of_the_week) (Section 6.2) columns.- To facilitate further exploration and modelling later, I will one-hot encode these columns.- I did not choose to label encode as assigning a running number series to categories has the disadvantage that the numerical values can be misinterpreted by machine learning algorithms as having some sort of hierarchy/order in them.- After encoding, the original column encoded will be removed.  As such, I shall store the encoded dataset in a separate dataframe i.e. df_rentals_1hot, as I may need the original un-encoded dataset at a later stage.  Also, not all algorithms require categorical variables to be one-hot encoded.<jupyter_code># One-hot encode the hr column
df_rentals_1hot = pd.get_dummies(df_rentals, columns=['hr'], prefix=['hr'])

# Verify hr encoding columns were created
df_rentals_1hot.columns
# Create binary values for weather category values
df_rentals_1hot = pd.get_dummies(df_rentals_1hot, columns=['weather'], prefix=['weather'])

# Verify weather encoding columns were created
df_rentals_1hot.columns
# One-hot encode the day_of_wk column
df_rentals_1hot = pd.get_dummies(df_rentals_1hot, columns=['day_of_wk'], prefix=['day_of_wk'])

# Verify day_of_wk encoding columns were created
df_rentals_1hot.columns<jupyter_output><empty_output><jupyter_text>7. Data Visualization 7.1 Correlation<jupyter_code># Column labels of all numerical independent variables
cols_numerical = ['guest_scooter', 'registered_scooter', 'temperature', 'feels_like_temperature', 
                  'relative_humidity', 'windspeed', 'psi']

# Column labels of weather one-hot encoded variables
cols_weather = ['weather_clear', 'weather_cloudy', 'weather_heavy snow/rain', 'weather_light snow/rain']

# Column labels of day of week one-hot encoded variables
cols_day_of_wk = ['day_of_wk_Friday', 'day_of_wk_Monday', 'day_of_wk_Saturday', 'day_of_wk_Sunday', 
                  'day_of_wk_Thursday', 'day_of_wk_Tuesday', 'day_of_wk_Wednesday']

# Column labels of hour one-hot encoded variables
cols_hr = ['hr_0', 'hr_1', 'hr_10', 'hr_11', 'hr_12', 'hr_13', 'hr_14', 'hr_15', 'hr_16', 'hr_17', 'hr_18', 'hr_19', 
           'hr_2', 'hr_20', 'hr_21', 'hr_22', 'hr_23', 'hr_3', 'hr_4', 'hr_5', 'hr_6', 'hr_7', 'hr_8', 'hr_9']

cols_categorical = []

cols_categorical.extend(cols_weather)
cols_categorical.extend(cols_day_of_wk)
cols_categorical.extend(cols_hr)

# Construct list of column labels of numerical and one-hot encoded variables 
cols_all = ['active_scooter']

cols_all.extend(cols_numerical)
cols_all.extend(cols_categorical)
# Create new dataset for visualization purposes
df_rentals_viz = df_rentals_1hot.loc[:, cols_all]

# Generate the correlation matrix between features of the rental dataset
corr_rentals = df_rentals_viz.corr()
plt.figure(figsize=(19, 17))

# Display heat map of the correlation matrix
sns.heatmap(corr_rentals, cmap='coolwarm', annot=False, vmax=1, vmin=-1) 
# Get the correlation coefficients of the target variable (active_scooter) 
coefficients = corr_rentals['active_scooter'].sort_values(ascending=False)

# Get 10 most positively correlated indepedent variable
coefficients.iloc[:10]
# Get 10 most negatively correlated indepedent variable
coefficients.iloc[-10:]<jupyter_output><empty_output><jupyter_text>- registered_scooter has strong positive correlation (> 0.99) to the target variable active_scooter.  - This is expected as active_scooter is the sum of registered_scooter and guest_scooter. - guest_scooter also has a positive correlation but not as strongly as registered_scooter. - This is because registered_scooter is the bigger part of the summation, on average it contributes up to 90% of the value of active_scooter.- Features most correlated to the target variable are registered_scooter, guest_scooter, temperature, feels_like_temperature and relative_humidity.<jupyter_code># Create a dataset to compare guest, registered and total active users 
df_users = df_rentals_viz.loc[:, ['active_scooter', 'guest_scooter', 'registered_scooter']]

# Create a column that shows the percentage of registered users in the total active users
df_users['registered_scooter_%'] = df_users['registered_scooter'] / df_users['active_scooter'] * 100

# Get the mean (in %) of registered_scooter's contribution towards the active_scooter value
df_users['registered_scooter_%'].mean()<jupyter_output><empty_output><jupyter_text>- temperature and feels_like_temperature presents strong positive correlation (0.99).  This is probably because feels_like_temperature, also known as heat index (refer to Section 5.4 [temperature, feels_like_temperature Columns](temperature_feels_like_temperature_columns), is derived from temperature and relative_humidity.- weather_clear and weather_light snow/rain are the more significant weather conditions.- day_of_wk_Saturday and day_of_wk_Sunday are the more significant days in the week.- It is intersting to note that day_of_wk_Saturday and day_of_wk_Sunday are positively correlated to guest_scooter but negatively correlated to registered_scooter.- We can infer that on weekends (Saturday, Sunday), the number of guest users will increase while the registered users will drop. - hr_8, hr_16, hr_17, hr_18, hr_19 are the more positively correlated hourly intervals. These time slots relate to the commute to work in the morning (08:00am) and the commute after work in the evening (04:00pm - 07:00pm).- hr_23, hr_0, hr_1, hr_2, hr_3, hr_4, hr_5, hr_6 are the more negatively correlated hourly intervals. These time slots relate to people resting at home (11:00pm - 06:00am) and thus do not commute or need e-scooters.  7.2 Outliers<jupyter_code># Select columns with numerical data excluding the target variable
ol_cols = cols_numerical.copy()

df_outliers = df_rentals_viz[ol_cols]

cols_count = len(ol_cols)

plt.figure(figsize=(12, 6))

# Generate box plots for all numerical independent variables  
for i in range(0, cols_count):
    
    plt.subplot(1, cols_count, i+1)
    
    sns.set_style('whitegrid')
    
    sns.boxplot(df_outliers[ol_cols[i]], color='green', orient='v')
    
    plt.tight_layout()<jupyter_output><empty_output><jupyter_text>- Independent variables guest_scooter, registered_scooter and windspeed contain large numbers of outliers.  These outliers may be removed later during pre-processing.- Independent variables temperature and feels_like_temperature have similar distribution of values.<jupyter_code># Generate quantile and maximum values of numerical features

ser_quantile_1st = df_outliers.quantile(0.25)
ser_quantile_3rd = df_outliers.quantile(0.75)
ser_iqr = ser_quantile_3rd - ser_quantile_1st
ser_3iqr = ser_iqr * 1.5
ser_max = ser_quantile_3rd + ser_3iqr

df_boxplots = pd.DataFrame({'3rd': ser_quantile_3rd, '1st': ser_quantile_1st, 'iqr': ser_iqr, 
                            '3iqr': ser_3iqr, 'max': ser_max})
df_boxplots
# Get number of outliers in the windspeed, guest_scooter, registered_scooter variables
len(df_outliers[df_outliers.windspeed>32]), len(df_outliers[df_outliers.guest_scooter>346]), len(df_outliers[df_outliers.registered_scooter>3491])<jupyter_output><empty_output><jupyter_text>7.3 Distribution Skewness<jupyter_code>plt.figure(figsize=(15, 6))

# Generate distribution plots for all numerical independent variables  
for i in range(0, cols_count):
    
    plt.subplot(1, cols_count, i+1)
    
    sns.distplot(df_outliers[ol_cols[i]], kde=True)<jupyter_output><empty_output><jupyter_text>- Independent variables guest_scooter, registered_scooter and windspeed are right/positively skewed.- Distribution plots of variables temperature, feels_like_temperature and relative_humidity are similar.  7.4 Bar Charts<jupyter_code># Show distribution of users across weather conditions
df_users = df_rentals[['weather', 'active_scooter']]

df_users.groupby(['weather']).sum().sort_values(by='active_scooter', ascending=False).plot(kind='bar', figsize=(10,5))

plt.ylabel('Active Users')
plt.xlabel('Weather');<jupyter_output><empty_output><jupyter_text>- There is an overwhelming number of users when weather was clear compared to other conditions.<jupyter_code># Show distribution of users across days in a week
df_users = df_rentals[['day_of_wk', 'active_scooter']]

df_users.groupby(['day_of_wk']).sum().sort_values(by='active_scooter', ascending=False).plot(kind='bar', figsize=(10,5))

plt.ylabel('Active Users')
plt.xlabel('Day of Week');<jupyter_output><empty_output><jupyter_text>- The number of users across work days (Mon-Fri) seem to be rather consistent. On weekends (Sat-Sun), there is a slight drop.<jupyter_code># Show distribution of users across days in a week
df_users = df_rentals[['hr', 'active_scooter']]

df_users.groupby(['hr']).sum().sort_values(by='active_scooter', ascending=False).plot(kind='bar', figsize=(10,5))

plt.ylabel('Active Users')
plt.xlabel('Hour of Day');<jupyter_output><empty_output><jupyter_text>- There are most number of users during the morning (08:00am) and evening (16:00pm - 19:00pm) commute, probably to and from work.- There are the least number of users after midnight into the early hours of the morning (01:00am - 05:00am), probably because people are resting at home.- This is in line with findings from Section 7.1 [Correlation](correlation).  8. Feature Selection<jupyter_code>from sklearn.feature_selection import SelectKBest
from sklearn.feature_selection import f_regression

cols_select = cols_all.copy()

cols_select.remove('active_scooter')

X_select = df_rentals_1hot[cols_select]
y_select = df_rentals_1hot['active_scooter']

# Select the best 20 features based on univariate regression f-value 
kbest = SelectKBest(score_func=f_regression, k=20)

fit_select = kbest.fit(X_select, y_select)

df_scores = pd.DataFrame(fit_select.scores_)
df_columns = pd.DataFrame(X_select.columns)

scores = pd.concat([df_columns, df_scores], axis=1)

scores.columns = ['Column','Score']

# Print top 20 features with the highest scores
print(scores.nlargest(20, 'Score'))<jupyter_output>Column         Score
1        registered_scooter  1.342875e+06
0             guest_scooter  9.516842e+03
2               temperature  2.696592e+03
3    feels_like_temperature  2.641487e+03
27                    hr_17  1.992050e+03
4         relative_humidity  1.742026e+03
28                    hr_18  1.476275e+03
40                     hr_8  9.374951e+02
36                     hr_4  7.417971e+02
35                     hr_3  7.048507e+02
37                     hr_5  6.486411e+02
30                     hr_2  6.258045e+02
19                     hr_1  5.594278e+02
18                     hr_0  4.176335e+02
29                    hr_19  3.758596e+02
26                    hr_16  2.878321e+02
38                     hr_6  2.553515e+02
10  weather_light snow/rain  2.478676e+02
34                    hr_23  2.300845e+02
7             weather_clear  1.946128e+02<jupyter_text>- Comparing the top 20 features from SelectKBest (above) and the most correlated (positively/negatively) features from Section 7.1 [Correlation](correlation), the following are the common features:        - registered_scooter        - guest_scooter        - temperature        - feels_like_temperature        - relative_humidity        - weather_light snow/rain        - hr_0, hr_1, hr_2, hr_3, hr_4, hr_5, hr_8, hr_16, hr_17, hr_18, hr_19, hr_23                - As mentioned in Section 7.1 [Correlation](correlation), temperature and feels_like_temperature are highly correlated. feels_like_temperature is a heat index, calculated from temperature and relative_humidity (refer to Section 5.4 [temperature, feels_like_temperature Columns](temperature_feels_like_temperature_columns)). As such, I will drop the feature feels_like_temperature as a predictor of active_scooter.- From Section 7.4 [Bar Charts](bar_charts), a large proportion of active users rented e-scooters when weather conditions were clear.  As such, I will include the feature weather_clear.- The feature day_of_wk_Sunday is amongst one of the top 20 scores from SelectKBest.  Also from Section 7.4 [Bar Charts](bar_charts), there seems to be less users on e-scooters on Sundays.  As such, I will include the feature day_of_wk_Sunday.- Below is the list of 19 selected features:        - registered_scooter        - guest_scooter        - temperature        - relative_humidity        - weather_clear        - weather_light snow/rain        - day_of_wk_Sunday        - hr_0, hr_1, hr_2, hr_3, hr_4, hr_5, hr_8, hr_16, hr_17, hr_18, hr_19, hr23<jupyter_code>cols_selected = ['active_scooter', 'registered_scooter', 'guest_scooter', 'temperature', 'relative_humidity', 
                 'weather_clear', 'weather_light snow/rain', 
                 'day_of_wk_Sunday', 
                 'hr_0', 'hr_1', 'hr_2', 'hr_3', 'hr_4', 'hr_5', 'hr_8', 'hr_16', 'hr_17', 'hr_18', 'hr_19', 'hr_23']


df_selected = df_rentals_1hot[cols_selected]<jupyter_output><empty_output><jupyter_text>9. Feature Pre-processing 9.1 Remove Outliers- From Section 7.2 [Outliers](outliers), independent variables guest_scooter, registered_scooter and windspeed contain large numbers of outliers.- I shall remove these outliers, taking reference from their box plots.<jupyter_code>len(df_selected)<jupyter_output><empty_output><jupyter_text>- Before removing outliers, we have 17,009 observations.<jupyter_code># Remove outliers from registered_scooter and guest_scooter base on their maximum values in the box plots
df_selected = df_selected[df_selected.registered_scooter<=3491]
df_selected = df_selected[df_selected.guest_scooter<=346]

len(df_selected)<jupyter_output><empty_output><jupyter_text>- After removing outliers, I'm left with 15,240 observations.  That's about a 10% reduction of observations.<jupyter_code>ol_cols = ['registered_scooter', 'guest_scooter', 'temperature', 'relative_humidity']

df_outliers = df_selected[ol_cols]

cols_count = len(ol_cols)

plt.figure(figsize=(12, 6))

# Generate box plots for all numerical independent variables  
for i in range(0, cols_count):
    
    plt.subplot(1, cols_count, i+1)
    
    sns.set_style('whitegrid')
    
    sns.boxplot(df_outliers[ol_cols[i]], color='green', orient='v')
    
    plt.tight_layout()<jupyter_output><empty_output><jupyter_text>- There still exist outliers in the variables guest_scooter and registered_scooter. However, their numbers have been siginificantly reduced as evident from the length of trailing dots.  8.2 Scaling - As mention in Section 4. [Summary Statistics](summary_statistics), there is a need to perform scaling to the features due to the difference in the range of values. - Many machine learning algorithms perform better or converge faster when features are on a relatively similar scale and/or close to normally distributed.- To cater for algorithms that require close to 0 mean and unit variance, I've decided to use standard scaling.<jupyter_code>cols_X = df_selected.columns.to_list()

cols_X.remove('active_scooter')
#cols_X.remove('registered_scooter')

# Independent variables
X = df_selected[cols_X] 

# Target/dependent variable
y = df_selected['active_scooter']
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split

# Split dataset into train and test subsets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=19)
# Compare number of observations in train, test and original datasets
len(X_train), len(X_test), len(df_selected)
std_scaler = StandardScaler()

# Standard scale the independent variables of the train dataset
df_X_train_ss = std_scaler.fit_transform(X_train)
df_X_train_ss = pd.DataFrame(df_X_train_ss, columns=X_train.columns)

# Standard scale the independent variables of the test dataset
df_X_test_ss = std_scaler.fit_transform(X_test)
df_X_test_ss = pd.DataFrame(df_X_test_ss, columns=X_train.columns)

fig, (ax1) = plt.subplots(ncols=1, figsize=(10, 8))
ax1.set_title('After StandardScaler')

sns.kdeplot(df_X_train_ss['guest_scooter'], ax=ax1)
sns.kdeplot(df_X_train_ss['registered_scooter'], ax=ax1)
sns.kdeplot(df_X_train_ss['temperature'], ax=ax1)
sns.kdeplot(df_X_train_ss['relative_humidity'], ax=ax1)
#sns.kdeplot(df_X_train_ss['weather_clear'], ax=ax1);
#sns.kdeplot(df_X_train_ss['day_of_wk_Sunday'], ax=ax1);
#sns.kdeplot(df_X_train_ss['hr_0'], ax=ax1);<jupyter_output><empty_output><jupyter_text>9. Modelling 9.1 Multi Linear Regression<jupyter_code>from sklearn.linear_model import LinearRegression
from sklearn import metrics

lr = LinearRegression()  
lr.fit(df_X_train_ss, y_train)
# Show coefficients regression model
df_coef = pd.DataFrame(lr.coef_, df_X_train_ss.columns, columns=['Coefficient'])  
# Predict active users using test dataset
y_pred = lr.predict(df_X_test_ss)

# Compare prediction against actual active users
df_compare = pd.DataFrame({'Actual': y_test, 'Predicted': y_pred})

df_compare.head(10)
df_compare = df_compare.head(50)

df_compare.plot(kind='bar',figsize=(18,10))

plt.grid(which='major', linestyle='-', linewidth='0.5', color='green')
plt.grid(which='minor', linestyle=':', linewidth='0.5', color='black')
plt.show()
print('Mean Absolute Error:', metrics.mean_absolute_error(y_test, y_pred))  
print('Mean Squared Error:', metrics.mean_squared_error(y_test, y_pred))  
print('Root Mean Squared Error:', np.sqrt(metrics.mean_squared_error(y_test, y_pred)))<jupyter_output>Mean Absolute Error: 3.7980859212990157
Mean Squared Error: 21.24319797301177
Root Mean Squared Error: 4.6090343861824
{'fraction_non_alphanumeric': 0.08477068717257437, 'fraction_numerical': 0.02876750099568028, 'mean_word_length': 5.361722861040732, 'pattern_counts': {'":': 0, '<': 171, '<?xml version=': 0, '>': 168, 'https://': 9, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '41', 'dir': 'jupyter-structured-clean-dedup', 'id': '455922', 'license': 'MIT', 'n_tokens_mistral': 10930, 'n_tokens_neox': 9868, 'n_words': 3273, 'path': 'eda.ipynb', 'repo_name': 'hakngrow/aice', 'size': '32641'}