<jupyter_start><jupyter_text>Goal of this analysis: explore Capital Bikeshare dataset- This notebook explores the dataset provided by Capital Bikeshare and downloaded from the Kaggle website- The goal of the exploration is to get insights into relationships between bike demand and potential predictors- In ols_linear_regression.ipynb, I use these insights to build a simple model based on ordinary least squares linear regression    Import external libraries<jupyter_code>import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Read training data from CSV file<jupyter_code>data = pd.read_csv("Data/train.csv", parse_dates=True, index_col = 0)<jupyter_output><empty_output><jupyter_text>1. Check columns, column types, index and raw data Target variables- The "count" column is the bike rental count, which is the target variable- The "count" column is the sum of both "casual" and "registered" columns. Both of these columns could therefore also represent target columns in separate models- The maximum bicycle count at any single hour is 977 bikes. The median and 75% quantiles are 145 and 284 respectively, indicating a heavy right tail- The minimum bicycle count is 1. In particular, bike count can not be negative Possible predictor variables- Boolean variables: holiday, workingday- Categorical variables: season and weather (possibly ordinal)- Numerical variables: temp, atemp, humidity, windspeed- The index is a timestamp- The dataset has 10886 records Further observations- "atemp" column is the apparent temperature (temperature equivalent perceived by humans caused by the combined effect of air temperature, humidity and wind speed)- "atemp" is likely to be highly correlated with the "temp" column<jupyter_code>data.info()
data.head()
data.tail()
data.describe()<jupyter_output><empty_output><jupyter_text>No missing values in dataset<jupyter_code>data.isna().sum()<jupyter_output><empty_output><jupyter_text>2. Plot bike rental count on various timescales Plot bike rental count vs. time using a 30-day rolling average- Bike rental demand has a seasonal cycle, peaking in the Summer months and with a trough in the Winter months- Bike rental demand has a strong positive annual trend<jupyter_code>data.rolling(window = 24 * 30).mean()['count'].plot()
plt.ylabel("bike count")<jupyter_output><empty_output><jupyter_text>Extract time features from index- Extracting time features makes it easier to group data, and find relationships and new features<jupyter_code>data['yr'] = data.index.year
data['mon'] = data.index.month
data['day'] = data.index.day
data['dayOfWeek'] = data.index.dayofweek
data['qtr'] = data.index.quarter
data['day_name'] = data.index.day_name()
data['hr'] = data.index.hour
data.head(5)<jupyter_output><empty_output><jupyter_text>Plot bike count vs. hour of day for first day of dataset- Bike rental demand has a daily cycle (night time vs. day time)<jupyter_code>data['count'].iloc[:24].plot()
plt.ylabel("bike count")
plt.title("daily cycle")<jupyter_output><empty_output><jupyter_text>Plot average bike count vs. hour of day - On average, bike rental demand has two large peaks in the morning at about 8am and in the evening at about 6pm- Only few bikes are used in the early morning hours- In the evening, bike counts decline sharply after 8 pm<jupyter_code>data.groupby('hr')['count'].mean().plot()
plt.ylabel("bike count")
plt.title("Average bike count vs. hour of day")<jupyter_output><empty_output><jupyter_text>Plot count vs. time for first week of dataset- Bike demand follows a weekly cycle as well- Week days have two sharp rush hour peaks- Saturdays and Sundays have a single broad peak, much lower than the rush hour peaks on week days<jupyter_code>data['count'].iloc[:24*9].plot()
plt.ylabel("bike count")
plt.title("weekly cycle")<jupyter_output><empty_output><jupyter_text>Plot count statistics vs. day of week- Although the days have very different hourly cycles, the mean bicycle count is stable across all days- 25%, 50%, 75% Quantiles are comparable across all days of the week- The maximum count is lower on weekend days (5 and 6 for Saturday and Sunday) than on week days- There are a large number of outliers for all days of the week, as defined by Q3 + 1.5* IQR- A logarithmic transformation of the target count variable takes care of the heavy tail<jupyter_code>data.groupby('dayOfWeek')['count'].mean().plot()
plt.axis(ymin=0, ymax = 250)
plt.title("Average count per week day")
day0 = pd.DataFrame(data.groupby('dayOfWeek')['count']).iloc[0][1]
day1 = pd.DataFrame(data.groupby('dayOfWeek')['count']).iloc[1][1]
day2 = pd.DataFrame(data.groupby('dayOfWeek')['count']).iloc[2][1]
day3 = pd.DataFrame(data.groupby('dayOfWeek')['count']).iloc[3][1]
day4 = pd.DataFrame(data.groupby('dayOfWeek')['count']).iloc[4][1]
day5 = pd.DataFrame(data.groupby('dayOfWeek')['count']).iloc[5][1]
day6 = pd.DataFrame(data.groupby('dayOfWeek')['count']).iloc[6][1]
days = pd.DataFrame(zip(day0, day1, day2, day3, day4, day5, day6))

days.boxplot()
plt.title("Boxplots of count per week day")
plt.xlabel("dayOfWeek")
data['logCount'] = np.log(data['count'])
day0 = pd.DataFrame(data.groupby('dayOfWeek')['logCount']).iloc[0][1]
day1 = pd.DataFrame(data.groupby('dayOfWeek')['logCount']).iloc[1][1]
day2 = pd.DataFrame(data.groupby('dayOfWeek')['logCount']).iloc[2][1]
day3 = pd.DataFrame(data.groupby('dayOfWeek')['logCount']).iloc[3][1]
day4 = pd.DataFrame(data.groupby('dayOfWeek')['logCount']).iloc[4][1]
day5 = pd.DataFrame(data.groupby('dayOfWeek')['logCount']).iloc[5][1]
day6 = pd.DataFrame(data.groupby('dayOfWeek')['logCount']).iloc[6][1]
days = pd.DataFrame(zip(day0, day1, day2, day3, day4, day5, day6))

days.boxplot()
plt.title("Boxplots of log(count) per week day")
plt.xlabel("dayOfWeek")
data.groupby('dayOfWeek')['count'].describe()<jupyter_output><empty_output><jupyter_text>Plot count vs. month of year- January has the lowest average count- June has the highest average count- "Inverted U-shape" of demand during the year<jupyter_code>data['count'].iloc[:24*18].plot() 
plt.ylabel("bike count")
plt.title("Missing data")<jupyter_output><empty_output><jupyter_text>**Data is missing e.g. between 19.01.2011 and 31.01.2011**- Only entire days are missing. Missing days are part of the hold-out test set- The hold-out test set contains data from the 20th day of each month until the end of the same month<jupyter_code>data.iloc[24*17+20:24*19].head()
data.groupby('mon')['count'].mean().plot()
plt.axis(ymin=0, ymax = 280)
plt.ylabel("bike count")
plt.title("mean bike count per month")<jupyter_output><empty_output><jupyter_text>Plot count vs. quarter- Quarterly mean count has similar "Inverted U-shape" as monthly mean data- Quarterly mean count is less granular than monthly mean count<jupyter_code>data.groupby('qtr')['count'].mean().plot()
plt.axis(ymin=0, ymax = 280)
plt.ylabel("bike count")
plt.title("mean bike count per quarter")<jupyter_output><empty_output><jupyter_text>Plot count vs. season- Count by season is very similar to count by quarter, effectively having the same information<jupyter_code>data.groupby('season')['count'].mean().plot()
plt.axis(ymin=0, ymax = 280)
plt.ylabel("bike count")
plt.title("mean bike count per season")<jupyter_output><empty_output><jupyter_text>3. Plot bike rental count vs. weather variables  Plot bike count vs. categorical "weather" column- "weather" contains 4 unique categories- weather types are fairly stable across all seasons- The value of 4 is observed only once in the entire dataset. It should be removed from the training data- Category 1 is by far most prevalent. Category two has less than half the records as category 1. Category 3 represents a small amount of records- Category 1 has the largest mean bicycle count- Category 3 has the lowest mean bicycle count- Relative differences in mean bicycle counts among categories is fairly stable across seasons<jupyter_code>data.weather.unique()<jupyter_output><empty_output><jupyter_text>Check "weather" categories by looking at mean values of weather variables- Categories 1, 2, and 3 differ mostly in mean humidity levels- Category 4 has a much lower mean temperature, but comprises only a single record<jupyter_code>data.groupby("weather")[['count','temp','atemp','humidity','windspeed']].mean()
data.groupby(["season","weather"]).mon.count().unstack()
data.groupby(["season","weather"]).mon.count().unstack().plot.bar()
plt.title("Total number of records by weather type")
data.groupby(["season","weather"])['count'].mean().unstack().plot.bar()
plt.title("mean bicycle count")<jupyter_output><empty_output><jupyter_text>Plot bike count vs. numerical weather columns Temperature- Bike count is positively linear in both "temp" and "atemp" variables for the mid-range- The bike count dips mid-range (29.812, 32.046) for the "atemp" variable- The bike count hits a ceiling and declines for large temperatures for both "temp" and "atemp"- Ideally, large temperatures should be "bucketed off" and mid-range temperatures kept numerical because of their strong linear relationship with bike count. However, linear regression cannot deal with missing values for the numerical variable. Instead, the temperature variable must be transformed either by applying    - higher order polynomial terms    - a simple binning of values and one-hot-encoding    - an inverse 1/x transformation to model the ceiling    - other non-linear transformations<jupyter_code>data['binnedTemp'] = pd.cut(data['temp'], bins=20)
data['binnedATemp'] = pd.cut(data['atemp'], bins=20)
data['binnedHum'] = pd.cut(data['humidity'], bins=20)
data['binnedWind'] = pd.cut(data['windspeed'], bins=10)
data.groupby(['binnedTemp'])['count'].count()
data.groupby(['binnedTemp'])['count'].mean().plot()
plt.ylabel("bike count")
plt.title("mean bicycle count vs. temp")
data.groupby(['binnedATemp'])['count'].count()
data.groupby(['binnedATemp'])['count'].mean().plot()
plt.ylabel("bike count")
plt.title("mean bicycle count vs. atemp")<jupyter_output><empty_output><jupyter_text>Humidity- Bike count is negatively linear in the humidity variable above a threshold of about 15- Similar to temperature, low humidity levels would ideally be "bucketed off" to define a threshold for the linear mid-range humidity levels. This is not possible with linear regression, which cannot deal with missing values for the numerical temperature variable- The same transformations can be applied as with temperature. The most promising are likely    - applying a second order term to approximate the "inverted U-shape"    - binning and one-hot-encoding the humidity values<jupyter_code>data.groupby(['binnedHum'])['count'].count()
data.groupby(['binnedHum'])['count'].mean().plot()
plt.ylabel("bike count")
plt.title("mean bicycle count vs. humidity")<jupyter_output><empty_output><jupyter_text>Wind speed- Bike count follows a general inverted non-linear "U-shape" with respect to the wind speed- For the largest wind speeds, the "inverted U-shape" is interrupted. This largest range contains only 3 records- Aggregating the largest and second to largest wind speed buckets into a single bucket gives a mean of ca. 133 bikes, in accordance with the general "inverted U-shape" pattern<jupyter_code>data.groupby(['binnedWind'])['count'].count()
data.groupby(['binnedWind'])['count'].mean().plot()
plt.ylabel("bike count")
plt.title("mean bicycle count vs. windspeed")
data[data["windspeed"] > 45.5]['count'].mean()<jupyter_output><empty_output><jupyter_text>4. Plot bike rental count vs. holiday/working day- There are more than twice as many working days as weekend days and holidays put together- The mean bicycle count is stable across weekend days, holidays and working days- On working days, rush hour is from 7-8h and 17-18h<jupyter_code>data.groupby(['holiday','workingday'])['count'].count()
data.groupby(['holiday','workingday'])['count'].mean().unstack(0).plot.bar()
plt.title("mean bicycle count per day type")
plt.ylabel("bike count")
data.groupby(['hr','workingday'])['count'].mean().unstack().plot.bar()
plt.title("mean hourly count per non-/ working day")
plt.ylabel("bike count")<jupyter_output><empty_output><jupyter_text>5. Plot bike rental count vs. registered / casual variablesNote: count = registered + casual, i.e. registered and casual variables are not potential predictors, but potential target variables- The majority of bike rentals are by registered users- Registered users rent slightly more on working days- Casual users rent significantly more on non-working days- Target count / registered / casual variables are positively skewed with a heavy right tail- Casual and registered users have different daily rental patterns    - Casual users do not participate in rush hours on working days<jupyter_code>data.groupby(['workingday'])[['casual','registered']].mean()
data[["casual","registered"]].mean().plot.pie(subplots=True)
plt.ylabel("")
data.groupby(['workingday'])[['registered']].mean().plot.pie(subplots=True)
plt.title("working day")
data.groupby(['workingday'])[['casual']].mean().plot.pie(subplots=True)
plt.title("working day")
sns.pairplot(data[['count','registered','casual', 'workingday']], hue = "workingday")
data.groupby(['hr','workingday'])['registered'].mean().unstack().plot.bar()
plt.title("mean hourly count by registered users")
plt.ylabel("bike count")
data.groupby(['hr','workingday'])['casual'].mean().unstack().plot.bar()
plt.title("mean hourly count by casual users")
plt.ylabel("bike count")<jupyter_output><empty_output><jupyter_text>6. Plot pair-wise correlations Pair-wise correlations due to similarity of information- season, quarter and month are strongly positively correlated- temp and atemp are strongly positively correlated- workingday is negatively correlated with holiday and day of week- humidity and weather are positively correlated Pair-wise correlations due to underlying physicality- season, quarter, month and hour are correlated with temp, atemp, humidity and windspeed Pair-wise correlations with target variable- count is correlated with all variables but holiday, workingday, day and dayOfWeek<jupyter_code>data[["holiday","workingday","weather","temp","atemp","humidity","windspeed","count","yr","season","qtr","mon","day","dayOfWeek","hr"]].corr().round(2)
plt.figure(figsize=(15,10))
sns.heatmap(data[["holiday","workingday","weather","temp","atemp","humidity","windspeed","count","yr","season","qtr","mon","day","dayOfWeek","hr"]].corr().round(2))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12550579521294836, 'fraction_numerical': 0.014127974761676154, 'mean_word_length': 6.493319630010277, 'pattern_counts': {'":': 0, '<': 89, '<?xml version=': 0, '>': 90, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '22', 'dir': 'jupyter-structured-clean-dedup', 'id': '375708', 'license': 'MIT', 'n_tokens_mistral': 4520, 'n_tokens_neox': 4218, 'n_words': 1476, 'path': 'data_exploration.ipynb', 'repo_name': 'rjt-weber/Predicting-bike-demand-using-linear-regression', 'size': '14581'}