<jupyter_start><jupyter_text>Ford GoBike Data Exploration by <NAME>In this notebook, I am doing Univariate, Bivariate, Multivariate explorations on the Ford GoBike data in 2018. My purpose is to investigate which variable(s) has the most effect on trip duration of the users.  Table of ContentsPreliminary WranglingUnivariate ExplorationBivariate ExplorationMultivariate ExplorationConclusion      Preliminary Wrangling> This data set includes information about individual rides made in a bike-sharing system covering the greater San Francisco Bay area in 2018<jupyter_code># import all packages and set plots to be embedded inline
from requests import get
from io import BytesIO
from zipfile import ZipFile
from os import listdir 
from datetime import date

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sb

%matplotlib inline
# create an empty folder to store the files
folder_name_of_csvs = 'trip_data_files'
# extract zip files from the url containing data
for month in range(1,13):
    month_string = str(month)
    month_leading_zero = month_string.zfill(2)
    
    bike_data_url = 'https://s3.amazonaws.com/fordgobike-data/2018' + month_leading_zero + '-fordgobike-tripdata.csv.zip'
    response = get(bike_data_url)
        
    # code below opens zip file; BytesIO returns a readable and writeable view of the contents;
    unzipped_file = ZipFile(BytesIO(response.content))
        
    # puts extracted zip file into folder trip_data_files
    unzipped_file.extractall(folder_name_of_csvs)
# combine all locally saved csvs into 1 dataframe
list_csvs = []
for file_name in listdir(folder_name_of_csvs):
    list_csvs.append(pd.read_csv(folder_name_of_csvs+'/'+file_name))
df = pd.concat(list_csvs)
# save df into csv format
df.to_csv('data.csv')
# load in the dataset into a pandas dataframe
rides = pd.read_csv('data.csv', index_col=0)
# high-level overview of data shape and composition
print(rides.shape)
print(rides.dtypes)
rides.head(10)
# remove unncecessary columns that are not relevant to the analysis: start_station_latitude, start_station_longitude,
# end_station_latitude, end_station_longitude, bike_share_for_all_trip
rides = rides.drop(['start_station_latitude', 'start_station_longitude', 'end_station_latitude', 'end_station_longitude', 'bike_share_for_all_trip'], axis=1)      
# convert member_gender column to category
rides['member_gender'] = rides['member_gender'].astype('category')
# test the cleaning steps
print(rides.dtypes)
print(rides.describe())
rides.head(10)<jupyter_output>duration_sec             int64
start_time              object
end_time                object
start_station_id       float64
start_station_name      object
end_station_id         float64
end_station_name        object
bike_id                  int64
user_type               object
member_birth_year      float64
member_gender         category
dtype: object
       duration_sec  start_station_id  end_station_id       bike_id  \
count  1.863721e+06      1.851950e+06    1.851950e+06  1.863721e+06   
mean   8.573026e+02      1.196744e+02    1.181730e+02  2.296851e+03   
std    2.370379e+03      1.003976e+02    1.004403e+02  1.287733e+03   
min    6.100000e+01      3.000000e+00    3.000000e+00  1.100000e+01   
25%    3.500000e+02      3.300000e+01    3.000000e+01  1.225000e+03   
50%    5.560000e+02      8.900000e+01    8.800000e+01  2.338000e+03   
75%    8.720000e+02      1.860000e+02    1.830000e+02  3.333000e+03   
max    8.636600e+04      3.810000e+02    3.810000e+02  6.234000e+03   

     [...]<jupyter_text>What is the structure of your dataset?> There are 1,863,721 ride data from 2018 in the dataset with 11 features (ride duration, start and end time, start station id and name, end station id and name, bike id, customer type, birth year, and gender). Most variables are string. What is/are the main feature(s) of interest in your dataset?> I'm most interested in figuring out what features are best for predicting the duration of the trip. Here are some questions I wish to answer:1. How long does average trip take?2. Does User Type (Subscriber or Customer) affect the duration?3. Do Age and Sex of user affect the duration?4. Do season (time of the year) affect the duration? What features in the dataset do you think will help support your investigation into your feature(s) of interest?> I expect that season will have the strongest effect on each ride's duration: the colder the season (Winter time), the lower the duration. I also think that the other features: user type, age, and sex, will have effects on the duration, though to a much smaller degree than the main effect of season.  Univariate Exploration> I'll start by looking at the distribution of the main variable of interest: duration.<jupyter_code># start with standard-scaled plot
binsize = 100
bins = np.arange(0, rides['duration_sec'].max()+binsize, binsize)

plt.figure(figsize=[8, 5])
plt.hist(data = rides, x = 'duration_sec', bins = bins)
plt.xlim(0, 3000)
plt.xlabel('Trip Duration (sec)')
plt.title('Distribution of Trip Duration')
plt.show();<jupyter_output><empty_output><jupyter_text>The distribution is right-skewed, with a really long tailIt was very difficult to view the plot, I had to reduce the bin size to 100 and put a limit of 3000 on x-axis. I will change the trip duration variable to minute instead of second.<jupyter_code># change trip duration variable from second to minute
rides['duration_min'] = rides['duration_sec'] / 60
rides.drop('duration_sec' , axis = 1 , inplace = True)<jupyter_output><empty_output><jupyter_text>I will plot the distribution of trip duration again, but in minutes and with log scale on the x-axis<jupyter_code># trip duration distribution in log scale
log_binsize = 0.1
bins = 10 ** np.arange(0, np.log10(rides['duration_min'].max())+log_binsize, log_binsize)
ticks =  [ 0.1 , 0.3 , 1 , 3, 10, 30, 100, 300]
labels = ['{}'.format(v) for v in ticks]

plt.figure(figsize=[8, 5])
plt.hist(data = rides, x = 'duration_min', bins = bins)
plt.xscale('log')
plt.xticks(ticks, labels)
plt.xlabel('Trip Duration (min)')
plt.title('Distribution of Trip Duration')
plt.show();<jupyter_output><empty_output><jupyter_text>Trip duration has a long-tailed distribution, with a lot of trips on the low duration end, and few on the high duration end. When plotted on a log-scale, the price distribution looks roughly unimodal, with one peak around 10 minutes.Next up, the first predictor variable of interest: season. I need to extract the months of the ride first and then bin them into seasons (winter, spring, summer, fall)<jupyter_code># extract the month from start_time column
rides['start_month'] = pd.DatetimeIndex(rides['start_time']).month
rides['start_month'].value_counts()
# create 4 seasons category
def season_of_month(month):
    seasons = {'spring': ('3', '4', '5'),
               'summer': ('6', '7', '8'),
               'fall': ('9', '10', '11'),
               'winter': ('12', '1', '2')}
    if month in seasons['spring']:
        return 'spring'
    if month in seasons['summer']:
        return 'summer'
    if month in seasons['fall']:
        return 'fall'
    if month in seasons['winter']:
        return 'winter'

rides['start_month'] = rides['start_month'].astype(str)
rides['season'] = rides.start_month.map(season_of_month)
rides.sample(10)
# convert season column in order
level_order = ['spring', 'summer', 'fall', 'winter']
ordered_season = pd.api.types.CategoricalDtype(ordered = True, categories=level_order)
rides['season'] = rides['season'].astype(ordered_season)

# plot distribution of season
plt.figure(figsize=[8, 5])

base_color = sb.color_palette()[0]
sb.countplot(data = rides, x = 'season', color = base_color)
plt.title('Trips in Four Seasons')
plt.xlabel('Seasons')

plt.show()<jupyter_output><empty_output><jupyter_text>Users ride more in Summer and Fall seasons where the weather is not too cold and as expected, users ride the least in the cold winter. But, does more rides mean more duration? I will look at this relationship in Bivariate Exploration section.I will take a look on the distribution of type of users (subscriber or customer)<jupyter_code># plot distribution of user type
plt.figure(figsize=[8, 5])
base_color = sb.color_palette()[0]
sb.countplot(data = rides, x = 'user_type', color = base_color)

n_points = rides.shape[0]
user_counts = rides['user_type'].value_counts()
locs, labels = plt.xticks()

# print frequency on the bar chart
for loc, label in zip(locs, labels):
    count = user_counts[label.get_text()]
    pct_string = '{:0.1f}%'.format(100*count/n_points)
    plt.text(loc, count-8, pct_string, ha='center', color='b')

plt.title('Trips by User Types')
plt.xlabel('User Type')

plt.show()<jupyter_output><empty_output><jupyter_text>85% of the users in 2018 were Subscriber or Member, while the rest (15%) were Customer or just Casual.The last variables that I want to look at are age and gender of the users.<jupyter_code># calculate age of each user
def calculate_age(born):
    today = date.today()
    return today.year - born

rides['age'] = rides.member_birth_year.map(calculate_age)
# plot distribution of age
plt.figure(figsize = [8, 5])

binsize = 5
bins = np.arange(0, rides['age'].max()+binsize, binsize)

plt.hist(data = rides, x = 'age', bins = bins)
plt.xlim(18, 70)
plt.xlabel('Age')
plt.title('Distribution of User Age')
plt.show()<jupyter_output><empty_output><jupyter_text>The distribution is right-skewed with the peak at between 30 to 35 and as age gets higher, user count decreases.<jupyter_code># plot distribution of gender
plt.figure(figsize=[8, 5])

base_color = sb.color_palette()[0]
gen_order = rides['member_gender'].value_counts().index
sb.countplot(data = rides, x = 'member_gender', color = base_color, order=gen_order)
plt.title('Distribution of User Gender')
plt.xlabel('Gender')

# print frequency on the bar chart
n_points = rides.shape[0]
gender_counts = rides['member_gender'].value_counts()
locs, labels = plt.xticks()

for loc, label in zip(locs, labels):
    count = gender_counts[label.get_text()]
    pct_string = '{:0.1f}%'.format(100*count/n_points)
    plt.text(loc, count-8, pct_string, ha='center', color='b')

plt.show()<jupyter_output><empty_output><jupyter_text>Almost 70% of the users are male and 23% of the users are female.  Discuss the distribution(s) of your variable(s) of interest. Were there any unusual points? Did you need to perform any transformations?> The duration variable took on a large range of values, so I looked at the data using a log transform. Under the transformation, the data looked unimodal, with one peak around 10 minutes.> The other variables (season, user type, age, and gender) were as I expected with male subscriber age 30-35 in summer as the highest count from these variables. Of the features you investigated, were there any unusual distributions? Did you perform any operations on the data to tidy, adjust, or change the form of the data? If so, why did you do this?> I did change the duration variable from seconds to minutes because seconds are just too unreasonable to measure a trip duration. I also engineered new variables: season and age from existing variabes.  Bivariate Exploration> To start off with, I want to look at how trip duration is affected by month and season<jupyter_code># convert month column back to integer
rides['start_month'] = rides['start_month'].astype(int)

# check the trip duration over each month.
plt.figure(figsize = [12,12])
plt.scatter(data = rides, x = 'start_month' , y = 'duration_min' , alpha = 0.1)
tick_labels = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]
plt.xticks(tick_labels, tick_labels)
plt.title('The Trip Duration Trend over Each Month ')
plt.xlabel('Month')
plt.ylabel('Trip Duration(min)');
# check the trip duration over each season
spring_dur = rides.query("season == 'spring'").duration_min.mean ()
summer_dur = rides.query("season == 'summer'").duration_min.mean()
fall_dur = rides.query("season == 'fall'").duration_min.mean()
winter_dur = rides.query("season == 'winter'").duration_min.mean()

plt.bar(['spring','summer','fall','winter'],[spring_dur , summer_dur , fall_dur , winter_dur] )
plt.title('Duration against Four Seasons ')
plt.xlabel('Seasons')
plt.ylabel('Trip Duration (min)');<jupyter_output><empty_output><jupyter_text>From the month vs duration scatter plot, June to August of 2018 has the most duration, while November and December of 2018 has the least duration, as expected.From the season vs duration bar plot, again, I saw that Summer has the longest trip duration , with average above 14 mins. However, surprisingly, Fall has the lowest trip duration, even lower than Winter, even though Fall season usually has the best weather in SF. After checking the weather data in Fall 2018, I remembered about the Wildfires in Northern California that happened in Nov 2018 which caused low air quality in SF, thus this might cause a decrease in users riding GoBike.Other than that, there is no significant difference of trip duration in between the four seasons. It is probably because the weather in San Francisco bay area does not fluctuate to the extremes (snow or heat wave).<jupyter_code># scatter plot of duration vs. age
plt.figure(figsize = [8, 6])
plt.scatter(data = rides, x = 'age', y = 'duration_min', alpha = 1/10)
plt.xlabel('Age')
plt.xlim(18, 60)
plt.ylabel('Trip Duration (min)')
plt.title('Duration Against User Age')
plt.show()<jupyter_output><empty_output><jupyter_text>Users with age of 31 and 36 seem to have the highest trip duration and as the users get older, the trip duration becomes less.<jupyter_code># query the data for duration only below 1 hour to focus the data
rides_dur = rides.query("duration_min < 60")

# plot relationship between gender and duration
base_color = sb.color_palette()[0]
sb.violinplot(data=rides_dur, x='member_gender', y='duration_min', color=base_color, inner='quartile')
plt.xlabel('Gender')
plt.ylabel('Duration (min)')
plt.title('Duration Against User Gender');<jupyter_output><empty_output><jupyter_text>Although in previous section it was shown that 70% of the users were male, but the trip duration does not seem to be affected by the user gender. The violin plots look similar for these 3 gender categories, with the median all similar (around 10 minutes).<jupyter_code># plot relationship between user type and duration
base_color = sb.color_palette()[0]
sb.violinplot(data=rides_dur, x='user_type', y='duration_min', color=base_color, inner='quartile')
plt.xlabel('User Type')
plt.ylabel('Duration (min)')
plt.title('Duration Against User Type');<jupyter_output><empty_output><jupyter_text>Subscriber users tend to ride the bikes in lesser duration than customer users. This might be because the subscriber users ride the bikes for regular commutes (work, school, etc.), but the customer users might be tourists that use the bikes for traveling around town.<jupyter_code># categorize the variables into number and categorical for a plot matrix
numeric_vars = ['duration_min', 'age']
categoric_vars = ['season', 'member_gender', 'user_type']
# plot matrix of numeric features against categorical features.
# can use a larger sample since there are fewer plots and they're simpler in nature.

samples = np.random.choice(rides_dur.shape[0], 2000, replace = False)
rides_samp = rides_dur.loc[samples,:]

def boxgrid(x, y, **kwargs):
    """ Quick hack for creating box plots with seaborn's PairGrid. """
    default_color = sb.color_palette()[0]
    sb.boxplot(x, y, color = default_color)

plt.figure(figsize = [10, 10])
g = sb.PairGrid(data = rides_samp, y_vars = numeric_vars, x_vars = categoric_vars,
                size = 3, aspect = 1.5)
g.map(boxgrid)
plt.show();<jupyter_output>/anaconda3/lib/python3.7/site-packages/pandas/core/indexing.py:1472: FutureWarning: 
Passing list-likes to .loc or [] with any missing label will raise
KeyError in the future, you can use .reindex() as an alternative.

See the documentation here:
https://pandas.pydata.org/pandas-docs/stable/indexing.html#deprecate-loc-reindex-listlike
  return self._getitem_tuple(key)
/anaconda3/lib/python3.7/site-packages/seaborn/axisgrid.py:1241: UserWarning: The `size` paramter has been renamed to `height`; please update your code.
  warnings.warn(UserWarning(msg))<jupyter_text>Talk about some of the relationships you observed in this part of the investigation. How did the feature(s) of interest vary with other features in the dataset?> Duration had a surprisingly low amount of correlation with the season. There is no significant difference between the duration in each season. This might be caused by the 'constant' temperature in SF area, unlike other cities with extreme temperature changes (for example New York). I also suspected that due to high frequency of subscriber users compared to customer, the riders were all regular users who need to use the bikes for their commutes regardless of the weather. > There was also an interesting relationship observed between duration and the categorical features. For genders, the trip duration does not seem to differ between Female, Male, and Other. For user type, Customer type has more trip duration than Subscriber type, and this is caused by the regularities in Subscriber trips, while Customer trips might be more random. Did you observe any interesting relationships between the other features (not the main feature(s) of interest)?> There is not interesting relationship between other feature. The relationship between user age and the categorial variables do not seem to have any significant difference.  Multivariate Exploration> I want to investigate on the relationship between the trip duration and the age and gender of the users.> Age from 18 to 56 takes 95% of the users. So, it's logical to remove users more than 60 years old. There were users more than 100 years old. I searched at Ford sign in webpage and realized they allow users to put some unusual birth year, such as 1918.<jupyter_code># remove users more than 60 years old
rides_age = rides_dur[rides_dur['age']<=60]

# generate a new field for member age group
rides_age['member_age_bins'] = rides_age['age'].apply(lambda x: '10 - 20' if 10<x<=20
                                                  else '20 - 30' if 20<x<=30
                                                  else '30 - 40' if 30<x<=40
                                                  else '40 - 50' if 40<x<=50
                                                  else '50 - 60' if 50<x<=60
                                                  else x)
# create bar plot on gender versus duration on levels of the age bins variable
plt.figure(figsize = [8, 8])

ax = sb.barplot(x='member_gender', y = 'duration_min', hue='member_age_bins', data=rides_age.sort_values(by= 'member_age_bins'))
plt.title('Trip Duration Across Gender and Age Group')
plt.xlabel('Gender')
plt.ylabel('Duration (min)')
leg = ax.legend()<jupyter_output>/anaconda3/lib/python3.7/site-packages/scipy/stats/stats.py:1713: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  return np.add.reduce(sorted[indexer] * weights, axis=axis) / sumval<jupyter_text>Talk about some of the relationships you observed in this part of the investigation. Were there features that strengthened each other in terms of looking at your feature(s) of interest?> In all gender types, the age range of 10-20 has the longest trip duration, this is maybe due to the young age still has more stamina, so they tend to ride the bikes longer either for recreation or for school. Were there any interesting or surprising interactions between features?> My biggest surprise was that the second longest trip duration in female and male actually belongs to the 50-60 age group. I would have thought that Users with age 20-30 would have the longest duration. My assumption is maybe the Age Group of 50-60 would ride in slower speed than 20-30 year olds and would need longer duration to reach their destinations.  Conclusion1. In 2018, the average trip duration that the users took was around 10 minutes. The trips were mostly in Summer season (between June - August) and most trips were taken by males. 85% of the total trips were taken by Subscribers, which are members of the GoBike program. Also, the age group with the most trips was between 30-35 years old.2. There was no significant difference between the seasons and the trip duration, but there was a low durations in Fall season due to the bad air quality caused by Northern California wildfires in November 2018.3. The Customers users (casual users) usually took more time in their rides than Subscribers.4. All gender types (male, female, other) with age range of 10-20 took the longest trip duration in average and surprisingly, in the male and female categories, age range of 50-60 took the second longest trip duration.<jupyter_code># save all changes made to a new csv
rides.to_csv('rides_clean.csv', 
                 index=False, encoding = 'utf-8')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08921181339352897, 'fraction_numerical': 0.03150865312264861, 'mean_word_length': 4.523376623376623, 'pattern_counts': {'":': 0, '<': 71, '<?xml version=': 0, '>': 75, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '321526', 'license': 'MIT', 'n_tokens_mistral': 6769, 'n_tokens_neox': 6119, 'n_words': 2518, 'path': 'gobike_exploration.ipynb', 'repo_name': 'nyuliani/Ford-GoBike-Data-Analysis', 'size': '21274'}