<jupyter_start><jupyter_text>Exploratory Analysis of Boris Bike trip data **Import Libraries and Data and do quick inspection**<jupyter_code>import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
import seaborn as sns

sns.set_style("darkgrid")
%pylab inline
data = pd.read_hdf('ride_data.hdf', 'df', encoding = 'utf-8')
data.head()
print data.dtypes
print data.shape<jupyter_output>start_time     datetime64[ns]
rental_id             float64
bike_id               float64
start_id               object
end_id                float64
end_time       datetime64[ns]
duration      timedelta64[ns]
dtype: object
(37266542, 7)<jupyter_text>**Filter out some records with obviously oncorrect dates and set index on start_time**<jupyter_code>data = data[data.start_time >= '2010-01-01']
data = data[data.end_time >= '2010-01-01']
data = data.set_index('start_time')
data = data.sort_index()<jupyter_output><empty_output><jupyter_text>**Function to help plotting**<jupyter_code>def plot_rides(type = 'd', start = '2012-01-01', end = '2017-01-01', title = None):
    
    grouping = {'d':'Day', 'h':'Hour', 'hm':'Hour / Minute', 'dh':'Day / Hour', 'dhm':'Day / Hour / Minute'}
    
    plt.figure(figsize=(15, 5))
    plot_data = data.ix[start:end]
    if type == 'd':
        plot_data.groupby(plot_data.index.date).count()['rental_id'].plot()
    elif type == 'h':
        plot_data.groupby(plot_data.index.hour).count()['rental_id'].plot()
    elif type == 'hm':
        plot_data.groupby([plot_data.index.hour, plot_data.index.minute]).count()['rental_id'].plot()
    elif type == 'dh':
        plot_data.groupby([plot_data.index.date, plot_data.index.hour]).count()['rental_id'].plot()
    elif type == 'dhm':
        plot_data.groupby([plot_data.index.date, plot_data.index.hour, plot_data.index.minute]).count()['rental_id'].plot()
    else:
        print 'incorrect grouping passed!'
    if title == None:
        title = 'Rides by ' + grouping[type] + ' ' + start + ' to ' + end 
        plt.suptitle(title, fontsize=20)
    else:
        plt.suptitle(title, fontsize=20)
plot_rides(title = 'Rides by Day')<jupyter_output><empty_output><jupyter_text>Find the biggest spike. Turns out that it's not bad data but the day of a [Tube Strike](http://www.standard.co.uk/news/london/boris-bike-use-at-twice-normal-level-as-londoners-seek-to-avoid-the-tube-strike-10378279.html)<jupyter_code>by_date = data.groupby(data.index.date).count()['rental_id']
by_date[by_date == by_date.max()]<jupyter_output><empty_output><jupyter_text>**Look at some of the low values, below are the 20 lowest recorded**<jupyter_code>by_date.sort_values().head(n = 20)<jupyter_output><empty_output><jupyter_text>The 25th of August 2014 sticks out as a bit of an obvious day to investigate further given that most of the others seem to be around Christmas whereas over the Summer period, you would expect usage to be more consistent.<jupyter_code>plot_rides('hm', '2014-08-25','2014-08-25')<jupyter_output><empty_output><jupyter_text>This pattern is clearly different to the prior week's data both in shape and magnitude! A quick check of the calendar reveals the reason - [Bank Holiday Monday](https://en.wikipedia.org/wiki/Bank_holiday)<jupyter_code>plot_rides('hm', '2014-08-18', '2014-08-18')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1404833836858006, 'fraction_numerical': 0.033534743202416915, 'mean_word_length': 4.490878938640133, 'pattern_counts': {'":': 0, '<': 28, '<?xml version=': 0, '>': 30, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '617575', 'license': 'MIT', 'n_tokens_mistral': 1231, 'n_tokens_neox': 1141, 'n_words': 256, 'path': 'trip_data_exploration.ipynb', 'repo_name': 'barnettjacob/boris_bikes', 'size': '3310'}