<jupyter_start><jupyter_text>Traffic Data ViewerA Jupyter Notebook for visualizing WSDOT traffic dataNote: The data is stored assuming UTC timezone, however most of these functions take the parameter `local_tz`, allowing you to convert to the timezone of your choice. The default value of `local_tz` is US/Pacific.<jupyter_code>%matplotlib notebook
import sys
sys.path.append('../')
import pandas
from scipy import stats
import numpy as np
import seaborn as sns
from matplotlib import pyplot as plt
import traffic.dbutil as dbutil
name_table = dbutil.get_nameidtable()
d = dbutil.get_travel_times([96])
route_name = d['Description'].any()
print(route_name)
def weekdayName(i):
    return ["Monday", "Tuesday", "Wednesday", "Thursday", "Friday"][i]<jupyter_output><empty_output><jupyter_text>FilteringFilter to weekdays only, in the hours 3pm-7pm, to look at the PM rush hour commute.Then we take the mean of the samples in that hour band (eg: the travel times at 4:15, 4:35, 4:55 are averaged and stored in the 4pm row for a particular date). We do this because we want to look at the entire hour as a unit and we know that the 3 samples taken during a single hour are not independent. In other words: bad traffic at 4:15pm probably implies bad traffic at 4:35pm as well. Taking the average gives us a single point for each hour.<jupyter_code>d1 = d.weekdays()
d1 = d1.hours_filter(15, 18)
d1['Hour'] = d1.apply(lambda row: row['TimeUpdated'].hour, axis=1)
d1['Weekday'] = d1.apply(lambda row: weekdayName(row['TimeUpdated'].weekday()), axis=1)
d1['Date'] = d1.apply(lambda row: row['TimeUpdated'].date(), axis=1)
tidy = d1[['Date', 'Weekday','Hour', 'CurrentTime']]
tidy = tidy.rename(columns={'CurrentTime': 'TravelTimeMinutes'})
tidy_mean = tidy.groupby(['Date', 'Weekday', 'Hour']).mean().reset_index()<jupyter_output><empty_output><jupyter_text>Plotting BoxplotThe boxplot allows us to easily compare between hours in a weekday and between weekdays. Whiskers represent the 1st and 99th percentiles.<jupyter_code>sns.boxplot(x="Weekday", y="TravelTimeMinutes", hue="Hour", data=tidy_mean, whis=[1, 99]).set_title(route_name)
#g = sns.FacetGrid(tidy_mean, col="Weekday", row="Hour", margin_titles=True)
#g.map(plt.hist, "TravelTimeMinutes", normed=True, cumulative=True)
#g2 = sns.FacetGrid(tidy_mean, col="Weekday", row="Hour", margin_titles=True)
#g2.map(plt.hist, "TravelTimeMinutes", normed=True)<jupyter_output><empty_output><jupyter_text>Next Steps1. Filter outliers > 2.5 stdevs2. Exclude Christmas/Thanksgiving weeks3. Boxplot of filtered data<jupyter_code># GroupBy - Calculate Mean
m = tidy_mean.groupby(['Weekday', 'Hour']).mean().rename(columns={'TravelTimeMinutes': 'MeanTime'}).reset_index()
# Join tidy_mean with GroupBy Mean
merged = pandas.merge(tidy_mean, m)
# GroupBy - Calculate Std
m2 = merged.groupby(['Weekday', 'Hour']).std().rename(columns={'TravelTimeMinutes': 'Std'}).reset_index()
m2 = m2[['Weekday','Hour', 'Std']]
merged2 = pandas.merge(merged, m2)
zscored = merged2
# Keep that data which is fewer than 3 standard deviations from the mean
zscored['ZScore'] = merged2.apply(lambda row: (row['TravelTimeMinutes'] - row['MeanTime'])/row['Std'], axis=1)
no_outliers = zscored[np.abs(zscored['ZScore']) < 3]
sns.boxplot(x="Weekday", y="TravelTimeMinutes", hue="Hour", data=no_outliers, whis=[1, 99]).set_title(route_name)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13550295857988165, 'fraction_numerical': 0.01893491124260355, 'mean_word_length': 6.899532710280374, 'pattern_counts': {'":': 0, '<': 18, '<?xml version=': 0, '>': 18, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '442515', 'license': 'MIT', 'n_tokens_mistral': 1203, 'n_tokens_neox': 1136, 'n_words': 332, 'path': 'src/python/jupyter/TrafficDataViewer.ipynb', 'repo_name': 'tblanarik/WSDOTTrafficData', 'size': '3380'}