<jupyter_start><jupyter_text>You can set the indices to be different from the natural integer indices using the index_col parameter in read_csv. It's also a good thing to parse dates.Take note of what our data represents. It's an hourly count of how many people cross the east/west bridge.<jupyter_code>data.plot()<jupyter_output><empty_output><jupyter_text>Notice that there might be too many datapoints for this to be digestible. Let's resample this by week instead of hourly. This visualization doesn't really tell us very much other than it seems like the peak number of bikers passing the West Sidewalk seems to be increasing year over year.<jupyter_code>data.resample('W').sum().plot()<jupyter_output><empty_output><jupyter_text>Notice that this resampling gives us a visualization that tells us more about the data than the previous one. This one tells a more interesting story. It's a visualization plotting the weekly number of bikers crossing either sidewalk. It suggests that the number of people passing by the East Sidewalk is decreasing year over year, while the number of people passing by the West Sidewalk is increasing year over year.Note: DataFrame.resample() is for resampling *time series* data. One question you might ask if whether or not there is any sort of growth or decline in the number of riders. We can see this if we look for a rolling sum year over year.<jupyter_code>ax = data.resample('D').sum().rolling(365).sum().plot()
ax.set_ylim(0,None)
# want to make sure that the axis starts at 0 in order to maintain magnitude effects<jupyter_output><empty_output><jupyter_text>.rolling(365) specifies each datapoint to be the sum of the number of riders over the last 365 days.Adding the total line lets you see what the overall trend in riders is. It seems like the number of bikers suddenly dipped between 2017 and 2018, then recovered.<jupyter_code>data.groupby(data.index.time).mean().plot()<jupyter_output><empty_output><jupyter_text>groupby is self-explanatory. It groups the data by a given parameter. In this case it's grouping by time of day. This allows us to see a daily trend of riders?It seems like there are two peaks around 8am and a little after 4:40 pm. The start and end of the workday!Perhaps we want to see some kind of daily trend.<jupyter_code>pivoted = data.pivot_table('Total', index=data.index.time, columns=data.index.date)
pivoted.iloc[:5,:5]<jupyter_output><empty_output><jupyter_text>.pivot_table allows us to put our data into a pivot table so we can manipulate how we're splitting our data. In this case we've turned the rows into the hours of the day and each datapoint corresponds to the number of total riders during that hour of each day of the year.<jupyter_code>pivoted.plot(legend=False, alpha=0.01)<jupyter_output><empty_output><jupyter_text>Looking at this plot, we still see the same 2 peaks, however we see a few days where there isn't really much of a trend. This might be indicative of how the bridge is being used during weekends.The moral of this is probably to make sure you keep an open mind an curiosity about exploring data in whatever way you can imagine. Each of these graphs is only slightly different from the last, but we slowly build out new insight by doing so.<jupyter_code>get_fremont_data?<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.05777374470659407, 'fraction_numerical': 0.008771929824561403, 'mean_word_length': 5.299047619047619, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '124100', 'license': 'MIT', 'n_tokens_mistral': 940, 'n_tokens_neox': 880, 'n_words': 466, 'path': 'Loading and Visualizing Data.ipynb', 'repo_name': 'devinthai/Jupyter-Workflow', 'size': '3306'}