<jupyter_start><jupyter_text>Benson GraphingThis notebook will import pickled Pandas dataframes that contain traffic data from 2018. Eventually it will export these graphs as high quality images for our presentation.<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import pickle as pkl

%matplotlib inline
# Read in edited dataframes representing MTA Traffic
# for 3 months before the gala. Create dataset spanning
# from 2016 to 2018

df_2018 = pd.read_pickle('NewTrafficManhattanDF2018')
df_2017 = pd.read_pickle('NewTrafficManhattanDF2017')
df_2016 = pd.read_pickle('NewTrafficManhattanDF2016')

df_traffic = pd.concat([df_2016, df_2017, df_2018])    
df_traffic.head()
# Create dataframe that contains the daily traffic for every station in the turnstile data
station_daily = df_traffic.groupby(['DATE', 'STATION']).sum()\
.sort_values(['DATE','DAILY_TRAFFIC'], ascending = [False,False])\
.reset_index()
station_daily.head()

#Get the average traffic per station over all days
avg_daily = station_daily.groupby(['STATION']).mean().sort_values(['DAILY_TRAFFIC'], ascending = False)
avg_daily.head(10)

station_daily.groupby(['STATION']).std().sort_values(['DAILY_TRAFFIC'], ascending = False).head(10)
# Calculate average total daily ridership (entries+exits, which we refer to as 'Traffic')
# This is a sanity check.
print(avg_daily['DAILY_TRAFFIC'].sum())

# Determine station names in the dataframe
avg_daily.index<jupyter_output>5952358.299247381<jupyter_text>There are two names to keep track of at this point. 'ts_station_names' is short for turnstile_station_names and as the name indicates, it is the names of the stations that we extracted from the turnstile data. 'station_names' will appear later, and it is the list of names we extracted from the MTA's stations.csv file. We will compare the two lists to determine what stations fall within Silicon Alley.<jupyter_code># Note: Grand central station was not initially appearing in the list.
# We determined that our initial "clean

ts_station_names = sorted(station_daily['STATION'].unique())
fig = plt.figure(figsize = [10,8])
n = 10
plt.bar(avg_daily.index[:n],avg_daily['DAILY_TRAFFIC'][:n]);
plt.xticks(rotation=80);
plt.ylabel('Average Daily Traffic')
plt.xlabel('Stations')
plt.title('Busiest MTA Subway Stations')
plt.tight_layout()

# Export figure as SVG
fig.savefig('./stations_all.svg')<jupyter_output><empty_output><jupyter_text>Adding in stations further filtered by locationAt this point we can rank stations in Manhattan by our ridership statistic, traffic. We will now read in a list of stations that are within the Silicon Alley.<jupyter_code># Get dataframe from Stations.csv data
with open("longer_list.pkl", 'rb') as picklefile: 
    silicon_stations = pkl.load(picklefile)
station_names = pd.Series(sorted([station.upper() for station in silicon_stations['Station_Name'].unique()]))<jupyter_output><empty_output><jupyter_text>We have noticed that certain string edits can be used to force matches between the two lists of stations. For example, removing 'th' from stations with numbers will result in a match. Regular expressions may have been efficient here, however at the time we found it simpler to manipulate the strings as needed. You will also see series converted to lists in order to use the `in` operator.<jupyter_code>station_names = station_names.str.replace('TH ', ' ')
station_names = station_names.str.replace('1ST ', '1 ')
station_names = station_names.str.replace('2ND ', '2 ')
station_names = station_names.str.replace('3RD ', '3 ')
station_names = station_names.str.replace('SQUARE', 'SQ')
station_names = station_names.str.replace('PLACE', 'PL')
# Determine which station names match or do not match
temp = list(station_names)
for station in ts_station_names:
    if station not in temp:
        print(station)<jupyter_output>103 ST
110 ST
116 ST
125 ST
135 ST
145 ST
155 ST
157 ST
168 ST
175 ST
181 ST
190 ST
191 ST
207 ST
215 ST
34 ST-HERALD SQ
34 ST-PENN STA
42 ST-BRYANT PK
5 AV/53 ST
5 AV/59 ST
57 ST-7 AV
72 ST
77 ST
79 ST
8 ST-NYU
86 ST
96 ST
BOWLING GREEN
CATHEDRAL PKWY
DYCKMAN ST
GRD CNTRL-42 ST
INWOOD-207 ST
RECTOR ST
SOUTH FERRY<jupyter_text>We can determine that certain stations from the turnstile data do not belong based on their geography. Those are as follows:  Too far north:* Cathedral Pkwy* Dyckman St* Inwood-207 St* Generally, any street more north than 63rd STToo far south:* Rector st* South Ferry* Bowling GreenShould be included:* 34 ST - Herald Sq* 42 ST - Bryant PKDubious:* 5 AV/53 ST (might be too north)* 5 AV/59 ST (might be too north)* 8 ST-NYU (probably too far east)* 57 ST/7 AV (might be too north)<jupyter_code># To - do: Replace the text in the ts_station_names for the stations above that should have a match
# 34 ST-HERALD SQ, and 42 ST-BRYANT PK

station_names<jupyter_output><empty_output><jupyter_text>The following graph highlights in red the high-volume stations from the turnstile data that are also in the high tech area.<jupyter_code>fig = plt.figure(figsize = [10,8])
anomalies = ['GRD CNTRL-42 ST', '34 ST-PENN STA', '34 ST-HERALD SQ']

n = 10
for idx in range(n):
    if avg_daily.index[idx] in list(station_names):
        shade = 'r'
    elif avg_daily.index[idx] in anomalies:
        shade = 'r'
    else:
        shade = 'b'
    plt.bar(avg_daily.index[idx],avg_daily['DAILY_TRAFFIC'][idx], color=shade)
plt.xticks(rotation=80);
plt.ylabel('Average Daily Traffic')
plt.xlabel('Stations')
plt.title('Busiest MTA Subway Stations')
plt.tight_layout()

# Export figure as SVG
fig.savefig('./stations_filtered.png')
# Create list of high traffic stations from the turnstile data that are in the high tech area
# Notice that 34th St - Herald Square, Grand Central, and Penn stations do not appear
#  because we have not reconciled those string differences.
keepers = []
for station in avg_daily.index:
    if station in list(station_names):
        keepers.append(station)
        
print(keepers)<jupyter_output>['23 ST', '14 ST-UNION SQ', 'TIMES SQ-42 ST', 'FULTON ST', '59 ST', 'CANAL ST', '14 ST', 'CHAMBERS ST', '28 ST', '50 ST', 'WALL ST', '7 AV', '8 AV', 'GRAND ST', '33 ST', '1 AV', '49 ST', '51 ST', 'SPRING ST', '2 AV', 'ASTOR PL', 'PRINCE ST', '57 ST', 'CORTLANDT ST', 'HOUSTON ST', 'EAST BROADWAY', 'BLEECKER ST', '3 AV', '18 ST', 'CITY HALL', 'FRANKLIN ST', '6 AV', 'BROAD ST', 'BOWERY']
{'fraction_non_alphanumeric': 0.10245581104332864, 'fraction_numerical': 0.03535116533708744, 'mean_word_length': 5.148076923076923, 'pattern_counts': {'":': 0, '<': 22, '<?xml version=': 0, '>': 22, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '207313', 'license': 'MIT', 'n_tokens_mistral': 2279, 'n_tokens_neox': 2057, 'n_words': 770, 'path': 'project_1/Benson_Visualization.ipynb', 'repo_name': 'skylerl2/skylerl2.github.io', 'size': '6393'}