<jupyter_start><jupyter_text>I saved a piece of the subway data as a file subway_data.txt. This is a CSV in everything but name, so I used the pd.read_csv to parse it and create the dataframe. I then used the lambda expression to combine the info in DATE and TIME columns to then convert them to datetime usinmg built in pd functions.I created the to_hourly functionm to determine the number of folks passing through turnstiles hourly. It currently has a hiccup of it just parses top to bottom and does not understand the need to adjust for changing turnstiles (since numbers vary greatly). I think we could run some groupby information through the function and have it do what we need it to do. But I am going to save that for tomorrow.<jupyter_code>filename = 'subway_data.txt'
subway2_df = pd.read_csv(filename)
subway2_df.columns = subway2_df.columns.str.strip()
subway2_df['datetime'] = subway2_df[['DATE','TIME']].apply(lambda x: ' '.join(x), axis=1)
subway2_df['datetime'] = pd.to_datetime(subway2_df['datetime'])
subway2_df.info()
def get_4hour(entries_or_exits):
    return entries_or_exits - entries_or_exits.shift(1)
subway2_df['Hourly_Entries'] = get_4hour(subway2_df['ENTRIES'])
subway2_df['Hourly_Exits'] = get_4hour(subway2_df['EXITS'])
subway3_df = subway2_df[(subway2_df['Hourly_Entries'] > 0) & (subway2_df['Hourly_Entries'] < 100000)& (subway2_df['Hourly_Exits'] > 0) & (subway2_df['Hourly_Exits'] < 100000)]
subway3_df[['datetime','Hourly_Entries', 'Hourly_Exits']].sample(50) #this is what clued me in to the error in how the function was parsing ridership.
subway3_df['Hourly_Entries'].max()
subway2_df.info()
subway4 = subway3_df.groupby(['STATION','LINENAME','datetime'])[['Hourly_Entries','Hourly_Exits']].sum()
subway4.to_csv('station_weekly.csv')
MostHourlyExits = subway3_df.nlargest(100,'Hourly_Exits')
MostHourlyEntries = subway3_df.nlargest(100,'Hourly_Entries')
MostHourlyExits.to_csv('Largest_hourly_exits.csv')
subway5 = subway3_df.groupby(['STATION','LINENAME','datetime'])[['Hourly_Entries','Hourly_Exits']].agg('mean')
subway5.reset_index(inplace=True)
MostHourlyExits.
subway = subway3_df[(subway3_df['SCP'] == '02-00-00') & (subway3_df['UNIT'] == 'R051')]
plt.plot(subway['datetime'], subway['Hourly_Entries'])
#this line was from a previous query we ran. I am keeping it for the code aspect
#it lets us query by 2 columns and I don't want to have to recreate it if I don't need to.
# subwaydf[(subwaydf['STATION'] == 'WORLD TRADE CTR') & (subwaydf['SCP'] == '01-05-00')]
(subwaydf
 .groupby(["C/A", "UNIT", "SCP", "STATION", "datetime"])
 .ENTRIES.count()
 .reset_index()
 .sort_values("ENTRIES", ascending=False)).head(5)
mask = ((subwaydf["C/A"] == "N183") & 
(subwaydf["UNIT"] == "R415") & 
(subwaydf["SCP"] == "00-00-00") & 
(subwaydf["STATION"] == "BROAD CHANNEL") &
(subwaydf["datetime"].dt.date == datetime.datetime(2018, 4, 4).date()))
subwaydf[mask].head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14787052810902895, 'fraction_numerical': 0.03202725724020443, 'mean_word_length': 6.808510638297872, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '641121', 'license': 'Apache-2.0', 'n_tokens_mistral': 1124, 'n_tokens_neox': 1056, 'n_words': 268, 'path': '1-Benson/Project 1/Outdated FIles/TEST.ipynb', 'repo_name': 'iamryanmurray/metis', 'size': '2935'}