<jupyter_start><jupyter_text>Import Stock Trades by Senators Using senatestockwatcher.com Author: <NAME><jupyter_code>import pandas as pd
#https://docs.google.com/spreadsheets/d/1lH_LpTgRlfzKvpRnWYgoxlkWvJj0v1r3zN3CeWMAgqI/edit?usp=sharing
try:
    sen_df = pd.read_csv("Senator Stock Trades/Senate Stock Watcher 04_16_2020 All Transactions.csv")
except:
    sen_df = pd.read_csv("https://github.com/pkm29/big_data_final_project3/raw/master/Senate%20Stock%20Trades/Senate%20Stock%20Watcher%2004_16_2020%20All%20Transactions.csv")
sen_df.head()
#number of rows
print("Number of Observations: " + str(len(sen_df.index)))

from datetime import datetime
from datetime import timedelta
from collections import defaultdict

sen_df['transaction_date'] = pd.to_datetime(sen_df['transaction_date'], format = '%m/%d/%Y')

print("Most Recent Transaction Date: " + str(sen_df['transaction_date'].max().date()))
print("Oldest Transaction Date: " + str(sen_df['transaction_date'].min().date()))<jupyter_output>Number of Observations: 8600
Most Recent Transaction Date: 2020-04-02
Oldest Transaction Date: 2012-06-14<jupyter_text>Initial Visualization of Senator's Stock Trades  Trade Amount Visualization<jupyter_code>#Count number of occurances of each type of amount
amount_table = sen_df['amount'].value_counts()
print("Trade volume based on amount per transaction")
print(amount_table)
#Histogram of amounts
import matplotlib.pyplot as plt
sen_df['amount'].value_counts().plot(kind='barh')
plt.title("Trade Amounts vs # of Transactions")<jupyter_output><empty_output><jupyter_text>There is a disproportionate amount of stock trades that take place within the 1k-15k range, with only a few in the million dollars range. Therefore:  1. It is necessary to factor the trade amount into our analysis, as a single trade can be larger than 100 small trades. 2. Theoretically, we could utilize this logarithmic histogram trend to estimate an average trade size for each bucket rather than simply use the middle of the bucket. For times sake, we will use the range to estimate profit.  Transaction Date VisualizationWe can also quickly view the nature of transaction dates.<jupyter_code>#histogram for days of the week traded
sen_df["transaction_date"].groupby(sen_df["transaction_date"].dt.day).count().plot(kind = "bar")
plt.title("Transaction Volumes for Each Day of the Month")<jupyter_output><empty_output><jupyter_text>An interesting cyclical trend is observed on different days of the month and while not obviously useful for our analysis, this could prove useful later on.  Next we will visualize senator trade volumes over all time.<jupyter_code>#plot frequency of trading dates over time
#count number of trades for each day, then set that date to the index 
trade_freq_df1 = sen_df["transaction_date"].value_counts().rename_axis('Date').reset_index(name='Count')
pd.to_datetime(sen_df['transaction_date'], format = '%Y-%m-%d')
trade_freq_df1.set_index('Date', inplace=True)
#Add rows for days when there weren't any trades
max_date = str(sen_df['transaction_date'].max().date())
min_date = str(sen_df['transaction_date'].min().date())
trade_freq_df = trade_freq_df1.reindex(pd.date_range(min_date, max_date), fill_value="0")
trade_freq_df.index.name = "Date"
trade_freq_df['Count'] = trade_freq_df['Count'].astype(int)
trade_freq_df.Count.plot(title = "Number of Trades by Date w/ Zero Trade Days")
plt.ylabel('Count')
plt.show()<jupyter_output><empty_output><jupyter_text>Using this graph we can observe there are several dates in which an usually high number of trades were placed in one day. These days could indicate suspicious activity due to insider information being released. On the other hand they could also be days when impactful news was released or other non-insider sources. It may be useful to examine these trade days to see if there is a a buildup or some other unusual activity.<jupyter_code>from datetime import datetime, timedelta
def graph_range(center, freq_df = trade_freq_df, width = 10):
    glanceback = timedelta(days= width)
    low = center - glanceback
    high = center + glanceback
    range_df = freq_df[low:high]
    
    range_df.Count.plot(title = str(center.strftime("%B, %d, %Y")))
    plt.ylabel('Count')
    plt.show()
    
    
#trade_freq_df.head()    
glanceback = timedelta(days= 5)
trade_freq_df1 = trade_freq_df1.sort_values(by=['Count'], ascending=False)
k = 0
for i in range(3):
    for j in range(2):
        date = trade_freq_df1.index.values[k]
        date_df = pd.to_datetime(str(date))
        graph_range(date_df)
        k = k + 1<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.099609375, 'fraction_numerical': 0.017361111111111112, 'mean_word_length': 5.838278931750742, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '588011', 'license': 'MIT', 'n_tokens_mistral': 1513, 'n_tokens_neox': 1419, 'n_words': 444, 'path': 'Collect Senator Data.ipynb', 'repo_name': 'paulmtree/Suspicious-Senator-Trading', 'size': '4613'}