<jupyter_start><jupyter_text>Interactive Dashboard Congressional DataThe congress doesn't seem to publish their own data API: https://www.congress.gov/about/data. Bulk data for download are available ProPublica Data StoreProPublica is an American nonprofit organization based in New York City. It is a nonprofit newsroom that aims to produce investigative journalism in the public interest. [Wikipeda](https://en.wikipedia.org/wiki/ProPublica)ProPublica won multiple [Pulitzer Prize for Investigative Reporting](https://en.wikipedia.org/wiki/Pulitzer_Prize_for_Investigative_Reporting)* In 2010, "The Deadly Choices At Memorial": a story of exhausted doctors when they were cut off by the floodwaters of Hurricane Katrina.* In 2011, "The Wall Street Money Machine". This was the first time a Pulitzer was awarded to a group of stories not published in print.* In 2016, Explanatory Reporting, in collaboration with The Marshall Project: an examination of law enforcement's failures to investigate reports of rape properly and to comprehend the traumatic effects on its victims* In 2017, Public Service for a series of reports on the use of eviction rules by the New York City Police Department.Propublica publishes extensive datasets used for their reporting: https://www.propublica.org/datastore/. Some examples:* [Trump Administration](https://www.propublica.org/datastore/collection/trump-administration)* [Education](https://www.propublica.org/datastore/datasets/education)* [Health](https://www.propublica.org/datastore/datasets/health)Some are free, some are not Sunlight Congress DataAnother data source that looks promising is here: https://sunlightlabs.github.io/congress/<jupyter_code>!pip install python-congress
!git clone https://github.com/eyeseast/propublica-congress
!cd propublica-congress && python setup.py install<jupyter_output>Cloning into 'propublica-congress'...
remote: Enumerating objects: 408, done.[K
remote: Total 408 (delta 0), reused 0 (delta 0), pack-reused 408[K
Receiving objects: 100% (408/408), 69.79 KiB | 9.97 MiB/s, done.
Resolving deltas: 100% (261/261), done.
running install
running bdist_egg
running egg_info
creating python_congress.egg-info
writing python_congress.egg-info/PKG-INFO
writing dependency_links to python_congress.egg-info/dependency_links.txt
writing requirements to python_congress.egg-info/requires.txt
writing top-level names to python_congress.egg-info/top_level.txt
writing manifest file 'python_congress.egg-info/SOURCES.txt'
reading manifest file 'python_congress.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'python_congress.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
creating build
creating build/lib
creating build/lib/congress
copying congress/bills.py -> build/lib/[...]<jupyter_text>ProPublica Congress API* Free Congress API: https://www.propublica.org/datastore/api/propublica-congress-api* A python interface library: https://github.com/eyeseast/propublica-congress* Author: https://github.com/eyeseast<jupyter_code>from congress import Congress
cg = Congress("SBYUPLXQXG7rZwGotxRK19M7PasVKw6t9R0u709Q")<jupyter_output><empty_output><jupyter_text>Congressional member dataEach congress-person has an ID: e.g., <NAME>'s ID is `P000197`<jupyter_code>pelosi = cg.members.get('P000197')
pelosi<jupyter_output><empty_output><jupyter_text>Some interesting attributes here:* Social media accounts* Bills sponsored, co-sponsored* Votes with party percentage Bills introduced by memberOne can pull data about bills introduced by any congress-person.<jupyter_code>pelosi_bills = cg.bills.by_member('P000197')
pelosi_bills<jupyter_output><empty_output><jupyter_text>Congress, Session, Bills, and all thatHere are some common terminology necessary for understanding the dataset:* **Chamber**: Senate or House* **[Congress](https://en.wikipedia.org/wiki/List_of_United_States_Congresses)**: number given to a two year period, e.g., [102nd Congress](https://en.wikipedia.org/wiki/102nd_United_States_Congress)* **Session**: first or second year of the two year period. Special sessions take place from time to time* **Roll-call**:      > A vote in which each senator votes "yea" or "nay" as his or her name is called by the clerk, so that the names of senators voting on each side are recorded. Under the Constitution, a roll call vote must be held if demanded by one-fifth of a quorum of senators present, a minimum of 11. [Senate Glossary](https://www.senate.gov/reference/glossary_term/roll_call_vote.htm)           An example of roll-call voting result can be found [US senate](https://www.senate.gov/legislative/LIS/roll_call_lists/roll_call_vote_cfm.cfm?congress=110&session=2&vote=00214)      An example with some visualization can be found at [govtrack](https://www.govtrack.us/congress/votes/116-2019/h233)* **Voice vote**:      > A vote in which the presiding officer states the question, then asks those in favor and against to say "Yea" or "Nay," respectively, and announces the result according to his or her judgment. The names or numbers of senators voting on each side are not recorded. [Senate Glossary](https://www.senate.gov/reference/glossary_term/voice_vote.htm)         Votes by month<jupyter_code>cg.votes.by_month('senate', '2019', '05')<jupyter_output><empty_output><jupyter_text>* Note that data are pulled from multiple sources:      * Vote results: https://www.senate.gov/legislative/LIS/roll_call_votes/vote1161/vote_116_1_00129.xml    * Bill information: https://api.propublica.org/congress/v1/116/bills/hr2157.json* Gives a summary of roll-call votes Congress-person pairwise  comparisons<jupyter_code>cg.members.compare('C001098', 'B000944', 'senate')<jupyter_output><empty_output><jupyter_text>This does not provide voting results, but useful as summaries  Get voting records<jupyter_code>cg.votes.get(chamber='senate', rollcall_num=127, session=1, congress=115)<jupyter_output><empty_output><jupyter_text>Dashboard of voting similarityLet's make a dashboard of analyzing voting similarity between congress-people and getting a broad sense of their voting patterns. In order to do this, we will create a dashboard for subsetting the data:* Select chamber* Select data extentsFollowing are convenience functions for converting between congress designation (e.g., 80) and the year (e.g., 1947). ProPublica data starts at* Senate: 80th congress, years 1947-1948* House: 102nd congress, years 1991-1992<jupyter_code>import ipywidgets as widgets

def congress_year_convert(congress=None, year=None):
    """
    Given congress or year (not both), return corresponding congress and year
    """
    # check only one is not None
    args = [congress, year]
    assert any(args) and not all(args) 
    
    if congress is not None:
        year = (congress - 80)*2 + 1947
    else:
        congress = (year - 1947)//2 + 80
        
    return (congress, year)


def get_congress_extent(extent):
    """
    Compute congress data extent depending on today's date
    """
    
    from datetime import datetime
    now = datetime.now()
    i = int(now.day>10) # include this year
    
    while True:
        c, y = extent[-1]
        if y < (now.year+i):
            new_congress = congress_year_convert(c+1)
            extent.append(new_congress)
        else:
            break
            
    return(extent)<jupyter_output><empty_output><jupyter_text>Compute the tuple list of (`congress`, `year`) pairs:<jupyter_code>data_extent = {'senate': [(80,1947)], 'house': [(102,1991)]}
 
data_extent['senate'] = get_congress_extent(data_extent['senate'])
data_extent['house']  = get_congress_extent(data_extent['house'])<jupyter_output><empty_output><jupyter_text>The menu items can say something like `congress (year)` as the menu item, and the values are `congress`.<jupyter_code>def get_select_menu(extent):
    
    menu = []
    for one in extent:
        c, y = one
        menu += [('%d (%d)' % (c, y), c)]
        
    return(menu)

get_select_menu(data_extent['house'])<jupyter_output><empty_output><jupyter_text>Dropdown boxesDrowpdown menus for selecting chamber and congress:<jupyter_code>w_chamber = widgets.Dropdown(
    description='Chamber:',
    options=[('Senate', 'senate'), ('House', 'house')],
    value='senate')

w_congress = widgets.SelectionSlider(
    description='Congress:',
    options=get_select_menu(data_extent['house']))

display(w_chamber, w_congress)<jupyter_output><empty_output><jupyter_text>In order to link the two widgets, selected values and menu states have to be consistent:* Since `house` is selected in the dropdown,* Slider should go from 80th congress since senate has more data availableWhy not link the two?<jupyter_code>w_chamber = widgets.Dropdown(
    description='Chamber:',
    options=[('Senate', 'senate'), ('House', 'house')],
    value='house')

w_congress = widgets.SelectionSlider(
    description='Congress:',
    options=get_select_menu(data_extent[w_chamber.value]))

display(w_chamber, w_congress)<jupyter_output><empty_output><jupyter_text>Events for dynamic updatesNotice the chamber choice does not change the slider. In order to create dynamic updates, we need to define events for the dropdown menu. Everytime dropdown menu is changed, slider will be updated.<jupyter_code>def update(change):
    
    from pprint import pprint
    
    print("\n*** calling update")
    print("old: ", change['old'])
    print("new: ", change['new'])
    
    if update.detail:
        print("------------")
        pprint(change)
        
update.detail = True # lab 1 trick

w_chamber.observe(update, names='value') # what happens for 'label'<jupyter_output><empty_output><jupyter_text>Function argument `change` is given automatically by `observe` function.* Dropdown: select chamber* SelectRangeSlider: select data extent range* IntProgress: progress bar do indicate download status* ToggleButton: selects go!First, we create and collect widgets into a list<jupyter_code>w = []

w_chamber = widgets.Dropdown(
    description='Chamber:',
    options=[('Senate', 'senate'), ('House', 'house')],
    value='senate')
w.append(w_chamber)

w_congress = widgets.SelectionRangeSlider(
    description='Congress:',
    options=get_select_menu(data_extent[w_chamber.value]),
    value=(114,116),
    layout=widgets.Layout(width='100%'))
w.append(w_congress)

w_current = widgets.Text(
    description='Receiving:',
    value='',
    layout=widgets.Layout(width='100%'))
w.append(w_current)

w_start = widgets.ToggleButton(
    description='Run Analysis',
    value=False,
    tooltip='Starts analysis',
)
w.append(w_start)<jupyter_output><empty_output><jupyter_text>Next, display the widgets by breaking out the list.<jupyter_code>display(*w)
def update_congress(change):
    
    if change['new'] == 'house':
        w_congress.options = get_select_menu(data_extent['house'])
    elif change['new'] == 'senate':
        w_congress.options = get_select_menu(data_extent['senate'])
        
w_chamber.observe(update_congress, names='value')
def run_analysis(change):
    
    print(w_chamber.value)
    print(w_congress.value)

w_start.observe(run_analysis, names='value')
def votes_df(rollcall, session, congress, chamber):
    
    from pandas import DataFrame
    from numpy import nan
    
    try:
        d = cg.votes.get(chamber, rollcall, session, congress)
        df = DataFrame(d['votes']['vote']['positions'])
        df['name'] = df.party + '-' + df.name
        df = df[['name', 'vote_position']] # just votes
        df.replace({'vote_position':{'Yes':1, 'No':0, 'Not Voting':nan, 'Present':nan}}, inplace=True) # recode values
        
        # ccsr: chamber, congress, session, rollcall
        cname = '_'.join(['ccsr', chamber[0], str(congress), str(session), str(rollcall)])
        df.rename(index=str, columns={'vote_position':cname}, inplace=True) # keep voting record as index
        
        df = df.set_index('name').transpose() # make wide
    except:
        
        # probably end of roll call votes for that session
        df = None
    
    return(df)

votes_df(320, 1, 115, 'senate')
votes_df(420, 1, 115, 'senate')
import pandas as pd
asdf = pd.DataFrame(cg.votes.get('senate', 320, 1, 115)['votes']['vote']['positions'])
asdf.head()
# asdf.name.str.split(' ').apply(lambda x: x[-1]) + '-' + asdf.party
df = []
for c in range(*w_congress.value):
    
    for s in [1, 2]:
        
        r = 1
        while True:
            
            w_current.value = ', '.join(['Congress:'+str(c), 'Session:'+str(s), 'RollCall:'+ str(r)])
            
            last_df = votes_df(r, s, c, w_chamber.value)
             
            if last_df is None:
                break
            else:
                df.append(last_df)

            # to save time
            if r >= 20: break
            
            # break
            r += 1
import pandas as pd
hist = pd.concat(df, sort=True)
hist = hist.loc[:,hist.isna().mean(axis=0) < 0.5] # those who served both terms
hist<jupyter_output><empty_output><jupyter_text>Missing valuesThere are missing values due to missing votes: e.g. missing work. Many analysis algorithms do not work if there are missing values. Here is one approach in filling-in missing values. Look at other senators that are similar in voting pattern, and fill-in with what they voted). A package named `fancyimpute` does this.<jupyter_code>!conda install -y -c anaconda mkl-service
!conda install -y -c brittainhard fancyimpute<jupyter_output>Collecting package metadata: done
Solving environment: done

## Package Plan ##

  environment location: /opt/conda

  added / updated specs:
    - fancyimpute


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    climate-0.4.6              |   py36h31ade9f_0          11 KB  brittainhard
    downhill-0.4.0             |   py36hcc12aa4_0          32 KB  brittainhard
    fancyimpute-0.2.0          |   py36h242efd1_0          38 KB  brittainhard
    knnimpute-0.1.0            |   py36h9580492_0          14 KB  brittainhard
    libgpuarray-0.7.6          |    h14c3975_1003         263 KB  conda-forge
    plac-1.0.0                 |             py_0          18 KB  conda-forge
    pygpu-0.7.6                |py36h3010b51_1000         657 KB  conda-forge
    theano-1.0.3               |           py36_0         3.7 MB  conda-forge
    ------------------------------------------------------------
 [...]<jupyter_text>Simulating missing values<jupyter_code>from fancyimpute import KNN

# X is the complete data matrix
# X_incomplete has the same values as X except a subset have been replace with NaN
import numpy as np
p = 10

np.random.seed(3)

from scipy.stats import multivariate_normal
mu = np.array([0, 0, 0, 0])
sig = np.array([[1, -0.6, 0, 0], [-0.6, 1, 0.5, 0], [0, 0.5, 1, 0.4], [0, 0, 0.4, 1]])
X = multivariate_normal.rvs(mean=mu, cov=sig, size=100)

indxi = np.random.randint(0, 4, 10)
indxj = np.random.randint(0, 50, 10)

X_incomplete = X.copy()
X_incomplete[tuple(indxj), tuple(indxi)] = np.nan

# Use 3 nearest rows which have a feature to fill in each row's missing features
X_filled_knn = KNN(k=5).complete(X_incomplete)
%matplotlib inline

import matplotlib.pyplot as plt
import seaborn as sns

missing = np.isnan(X_incomplete)

fig, axn = plt.subplots(2, 2, figsize=(15, 10), sharex=True, sharey=True)

fig.tight_layout(rect=[0, 0, .9, 1])
ax = axn.flat

cbar_ax = fig.add_axes([.91, .3, .03, .4])

amap = sns.color_palette("BrBG", 11)

sns.heatmap(X, ax=ax[0], cmap=amap, vmin=-2.5, vmax=2.5, cbar_ax=None, cbar=False)

temp = missing 
sns.heatmap(missing, ax=ax[1], cmap=sns.color_palette("gray_d", 3,), vmin=-2.5, vmax=2.5, cbar_ax=cbar_ax, cbar=True)

temp = X.copy()
temp[~missing] = np.nan
sns.heatmap(temp, ax=ax[2], cmap=amap, vmin=-2.5, vmax=2.5, cbar_ax=cbar_ax, cbar=True)

temp = X_filled_knn.copy()
temp[~missing] = np.nan
sns.heatmap(temp, ax=ax[3], cmap=amap, vmin=-2.5, vmax=2.5, cbar_ax=cbar_ax, cbar=True)<jupyter_output><empty_output><jupyter_text>Filling-in Voting Data<jupyter_code>histX = hist.values.copy().astype(np.float32)
missing = np.isnan(histX)

histX_filled = KNN(k=5).complete(histX)

fig, axn = plt.subplots(2, 2, figsize=(15, 10), sharex=True, sharey=True)

fig.tight_layout(rect=[0, 0, .9, 1])
ax = axn.flat

cbar_ax = fig.add_axes([.91, .3, .03, .4])

amap = sns.color_palette("gray", 11)

sns.heatmap(histX, ax=ax[0], cmap=amap, vmin=-2.5, vmax=2.5, cbar_ax=None, cbar=False)

sns.heatmap(missing, ax=ax[1], vmin=-2.5, vmax=2.5, cbar_ax=cbar_ax, cbar=True)

temp = histX.copy()
temp[~missing] = np.nan
sns.heatmap(histX_filled, ax=ax[2], cmap=amap, vmin=-2.5, vmax=2.5, cbar_ax=cbar_ax, cbar=True)

# temp = X_filled_knn.copy()
# temp[~missing] = np.nan
# sns.heatmap(temp, ax=ax[3], cmap=amap, vmin=-2.5, vmax=2.5, cbar_ax=cbar_ax, cbar=True)<jupyter_output>Imputing row 1/80 with 1 missing, elapsed time: 0.004<jupyter_text>Transforming Yes-No votesDifferent proportion of Yes votes effect calculation of underlying relationships. So we standardize each person's yes/no vote as if they were numeric attributes. This can be done if we are not interested in interpreting the values in the estimator.<jupyter_code>x = histX_filled.copy()
x -= x.mean(axis=0)
x /= x.std(axis=0)
x
from sklearn.covariance import GraphicalLassoCV

model = GraphicalLassoCV(cv=5)
model.fit(x)
cov_ = model.covariance_
prec_ = model.precision_

preccopy = prec_.copy()
np.fill_diagonal(preccopy, 0)
dfprecision = pd.DataFrame(preccopy, index=hist.columns, columns=hist.columns)
dfprecision.round(2)
plt.figure(figsize=(20,20))
sns.heatmap(dfprecision, cmap=sns.color_palette("BrBG", 11), cbar=False, vmin=-0.5, vmax=0.5)
!pip install plotly
import plotly.plotly as py
import plotly.graph_objs as go

import plotly

plotly.offline.init_notebook_mode(connected=True)

trace = go.Heatmap(z=[[1, 20, 30, 50, 1], [20, 1, 60, 80, 30], [30, 60, 1, -10, 20]],
                   x=['Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday'],
                   y=['Morning', 'Afternoon', 'Evening'])
data=[trace]
plotly.offline.iplot(data, filename='labelled-heatmap')
trace = go.Heatmap(z=dfprecision,
                   x=dfprecision.columns, y=dfprecision.columns, 
                   zmin=-0.4, zmax=0.4, colorscale='Picnic')
fig = go.Figure(dict(data=[trace], 
                     layout=go.Layout(yaxis=dict(scaleanchor="x", scaleratio=1), height=1000)))

plotly.offline.iplot(fig, filename='labelled-heatmap')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1253761822871883, 'fraction_numerical': 0.029180997420464316, 'mean_word_length': 4.026742301458671, 'pattern_counts': {'":': 1, '<': 77, '<?xml version=': 0, '>': 80, 'https://': 20, 'lorem ipsum': 0, 'www.': 11, 'xml': 1}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '528255', 'license': 'Apache-2.0', 'n_tokens_mistral': 6619, 'n_tokens_neox': 6100, 'n_words': 1639, 'path': 'lecture-notes/16-Congressional-Voting-Data-Analysis.ipynb', 'repo_name': 'dchu1234/Spring2019', 'size': '18614'}