<jupyter_start><jupyter_text>This is a Jupyter Notebook. It contains a combination of markdown and code cells. This is an example of a markdown cell where I can *change* **formatting** in:* lots* of* different* waysYou can double click on this cell to see the underlying markdown and edit it. "Execute" the cell by typing `Shift`+`Enter`  This is a big header I made in markdown by putting a  at the front More importantly, you can also add cells where you can execute Python code. Execute the cell below by clicking it and typing `Shift+Enter`.<jupyter_code>total = 1 + 1
my_name = 'Brian'

print('1 + 1 = {0}'.format(1+1))
print('My name is {0}'.format(my_name))<jupyter_output><empty_output><jupyter_text>You can add, delete, and move cells as well. Add a cell below by typing `Control+M`, letting go, then press `B`. You can also go to "Insert" in the menu and select "Insert Cell Below". You can delete cells by selecting the cell, typing `Ctrl+M`, letting go, then press `D` twice. You can also select the cell, go to "Edit" in the menu and select "Delete Cells." You can move cells around with the up and down arrows in the toolbar above.<jupyter_code>I'm #1. Put me on top.
I'm #2. Put me in the middle.
I'm #3. Put me on the bottom.<jupyter_output><empty_output><jupyter_text>Here are some great resources on using Jupyter Notebooks:* [Jupyter Notebook Tutorial](https://www.datacamp.com/community/tutorials/tutorial-jupyter-notebook)* [Jupyter Notebook Quick Start Guide](https://jupyter-notebook-beginner-guide.readthedocs.io/en/latest/index.html)* [Gallery of Interesting Jupyter Notebooks](https://github.com/jupyter/jupyter/wiki/A-gallery-of-interesting-Jupyter-Notebooks)* ["I Don't Like Notebooks"](https://docs.google.com/presentation/d/1n2RlMdmv1p25Xy5thJUhkKGvjtV-dkAIsUXP-AL4ffI/edit)  Get data from the web using `requests`Let's revisit the structured Member Data the House of Representatives publishes about its members. Open this link in another tab to use as a reference: http://clerk.house.gov/xml/lists/MemberData.xmlWe have to pretend to be a web browser in order for it to actually give us data. I would argue this is ethical in the context of getting data from a public institution for education and research, but faking headers or spoofing the "user agent" like this to get data from websites is not always ethical.<jupyter_code>headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'}

# Congress webmasters actually have nicely parsed member data into XML for you
house_raw = requests.get('http://clerk.house.gov/xml/lists/MemberData.xml',headers=headers).text<jupyter_output><empty_output><jupyter_text>Inspect what's inside the `house_raw` object. It's text, but in a very structured format.<jupyter_code>house_raw[:500]<jupyter_output><empty_output><jupyter_text>Parse data using `BeautifulSoup` Parse the text into something usable for Python using `BeautifulSoup`.<jupyter_code>from bs4 import BeautifulSoup

house_soup = BeautifulSoup(house_raw,'lxml')<jupyter_output><empty_output><jupyter_text>Navigating soupWe can navigate down into the tree. We can access the children inside members by using the dot notation with `.members` and then parsing each child into a list of `` objects with `contents`.Here are the first 3 in that list.<jupyter_code>house_soup.members.contents[:3]<jupyter_output><empty_output><jupyter_text>How many members are there total? Hopefully 435, but it's actually 441 -- probably something worth looking into who these extra 6 are.<jupyter_code>len(house_soup.members.contents)<jupyter_output><empty_output><jupyter_text>We can get the names of all the states they represent. Here's a basic example of how to access the "state-fullname" for the first member's information.<jupyter_code>house_soup.members.contents[0].find('state-fullname')<jupyter_output><empty_output><jupyter_text>We can access the text inside the tag with `.text`<jupyter_code>house_soup.members.contents[0].find('state-fullname').text<jupyter_output><empty_output><jupyter_text>A list comprehension can get us all these "state_fullname"s, which we'll save as `member_states`.<jupyter_code>member_states = [member.find('state-fullname').text for member in house_soup.members.contents]
member_states[:10]<jupyter_output><empty_output><jupyter_text>A `set` operation on the list will get us only the unique names. Sure enough, there's American Samoa, District of Columbia, Guam, Northern Mariana Islands, Puerto Rico, and Virgin Islands, which correspond to the extra 6 representatives (who are seated, but cannot vote on legislation).<jupyter_code>set(member_states)<jupyter_output><empty_output><jupyter_text>There is also a "Committees" tree as well. Here is the first committee.<jupyter_code>house_soup.committees.contents[0]<jupyter_output><empty_output><jupyter_text>How many total committees?<jupyter_code>len(house_soup.committees.contents)<jupyter_output><empty_output><jupyter_text>What are the names of each committee? We can't access the "committee-fullname" attribute like we did before because of the hyphen. This should return a `NameError`.<jupyter_code>house_soup.committees.contents[0].committee-fullname<jupyter_output><empty_output><jupyter_text>Instead, we'll use "find" to get this tag's information.<jupyter_code>house_soup.committees.contents[0].find('committee-fullname')<jupyter_output><empty_output><jupyter_text>And use `text` to get the name of the committee.<jupyter_code>house_soup.committees.contents[0].find('committee-fullname').text<jupyter_output><empty_output><jupyter_text>Let's put it all together and make a list comprehension of all the committee names.<jupyter_code>[committee.find('committee-fullname').text for committee in house_soup.committees.contents]<jupyter_output><empty_output><jupyter_text>Finding soupNavigating soup can be helpful, but you likely have some tag in mind that you want to find. For that, we'll use the [`find_all`](https://www.crummy.com/software/BeautifulSoup/bs4/doc/find-all) function to find all the tags matching some criteria.We pass the name of any tag and it will find all the instances of that tag. Here are those 441 members organized by their tag.<jupyter_code>len(house_soup.find_all('member'))<jupyter_output><empty_output><jupyter_text>But we could also get members' sworn-dates.<jupyter_code>sworn_dates = house_soup.find_all('sworn-date')
sworn_dates[:10]<jupyter_output><empty_output><jupyter_text>These sworn dates have additional text and attributes, which might be helpful. For instance, the 121st record has a "sworn-date" of June 26, 2017.We can access this 121st element in the list.<jupyter_code>sworn_dates[121]<jupyter_output><empty_output><jupyter_text>And then access the "date" attribute (inside the tag) like a dictionary, which should return "20170626".<jupyter_code>sworn_dates[121]['date']<jupyter_output><empty_output><jupyter_text>Or access the text of the string itself, which should return "June 26, 2017"<jupyter_code>sworn_dates[121].text<jupyter_output><empty_output><jupyter_text>We could also use `find_all` to get the information about each committee's sub-committees. We can call `find_all` if we've navigated down into a part of the tree already. This should return only the subcommittees for the first committee in the list.<jupyter_code>house_soup.committees.contents[0].find('committee-fullname').text
house_soup.committees.contents[0].find_all('subcommittee')<jupyter_output><empty_output><jupyter_text>Even better, we could just get the "subcommittee-fullname"s.<jupyter_code>house_soup.committees.contents[0].find_all('subcommittee-fullname')<jupyter_output><empty_output><jupyter_text>A simple list comprehension can turn this list of tags back into strings.<jupyter_code>[sc.text for sc in house_soup.committees.contents[0].find_all('subcommittee-fullname')]<jupyter_output><empty_output><jupyter_text>We can tie this all together and make a dictionary keyed by the committee name with a list of sub-committees.<jupyter_code>committees = {}

for committee in house_soup.committees.find_all('committee'):
    # Get the name of the committee
    committee_name = committee.find('committee-fullname').text
    
    # Get all the subcommittees inside this committee
    subcommittees = committee.find_all('subcommittee-fullname')
    
    # Subcommittee names
    subcommittee_names = [sc.text for sc in subcommittees]
    
    # Assign the committee name to the 
    committees[committee_name] = subcommittee_names
    
committees<jupyter_output><empty_output><jupyter_text>Building network of member-subcommittee assignments<jupyter_code>house_soup.members.contents[13]<jupyter_output><empty_output><jupyter_text>Get the contents of the "committee-assignments" tag for this member as an example. We can see this includes the parent committees as well as the specific subcommittees. Unfortunately we only have codes for the committees and sub-committees.<jupyter_code>house_soup.members.contents[13].find('committee-assignments').contents<jupyter_output><empty_output><jupyter_text>We could do a `.find_all('subcommittee')` to get only the subcommittee assignments.<jupyter_code>[sc['subcomcode'] for sc in house_soup.members.contents[13].find_all('subcommittee')]<jupyter_output><empty_output><jupyter_text>Let's map these subcommittee codes back to their full names.<jupyter_code>subcommittee_codes_names = {}

for subcommittee in house_soup.committees.find_all('subcommittee'):
    code = subcommittee['subcomcode']
    name = subcommittee.find('subcommittee-fullname').text
    subcommittee_codes_names[code] = name

subcommittee_codes_names<jupyter_output><empty_output><jupyter_text>With these examples in hand, write a loop that makes a bipartite edgelist of members and their sub-committee assignments.<jupyter_code>member_subcommittee_edgelist = []

for member in house_soup.members.contents:
    member_name = member.find('sort-name').text
    members_subcommittees = member.find_all('subcommittee')
    for sc in members_subcommittees:
        edge = (member_name,sc['subcomcode'],sc['rank'])
        member_subcommittee_edgelist.append(edge)
        
member_subcommittee_df = pd.DataFrame(member_subcommittee_edgelist,columns=['Member','Code','Rank'])
member_subcommittee_df['Name'] = member_subcommittee_df['Code'].map(subcommittee_codes_names)
member_subcommittee_df.head()<jupyter_output><empty_output><jupyter_text>Have to do a bit of work to give the nodes colors.<jupyter_code>member_colors = {member_name:'blue' for member_name in member_subcommittee_df['Member'].unique()}
subcommittee_colors = {sc:'green' for sc in member_subcommittee_df['Name'].unique()}
combined = {**member_colors,**subcommittee_colors}<jupyter_output><empty_output><jupyter_text>Try to visualize the network.<jupyter_code># networkx is a network analysis library in Python
import networkx as nx

# Take the pandas edgelist and turn it into a networkx graph object
g = nx.from_pandas_edgelist(member_subcommittee_df,source='Member',target='Name',edge_attr=True)

# Make the node colors
node_colors = [combined[n] for n in g.nodes()]

# Compute the note positions
g_pos = nx.spring_layout(g,scale=1000,iterations=50)

# Draw the graph
f,ax = plt.subplots(figsize=(12,12))
nx.draw(G = g,
        ax = ax,
        pos = g_pos,
        with_labels = True,
        node_size = 100,
        font_size = 5,
        node_color = node_colors,
        width = 1
       )<jupyter_output><empty_output><jupyter_text>We can also project this two-mode network down to one-mode of sub-committees linked to sub-committees if they share members in common.<jupyter_code>wpg = nx.bipartite.weighted_projected_graph(g,nodes = member_subcommittee_df['Member'])

# Compute the note positions
wpg_pos = nx.spring_layout(wpg,scale=1000,iterations=50)

# Draw the graph
f,ax = plt.subplots(figsize=(12,12))
nx.draw(G = wpg,
        ax = ax,
        pos = wpg_pos,
        with_labels = True,
        node_size = 100,
        font_size = 8,
        width = 1
       )<jupyter_output><empty_output><jupyter_text>JSON from Twitter Below is an example of a [tweet status](https://dev.twitter.com/overview/api/tweets) object that Twitter's [API returns](https://dev.twitter.com/rest/reference/get/statuses/show/id). This `obama_tweet` dictionary corresponds to [this tweet](https://twitter.com/BarackObama/status/831527113211645959).<jupyter_code>obama_tweet = {'created_at': 'Tue Feb 14 15:34:47 +0000 2017',
               'favorite_count': 1023379,
               'hashtags': [],
               'id': 831527113211645959,
               'id_str': '831527113211645959',
               'lang': 'en',
               'media': [{'display_url': 'pic.twitter.com/O0UhJWoqGN',
                          'expanded_url': 'https://twitter.com/BarackObama/status/831527113211645959/photo/1',
                          'id': 831526916398149634,
                          'media_url': 'http://pbs.twimg.com/media/C4otUykWcAIbSy1.jpg',
                          'media_url_https': 'https://pbs.twimg.com/media/C4otUykWcAIbSy1.jpg',
                          'sizes': {'large': {'h': 800, 'resize': 'fit', 'w': 1200},
                                    'medium': {'h': 800, 'resize': 'fit', 'w': 1200},
                                    'small': {'h': 453, 'resize': 'fit', 'w': 680},
                                    'thumb': {'h': 150, 'resize': 'crop', 'w': 150}},
                          'type': 'photo',
                          'url': 'https://t.co/O0UhJWoqGN'}],
               'retweet_count': 252266,
               'source': '<a href="http://twitter.com" rel="nofollow">Twitter Web Client</a>',
               'text': 'Happy Valentine’s Day, @michelleobama! Almost 28 years with you, but it always feels new. https://t.co/O0UhJWoqGN',
               'urls': [],
               'user': {'created_at': 'Mon Mar 05 22:08:25 +0000 2007',
                        'description': 'Dad, husband, President, citizen.',
                        'favourites_count': 10,
                        'followers_count': 84814791,
                        'following': True,
                        'friends_count': 631357,
                        'id': 813286,
                        'lang': 'en',
                        'listed_count': 221906,
                        'location': 'Washington, DC',
                        'name': '<NAME>',
                        'profile_background_color': '77B0DC',
                        'profile_background_image_url': 'http://pbs.twimg.com/profile_background_images/451819093436268544/kLbRvwBg.png',
                        'profile_banner_url': 'https://pbs.twimg.com/profile_banners/813286/1484945688',
                        'profile_image_url': 'http://pbs.twimg.com/profile_images/822547732376207360/5g0FC8XX_normal.jpg',
                        'profile_link_color': '2574AD',
                        'profile_sidebar_fill_color': 'C2E0F6',
                        'profile_text_color': '333333',
                        'screen_name': 'BarackObama',
                        'statuses_count': 15436,
                        'time_zone': 'Eastern Time (US & Canada)',
                        'url': 'https://t.co/93Y27HEnnX',
                        'utc_offset': -18000,
                        'verified': True},
               'user_mentions': [{'id': 409486555,
                                  'name': '<NAME>',
                                  'screen_name': 'MichelleObama'}]}<jupyter_output><empty_output><jupyter_text>When was this tweet created? What's the text of the tweet? Who wrote the tweet? How many times was the tweet retweeted and favoritedd? Who does this tweet mention? What images are in the tweet? Let's look at the image!<jupyter_code>from IPython.display import Image
Image(obama_tweet['media'][0]['media_url'])<jupyter_output><empty_output><jupyter_text>Getting data from the Wikipedia APIWikipedia stores every single change that's been made to every one of its articles (in all its languages!) since approximately 2002. You can view [the revision history here](https://en.wikipedia.org/w/index.php?title=University_of_Colorado_Boulder&action=history) for the [University of Colorado Boulder](https://en.wikipedia.org/wiki/University_of_Colorado_Boulder) article. You can also access this data programatically via the "[revision](https://www.mediawiki.org/wiki/API:Revisions)" endpoint in Wikipedia's API.<jupyter_code>import requests                      # Import the requests library to let us talk to APIs

# Make a request dictionary with all the parameters the Wikipedia API wants for requests to use
request = {}                         # Start with an empty dictionary
request['action'] = 'query'          # Talk to the query end-point of the API
request['format'] = 'json'           # Return the data in JSON format
request['prop'] = 'revisions'        # Get the revisions of an article
request['titles'] = "University of Colorado Boulder"    # Get the data for an article
request['rvlimit'] = 10             # Get the 100 most recent revisions to the article
request['formatversion'] = 2         # Output the data in a friendlier version
request['rvdir'] = 'older'           # Output the data in a friendlier version

# Ask for specific properties of every revision, joining them together by pipes "|"
# User is the username, 
# ids is the unique revision id
# size is the number of characters in the content
# sha1 is the hash of the article version
# include "content" if you want the wikitext markup generating the article's content
request['rvprop'] = '|'.join(['user','timestamp','ids','size','sha1'])

# Make the request to Wikipedia's API and use the .json() method to return JSON response to a Python dictionary
result = requests.get('http://en.wikipedia.org/w/api.php', params=request).json()

# Look at the results
result<jupyter_output><empty_output><jupyter_text>This is fundamentally tabular data, so let's use the pandas library to turn the data into a tabular structure.<jupyter_code>import pandas as pd
pd.DataFrame(result['query']['pages'][0]['revisions'])<jupyter_output><empty_output><jupyter_text>Getting HTML data from WikipediaWikipedia interestingly has what appears to be a pretty exhaustive collection of data about historical school shootings in the United States: https://en.wikipedia.org/wiki/List_of_school_shootings_in_the_United_StatesWikipedia gets really cranky when you try to use `requests.get()` directly on its articles and prefers that you get the data from its API instead. Let's respect their wishes and get the data from the API.<jupyter_code># Make a request dictionary with all the parameters the Wikipedia API wants for requests to use
request = {}                         # Start with an empty dictionary
request['action'] = 'parse'          # Talk to the "parse" end-point of the API
request['format'] = 'json'           # Return the data in JSON format (for now)
request['prop'] = 'text|sections'    # Get the HTML markup of an article
request['formatversion'] = 2         # Output the data in a friendlier version
request['page'] = 'List of school shootings in the United_States'
#request['section'] = 38              # Only get the HTML for a specific section "Historical details" is the 34th
request['disabletoc'] = True         # Simplify things a bit
request['disableeditsection'] = True # Simplify things a bit

# Make the request to Wikipedia's API and use the .json() method to return JSON response to a Python dictionary
school_shootings_json = requests.get('http://en.wikipedia.org/w/api.php', params=request).json()

# The HTML for the page lives in here as a big string
school_shootings_html = school_shootings_json['parse']['text']<jupyter_output><empty_output><jupyter_text>Convert the raw HTML string to Soup so we can do things to it.<jupyter_code>school_shootings_soup = BeautifulSoup(school_shootings_html,'html.parser')<jupyter_output><empty_output><jupyter_text>Find all the tables in the soup.<jupyter_code>school_shootings_tables = school_shootings_soup.find_all('table')
print("There are a total of {0:,} tables in the data.".format(len(school_shootings_tables)))<jupyter_output>There are a total of 19 tables in the data.<jupyter_text>What does one of them look like?<jupyter_code>school_shootings_tables[0]<jupyter_output><empty_output><jupyter_text>There's 18 of these tables and they're pretty gnarly to parse. Luckily pandas has a [read_html](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_html.html) function that will (try to!) turn this mess into something tabular.Let's take the most recent sub-table (`school_shootings_tables[-1]`), cast it back to a string (`str`), tell pandas to use the first row as a header (`header=0`).<jupyter_code>pd.read_html(str(school_shootings_tables[-1]),header=0)[0].tail()<jupyter_output><empty_output><jupyter_text>Loop through the list of tables in `school_shootings_tables` and conver them to DataFrames and concatenate them all together into a list of all the shootings.<jupyter_code># Start with an empty list
table_list = []

# Write a loop to go over every table in the list of tables
for table in school_shootings_tables:
    
    # Pull out the table and parse it into a DataFrame
    _df = pd.read_html(str(table),header=0)[0]
    
    # Add the parsed table back to the table_list
    table_list.append(_df)
    
# Concatenate the list of tables into a giant table of all the shootings
shootings_df = pd.concat(table_list)

# Parse the Date column into a proper datetime, coercing bad dates to something close
shootings_df['Date'] = pd.to_datetime(shootings_df['Date'],errors='coerce')

# Inspect the 5 most recent shootings
shootings_df.tail()<jupyter_output><empty_output><jupyter_text>Plot out the cumulative number of deaths over time according to this data.<jupyter_code>%matplotlib inline
import matplotlib.pyplot as plt

# Pull out only the Date and Deaths columns, reshape, and cumulatively sum
cumulative_shootings = shootings_df[['Date','Deaths']].set_index('Date')['Deaths'].cumsum()

# Create a plot
ax = cumulative_shootings.plot(lw=5)

# Log-scale the y-axis and set limits to 1e0 thru 1e3
ax.set_yscale('log')
ax.set_ylim((1e0,1e3))

# Always label your axes!
ax.set_ylabel('Cumulative deaths')
ax.set_xlabel('Year')

# Include a reference line for a major piece of legislation
ax.axvline(pd.Timestamp('1968-01-01'),c='g') # Gun Control Act of 1968
#ax.axvline(pd.Timestamp('1977-01-01'),c='r') # NRA Cincinnati Revolution of 1977<jupyter_output><empty_output><jupyter_text>Getting data from Twitter's API*<jupyter_code>import twitter


api = twitter.Api(consumer, consumer_secret, access, access_secret)

status = api.GetStatus(tweet_id)
status
status.AsDict()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10198449204888947, 'fraction_numerical': 0.022166732378323915, 'mean_word_length': 4.3890462700661, 'pattern_counts': {'":': 0, '<': 188, '<?xml version=': 0, '>': 188, 'https://': 19, 'lorem ipsum': 0, 'www.': 3, 'xml': 5}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '46', 'dir': 'jupyter-structured-clean-dedup', 'id': '397529', 'license': 'MIT', 'n_tokens_mistral': 7448, 'n_tokens_neox': 6818, 'n_words': 2061, 'path': 'CARTSS-ITSS Workshop 01.ipynb', 'repo_name': 'brianckeegan/ITSS-Fall-2018', 'size': '22841'}