<jupyter_start><jupyter_text>RSS You can find RSS feeds on many different sites. [Library of Congress](https://www.loc.gov/rss/) has a lot. Most blogs and news web sites have them, for example [Tech Crunch](https://techcrunch.com/rssfeeds/), [New York Times](http://www.nytimes.com/services/xml/rss/index.html), and [NPR](https://help.npr.org/customer/portal/articles/2094175-where-can-i-find-npr-rss-feeds-). The [DC Public Library](http://www.dclibrary.org/) even gives you an RSS feed of your [catalog searches](https://catalog.dclibrary.org/client/rss/hitlist/dcpl/qu=python). iTunes delivers podcasts by [aggregating RSS feeds](http://itunespartner.apple.com/en/podcasts/faq) from content creators. Today we are going to take a look at the [Netflix Top 100 DVDs](https://dvd.netflix.com/RSSFeeds). We will use the Python package [FeedParser](https://pypi.python.org/pypi/feedparser) to work with the RSS feed. FeedParser will allow us to deconstruct the data in the feed.<jupyter_code>import feedparser
import pandas as pd
RSS_URL = "http://dvd.netflix.com/Top100RSS"
feed = feedparser.parse(RSS_URL)
type(feed)<jupyter_output><empty_output><jupyter_text>"parse" is the primary function in FeedParser. The returned object is dictionary like and can be handled similarly to a dictionary. For example, we can look at the keys it contains and what type of items those keys are.<jupyter_code>feed.keys()
type(feed.bozo)
type(feed.feed)<jupyter_output><empty_output><jupyter_text>We will look at some, but not all, of the data stored in the feed. For more information about the keys, see the [documentation](http://pythonhosted.org/feedparser/).We can use the version to check which type of feed we have.<jupyter_code>feed.version<jupyter_output><empty_output><jupyter_text>Bozo is an interesing key to know about if you are going to parse a RSS feed in code. FeedParser sets the bozo bit when it detects a feed is not well-formed. (FeedParser will still parse the feed if it is not well-formed.) You can use the bozo bit to create error handling or just print a simple warning.<jupyter_code>if feed.bozo == 0:
    print("Well done, you have a well-formed feed!")
else:
    print("Potential trouble ahead.")<jupyter_output>Well done, you have a well-formed feed!<jupyter_text>We can look at some of the feed elements through the feed attribute.<jupyter_code>feed.feed.keys()
print(feed.feed.title)
print(feed.feed.link)
print(feed.feed.description)<jupyter_output>Netflix Top 100
http://dvd.netflix.com
Top 100 Netflix movies, published every 2 weeks.<jupyter_text>The [reference section](http://pythonhosted.org/feedparser/reference.html) of the feedparser documenation shows us all the inforamtion thatcan be in a feed. [Annotated Examples](http://pythonhosted.org/feedparser/annotated-examples.html) are also provided. But note the caution provided-"Caution: Even though many of these elements are required according to the specification, real-world feeds may be missing any element. If an element is not present in the feed, it will not be present in the parsed results. You should not rely on any particular element being present."For example, our feed is RSS 2.0. One of the elements available in this version is the published date.<jupyter_code>feed.feed.published<jupyter_output><empty_output><jupyter_text>We can see from our error, our feed is not using 'published'.As with [standard python dictionaries](https://docs.python.org/3.5/library/stdtypes.htmldict), we can use the "get" method to see if a key exists. This is useful if we are writing code.<jupyter_code>feed.feed.get('published', 'N/A')<jupyter_output><empty_output><jupyter_text>The data we are looking for are contained in the entries. Given the feed we are working with, how many entries do you think we have?<jupyter_code>len(feed.entries)<jupyter_output><empty_output><jupyter_text>The items in entries are stored as a list.<jupyter_code>type(feed.entries)
type(feed.entries[0])
feed.entries[0].keys()
feed.entries[0].title
i = 0
for entry in feed.entries:
    print(i, feed.entries[i].title)
    i += 1<jupyter_output>0 Game of Thrones
1 Sully
2 <NAME>
3 The Magnificent Seven
4 The Accountant
5 Hell or High Water
6 Hacksaw Ridge
7 Star Trek Beyond
8 The Legend of Tarzan
9 Deepwater Horizon
10 Manchester by the Sea
11 Free State of Jones
12 Suicide Squad
13 Independence Day: Resurgence
14 The Girl on the Train
15 Inferno
16 Bad Moms
17 Homeland
18 X-Men: Apocalypse
19 Ghostbusters
20 Central Intelligence
21 Money Monster
22 Passengers
23 The Secret Life of Pets
24 <NAME>
25 Mechanic: Resurrection
26 Outlander
27 The Infiltrator
28 War Dogs
29 <NAME>: Never Go Back
30 Miss Peregrine's Home for Peculiar Children
31 Finding Dory
32 Now You See Me 2
33 Arrival
34 Doctor Strange
35 The Nice Guys
36 Moonlight
37 Me Before You
38 Snowden
39 The Jungle Book (2016)
40 Don't Breathe
41 The BFG
42 Ben-Hur
43 The Shallows
44 Blood Father
45 Captain Fantastic
46 Captain America: Civil War
47 Sausage Party
48 The Huntsman: Winter's War
49 Pete's Dragon
50 The Man Who Knew Infinity
51 W[...]<jupyter_text>Given that information, what is something we can do with this data? Why not make it a dataframe?<jupyter_code>df = pd.DataFrame(feed.entries)
df.head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08486804632618189, 'fraction_numerical': 0.02449212075185115, 'mean_word_length': 5.13986013986014, 'pattern_counts': {'":': 0, '<': 41, '<?xml version=': 0, '>': 41, 'https://': 7, 'lorem ipsum': 0, 'www.': 3, 'xml': 1}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '189461', 'license': 'MIT', 'n_tokens_mistral': 1731, 'n_tokens_neox': 1606, 'n_words': 655, 'path': 'rss/rss_notebook.ipynb', 'repo_name': 'jenniferp1/PyCon2017_web_scraping', 'size': '5296'}