<jupyter_start><jupyter_text>Import needed libraries<jupyter_code>import requests
from lxml import html<jupyter_output><empty_output><jupyter_text>We used the library "request" last time in getting Twitter data (REST-ful).  We are introducing the new "lxml" library for analyzing & extracting HTML elements and attributes here.  Use Requests to get HackerNews content HackerNews is a community contributed news website with an emphasis on technology related content.  Let's grab the set of articles that are at the top of the HN list.<jupyter_code>response = requests.get('http://news.ycombinator.com/')
response
response.content<jupyter_output><empty_output><jupyter_text>We will now use lxml to create a programmatic access to the content from HackerNews.  Analyzing HTML Content<jupyter_code>page = html.fromstring(response.content)
page<jupyter_output><empty_output><jupyter_text>CSS SelectorsFor those of you who are web designers, you are likely very familiar with Cascading Stylesheets (CSS).  Here is an example for how to use CSS selector for finding specific HTML elements<jupyter_code>posts = page.cssselect('.title')
len(posts)<jupyter_output><empty_output><jupyter_text>Details of how to use CSS selectors can be found in the w3 schools site:http://www.w3schools.com/cssref/css_selectors.asp  XPathAlternatively, we can use a standard called "XPath" to find specific content in the HTML.<jupyter_code>posts = page.xpath('//td[contains(@class, "title")]')
len(posts)<jupyter_output><empty_output><jupyter_text>We are only interested in those "td" tags that contain an anchor link to the referred article.<jupyter_code>posts = page.xpath('//td[contains(@class, "title")]/a')
len(posts)<jupyter_output><empty_output><jupyter_text>So, only half of those "td" tags with "title" contain posts that we are interested in.  Let's take a look at the first such post.<jupyter_code>first_post = posts[0]
first_post.text<jupyter_output><empty_output><jupyter_text>There is a lot of "content" in the td tag's attributes.<jupyter_code>first_post.attrib
first_post.attrib["href"]
all_links = []
for p in posts:
    all_links.append((p.text, p.attrib["href"]))
all_links<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09198542805100182, 'fraction_numerical': 0.001366120218579235, 'mean_word_length': 6.735915492957746, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 3}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '385839', 'license': 'MIT', 'n_tokens_mistral': 689, 'n_tokens_neox': 673, 'n_words': 221, 'path': 'lecture04.data.wrangling/lecture04.web.scaping.ipynb', 'repo_name': 'philmui/algorithmic-bias-2019', 'size': '2196'}