<jupyter_start><jupyter_text>Scraping the iConference 2015 Program The purpose of this exercise is to demonstrate some basic web scraping practices using the python programming language.<jupyter_code># import 3rd party libraries for fetching and parsing HTML documents 
from lxml import html
import requests<jupyter_output><empty_output><jupyter_text>The website we are going to scrape is the [2015 iConference program](https://www.conftool.com/iConference2015/sessions.php). To make our jobs easier, we are going to scape the [print and list view](https://www.conftool.com/iConference2015/index.php?page=browseSessions&print=head&mode=list) of the page.- URL: https://www.conftool.com/iConference2015/index.php?page=browseSessions&print=head&mode=list<jupyter_code># put the base URL for the web scrape into a variable called "urly"
urly = "https://www.conftool.com/iConference2015/index.php?page=browseSessions&print=head&mode=list"
# fetch the web page containing the program for the iConference
response = requests.get(urly) # put the response into a variable called "response"
# parse the HTML document stored in the response variable 
# using the lxml parser imported above
iconf_program = html.document_fromstring(response.content)<jupyter_output><empty_output><jupyter_text>Ok, now we have *fetched* and *parsed* the HTML document we can *extract* data.What data do we want to extract? How about a list of all the events!Lets do an *inspect element* on the [program page](https://www.conftool.com/iConference2015/index.php?page=browseSessions&print=head&mode=list) and see what the HTML structure looks like.![Finding individual events in the iConference program page](iconf-events.png)If you look carefully you can see the tag `` indentifies each row in the table of events. We can use that to select only the information we want from the rest of the page.<jupyter_code># crafting an XPATH selector that extracts the <b> tags 
# that are children of a <tr> tag with a CSS class equal to 'whitebg'
# this basically finds the event titles from the page
selector = "//tr[@class='whitebg']//b"

# pull out all the event titles from the HTML document
event_list = iconf_program.xpath(selector)

# loop over the extracted information and print the event title
for event in event_list:
    print(event.text_content())<jupyter_output>Workshop 8: Values as Generative Forces in Design (Session 1)
Workshop 9: Visualization Pedagogy in iSchools (Session 1)
Undergraduate Education in iSchools (Session 1)
Workshop 4: Digital Youth Research Network: Defining The Field, Building Connections, and Exploring Collaborations (Session 1)
Workshop 5: ICT for Sustainability â Current and future research directions (Session 1)
Workshop 2: Exploring Gender, Race, and Sexuality with Social Media Data (Session 1)
Workshop 6: Sociotechnical Approaches to Fieldwork and Trace Data Integration (Session 1)
Workshop 7: Authoring, Designing, and Delivering Ebooks: A Research and Practice Agenda (Session 1)
Workshop 1: Trace Ethnography Workshop (Session 1)
Break (Tuesday AM)
Workshop 8: Values as Generative Forces in Design (Session 2)
Workshop 9: Visualization Pedagogy in iSchools (Session 2)
Undergraduate Education in iSchools (Session 2)
Workshop 4: Digital Youth Research Network: Defining The Field, Building Connections, and Exploring [...]<jupyter_text>**Awesome!**But now everything is mooshed together into a undifferentiated pile. Lets see if we can separate the different events.![See how background color is used to distinguish events](distinguish-events.png)Different events have different background colors. Those colors are defined using the CSS `background` attribute. We can use this information to specifically extract each of the different event types.<jupyter_code># list the completed papers by selecting rows with the background color e0ffff
selector = "//td[@style='background:#e0ffff']/a"
for row in iconf_program.xpath(selector):
    print(row.text_content())
# list the workshops by selecting rows with the background color f0f8ff

selector = "//td[@style='background:#f0f8ff']/a"
for row in iconf_program.xpath(selector):
    print(row.text_content())
# list the keynotes by selecting rows with the background color fff8cd

selector = "//td[@style='background:#fff8cd']/a"
for row in iconf_program.xpath(selector):
    print(row.text_content())
# list the preliminary papers by selecting rows with the background color fff5ee

selector = "//td[@style='background:#fff5ee']/a"
for row in iconf_program.xpath(selector):
    print(row.text_content())<jupyter_output>Preliminary Papers 1: Sound, Language and Culture
Preliminary Papers 2: Big Data, Big Infrastructure, Big Knowledge
Preliminary Papers 3: Visualization and Interaction
Preliminary Papers 4: Government Related Infrastructure
Preliminary Papers 5: Thematic and Meta Analysis
Preliminary Papers 6: Improving the Academy
Preliminary Papers 7: Voice of the Underrepresented
Preliminary Papers 8: Knowledge Management in Academia
Preliminary Papers 9: The Other Side of Social Media
Preliminary Papers 10: Academic Infrastructures and Knowledge Management<jupyter_text>Extracting Author names Ok, now let's do something more interesting and extract the names of all the authors who wrote completed papers. Bu the information is not on this page! We need to follow links to the completed papers sessions. This means we need to make a very basic web crawler!![Inspecting a completed papers session page](paper-session.png)<jupyter_code># extract the links to just the paper sessions
selector = "//td[@style='background:#e0ffff']/a/@href"

# loop over all of the links
for url in iconf_program.xpath(selector):
    # fetch and parse each of the paper session pages
    paper_session = html.fromstring(requests.get(url).content)
    
    # extract the author names from the session pages
    author_selector = "//p[@class='paper_author']"
    author_elements = paper_session.xpath(author_selector)
    
    # loop over all the authors & print out the names
    for author in author_elements:
        print(author.text_content())
        
    
    
# lets do it again, but concatinate them all together into a variable
author_string = ""

# fetch the links to paper sessions
selector = "//td[@style='background:#e0ffff']/a/@href"
for url in iconf_program.xpath(selector):
    paper_session = html.fromstring(requests.get(url).content)
    author_selector = "//p[@class='paper_author']"
    author_elements = paper_session.xpath(author_selector)
    for author in author_elements:
        author_string += author.text_content() + ","
        
author_string # list of all the authors separated by commas<jupyter_output><empty_output><jupyter_text>DATA CLEANING!<jupyter_code>author_string.split(',')

cleaned_authors = [] # for storing the cleaned authornames

for author in author_string.split(',')[:-1]: # skip last element
    if author[-1].isdigit(): # is the last char is a digit
        if author[:-1]: # is the entry only a digit
            cleaned_authors.append(author[:-1]) # save the clean author
    else: # this author doesn't need cleaning
        cleaned_authors.append(author) 
        
cleaned_authors
# ok we hae a nice list, now let's see if we can sort by last name
authors_lastname = []

for author in cleaned_authors:
    split_author = author.split(' ')
    author_lastname = ",".join([split_author[-1]," ".join(split_author[:-1])])
    authors_lastname.append(author_lastname)

authors_lastname
sorted(authors_lastname)
# remove the duplicate authors with set operations and print out a nice list
print("There are", len(set(authors_lastname)), "authors of completed papers.\n")
for author in sorted(set(authors_lastname)):
    print(author)<jupyter_output>There are 127 authors of completed papers.

<NAME>
<NAME>
<NAME>
<NAME>
<NAME>
<NAME>
<NAME>
Boden, Alexander
Bogers, Toine
Bowker, <NAME>.
Bronstein, Jenny
Brooks, <NAME>
Carlyle, Allyson
Carrington, Patrick
Carroll, <NAME>.
Carter, Daniel
Centivany, <NAME>
Chen, Feng
Chen, Miao
Cheng, James
Choi, Heekyung
Choudhury, Munmun De
Cotten, S.R.
Cotten, Shelia
Cronholm, Stefan
Dai, Bin
Dang, T<NAME>
Darch, <NAME>.
Dedrick, Jason
Dessne, Karin
Doty, Colin
Dumas, Catherine
Edelblute, Trevor
Eikey, Elizabeth
Ekbia, <NAME>.
Fonseca, Frederico
Foster, Jonathan
Gabb, <NAME>.
Gasson, Susan
Gipp, Bela
Gonzalez-IbaÃ±ez, <NAME>
Grace, Rob
Guo, Siyuan
Gustafsson, Eva
Hagen, Loni
Han, Kyungsik
Harrison, <NAME>.
Hartmann, Sarah
Hjalmarsson, Anders
Hosmer, Shannon
Huang, Kuo-Ting
Huang, Yun
Hurst, Amy
Ji, Cheng
Jiang, Mengtian
Joshi, Ritu Virendra
Kane, Sh<NAME>.
Kim, Jinyoung
Kitzie, Vanessa
Koizumi, Masanori
Kotfi[...]
{'fraction_non_alphanumeric': 0.08503519095419407, 'fraction_numerical': 0.009691934925579786, 'mean_word_length': 5.290275761973875, 'pattern_counts': {'":': 0, '<': 42, '<?xml version=': 0, '>': 42, 'https://': 5, 'lorem ipsum': 0, 'www.': 5, 'xml': 2}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '607727', 'license': 'CC-BY-2.0', 'n_tokens_mistral': 2663, 'n_tokens_neox': 2524, 'n_words': 982, 'path': 'Web Scraping Example.ipynb', 'repo_name': 'DSSatPitt/web-scraping-workshop-2015', 'size': '8757'}