<jupyter_start><jupyter_text>Python web crawling example This tutorial is written based on chapters 11-13 from the book "Python for Everyone" https://www.py4e.com Step 1: import all necessary packages<jupyter_code>import re
import urllib
from bs4 import BeautifulSoup
import pprint
import pandas as pd<jupyter_output><empty_output><jupyter_text>Step 2: download a sample webpage. You can save the html page onto your computer and use text editor to view its content<jupyter_code>url = "http://www.metrolyrics.com/you-belong-with-me-lyrics-taylor-swift.html"
html = urllib.request.urlopen(url).read()<jupyter_output><empty_output><jupyter_text>Step 3: use BeautifulSoup to parse the webpage and extract the lyrics content. The division that includes the lyrics starts from the html tag "lyrics-body-text"<jupyter_code>soup = BeautifulSoup(html, 'html.parser')
print(soup.title.string)

text = soup.body.find_all(id='lyrics-body-text')
text = text[0].text
print(text)<jupyter_output><NAME> - You Belong With Me Lyrics | MetroLyrics


You're on the phone with your girlfriend—she's upset
She's going off about something that you said
'Cause she doesn't get your humor like I do.I'm in the room, it's a typical Tuesday night.
I'm listening to the kind of music she doesn't like.
And she'll never know your story like I doBut she wears short skirts
I wear t-shirt
She's cheer captain
And I'm on the bleachersDreaming about the day when you wake up and find
That what you're looking for has been here the whole time.



Related




 






11 Delicious Misheard Lyrics About Food







NEW SONG: <NAME> - 'Lover' - LYRICS







Prime Day concert by Amazon Music will be headlined by <NAME>






 
 

If you could see
That I'm the one
Who understands you
Been here all along
So, why can't you see
You belong with me
You belong with me?Walk in the streets with you in your worn out jeans
I can't help thinking this is how it ought to be.
Laughing on a park bench think[...]<jupyter_text>Step 4: split text into individual words<jupyter_code>words = text.split()
print(words)<jupyter_output>["You're", 'on', 'the', 'phone', 'with', 'your', "girlfriend—she's", 'upset', "She's", 'going', 'off', 'about', 'something', 'that', 'you', 'said', "'Cause", 'she', "doesn't", 'get', 'your', 'humor', 'like', 'I', "do.I'm", 'in', 'the', 'room,', "it's", 'a', 'typical', 'Tuesday', 'night.', "I'm", 'listening', 'to', 'the', 'kind', 'of', 'music', 'she', "doesn't", 'like.', 'And', "she'll", 'never', 'know', 'your', 'story', 'like', 'I', 'doBut', 'she', 'wears', 'short', 'skirts', 'I', 'wear', 't-shirt', "She's", 'cheer', 'captain', 'And', "I'm", 'on', 'the', 'bleachersDreaming', 'about', 'the', 'day', 'when', 'you', 'wake', 'up', 'and', 'find', 'That', 'what', "you're", 'looking', 'for', 'has', 'been', 'here', 'the', 'whole', 'time.', 'Related', '11', 'Delicious', 'Misheard', 'Lyrics', 'About', 'Food', 'NEW', 'SONG:', 'Taylor', 'Swift', '-', "'Lover'", '-', 'LYRICS', 'Prime', 'Day', 'concert', 'by', 'Amazon', 'Music', 'will', 'be', 'headlined', 'by', 'Taylor', 'Swift', 'If', 'you', 'could'[...]<jupyter_text>Remove stopwords<jupyter_code>stopwords = ['is', 'are', 'the', 'a', 'an']
def removeStopwords(wordlist, stopwords):
    return [w for w in wordlist if w not in stopwords]
words = removeStopwords(words, stopwords)
print(words)<jupyter_output>["You're", 'on', 'phone', 'with', 'your', "girlfriend—she's", 'upset', "She's", 'going', 'off', 'about', 'something', 'that', 'you', 'said', "'Cause", 'she', "doesn't", 'get', 'your', 'humor', 'like', 'I', "do.I'm", 'in', 'room,', "it's", 'typical', 'Tuesday', 'night.', "I'm", 'listening', 'to', 'kind', 'of', 'music', 'she', "doesn't", 'like.', 'And', "she'll", 'never', 'know', 'your', 'story', 'like', 'I', 'doBut', 'she', 'wears', 'short', 'skirts', 'I', 'wear', 't-shirt', "She's", 'cheer', 'captain', 'And', "I'm", 'on', 'bleachersDreaming', 'about', 'day', 'when', 'you', 'wake', 'up', 'and', 'find', 'That', 'what', "you're", 'looking', 'for', 'has', 'been', 'here', 'whole', 'time.', 'Related', '11', 'Delicious', 'Misheard', 'Lyrics', 'About', 'Food', 'NEW', 'SONG:', 'Taylor', 'Swift', '-', "'Lover'", '-', 'LYRICS', 'Prime', 'Day', 'concert', 'by', 'Amazon', 'Music', 'will', 'be', 'headlined', 'by', 'Taylor', 'Swift', 'If', 'you', 'could', 'see', 'That', "I'm", 'one', 'Who', 'understa[...]<jupyter_text>count word frequency<jupyter_code>counts = dict()
for word in words:
    counts[word] = counts.get(word,0) + 1
sorted(counts, key=counts.__getitem__, reverse=True)
pprint.pprint(counts)<jupyter_output>{'"Hey,': 1,
 "'Cause": 1,
 "'Lover'": 1,
 "'bout": 1,
 '-': 2,
 '11': 1,
 'About': 1,
 'All': 2,
 'Amazon': 1,
 'And': 4,
 'Been': 3,
 'Day': 1,
 'Delicious': 1,
 'Food': 1,
 'Have': 1,
 'Hey,': 1,
 'I': 9,
 "I'm": 7,
 'If': 1,
 'In': 1,
 'LYRICS': 1,
 'Laughing': 1,
 'Lyrics': 1,
 'Misheard': 1,
 'Music': 1,
 'NEW': 1,
 'Prime': 1,
 'Related': 1,
 'SONG:': 1,
 'She': 1,
 "She's": 3,
 'Since': 1,
 'So,': 3,
 'Swift': 2,
 'Taylor': 2,
 'That': 5,
 'Think': 2,
 'Tuesday': 1,
 'When': 1,
 'Who': 3,
 'You': 11,
 "You're": 1,
 'about': 4,
 'all': 3,
 'along': 3,
 'and': 4,
 'at': 2,
 'awhile': 1,
 'baby?': 2,
 'backdoor.': 2,
 'be': 1,
 'be.': 1,
 'been': 2,
 'belong': 12,
 'bench': 1,
 'better': 1,
 'bleachersDreaming': 2,
 'brought': 1,
 'by': 4,
 'can': 1,
 "can't": 4,
 'captain': 2,
 'cheer': 2,
 'concert': 1,
 'could': 4,
 'cryI': 1,
 'day': 2,
 "do.I'm": 1,
 'doBut': 1,
 "doesn't": 2,
 'doing': 1,
 'down.You': 1,
 'dreams': 1,
 'driving': 1,
 'easy?"And': 1,
 'ever': 1,
 'favorite': [...]<jupyter_text>sort words by frequency The above method uses loop, which needs quite a lot of programming, and is also slow. The following method uses the dataframe data structure in the pandas package to quickly count and sort words by frequencies. Pandas documentation includes more details on its powerful data structurehttps://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.html<jupyter_code>df=pd.DataFrame(words, columns=['word'])
x=df["word"].value_counts()
pprint.pprint(x)
import nltk
tokens = nltk.word_tokenize(text)
tags = nltk.pos_tag(tokens)
print(tags[0][0], tags[0][1])<jupyter_output>You PRP<jupyter_text>Remove Stopwords<jupyter_code>def removeStopwords(wordlist, stopwords):
    return [w for w in wordlist if w not in stopwords]
words = removeStopwords(words, stopwords)
print(words)
counts = dict()
for word in words:
    counts[word] = counts.get(word, 0) + 1
sorted(counts, key=counts.__getitem__, reverse=True)
pprint.pprint(counts)
df = pd.DataFrame(words, columns=['word'])
x = df["word"].value_counts()
pprint.pprint(x)<jupyter_output>you          23
with         15
belong       12
You          11
I             9
             ..
listening     1
short         1
do.I'm        1
time.         1
just          1
Name: word, Length: 186, dtype: int64
{'fraction_non_alphanumeric': 0.20988194140795802, 'fraction_numerical': 0.018073167176796384, 'mean_word_length': 4.835034013605442, 'pattern_counts': {'":': 9, '<': 30, '<?xml version=': 0, '>': 30, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '665650', 'license': 'MIT', 'n_tokens_mistral': 2782, 'n_tokens_neox': 2523, 'n_words': 815, 'path': 'assets/all_html/2019_10_15_python-web-crawling-example.ipynb', 'repo_name': 'dskw1/dskw1.github.io', 'size': '6879'}