<jupyter_start><jupyter_text>requests 연습<jupyter_code>requests.get("https://www.google.com/")
url = "https://www.google.com"
response = requests.get(url)
response
response.text[:20]
response.encoding
response.headers['Content-Type']
url = "https://www.theguardian.com/uk/technology"
response = requests. get(url)
response.text[:100]
html = response.text<jupyter_output><empty_output><jupyter_text>BeautifulSoup<jupyter_code>soup = BeautifulSoup(html, 'html.parser') # 'html.parser', 'lxml'
# <ul class = "headline-list">
ul = soup.find('ul', attrs={'class':'headline-list'}) 
#find():해당값중 첫번째만 return, find_all(): 해당하는 모든 값을 list로 return
lis = ul.find_all('li')
lis[9]
#lis[0].find('a').span.span.text
news_title = lis[0].find('a').text.strip()
news_title
lis[0].find('a').attrs
news_link = lis[0].find('a').attrs['href']
news_link
results = []
for li in lis:
    news_title = li.find('a').text.strip()
    news_link = li.find('a').attrs['href']
    print(news_title, news_link)
    results.append((news_title, news_link))
    
results<jupyter_output>Apple 12.9in iPad Pro review: bringing back the wow factor https://www.theguardian.com/technology/2018/dec/12/apple-129in-ipad-pro-review-bringing-back-the-wow-factor
Broadband too slow in more than a quarter of UK homes – report https://www.theguardian.com/technology/2018/dec/12/broadband-too-slow-in-more-than-a-quarter-of-uk-homes-report
<NAME>: Canadian court frees Huawei CFO on bail https://www.theguardian.com/technology/2018/dec/12/meng-wanzhou-canada-frees-huawei-cfo-on-bail
Revealed: Google's 'two-tier' workforce training document https://www.theguardian.com/technology/2018/dec/11/google-tvc-full-time-employees-training-document
Why Facebook's new rule about sex is its weirdest yet https://www.theguardian.com/technology/commentisfree/2018/dec/11/why-facebooks-new-rule-about-sex-is-its-weirdest-yet
Google CEO Sundar Pichai refuses to rule out censored Chinese search engine https://www.theguardian.com/technology/2018/dec/11/google-ceo-sundar-pichai-congress
Encryption laws w[...]<jupyter_text>Rotten tomatoes<jupyter_code>url = "https://www.rottentomatoes.com"
response = requests.get(url)    
html = response.text
soup = BeautifulSoup(html, 'html.parser')
table = soup.find('table', attrs = {'id' : "Top-Box-Office"})
table
trs = table.find_all('tr')
trs
rotten_index = trs[0].find('td', attrs = {'class' : 'left_col'}).text.strip("\n%")
rotten_index
rotten_title = trs[0].find('td', attrs = {'class' : 'middle_col'}).text
rotten_title
rotten_url = trs[0].find('a').attrs['href']
rotten_url
rotten_revenue = trs[0].find('td', attrs = {'class' : 'right_col'}).text.strip("\n")
rotten_revenue
results = []
for tr in trs:
    rotten_index = tr.find('td', attrs = {'class' : 'left_col'}).text.strip("\n%")
    rotten_title = tr.find('td', attrs = {'class' : 'middle_col'}).text.strip("\n")
    rotten_url = tr.find('a').attrs['href']
    rotten_revenue = tr.find('td', attrs = {'class' : 'right_col'}).text.strip("\n")
    
    print(rotten_index, rotten_title, url+rotten_url, rotten_revenue)
    results.append((rotten_index, rotten_title, url+rotten_url, rotten_revenue))
    
results<jupyter_output>88 Ralph Breaks the Internet https://www.rottentomatoes.com/m/ralph_breaks_the_internet $16.4M
57 The Grinch https://www.rottentomatoes.com/m/the_grinch $15.1M
83 Creed II https://www.rottentomatoes.com/m/creed_ii $10.1M
38 Fantastic Beasts: The Crimes of Grindelwald https://www.rottentomatoes.com/m/fantastic_beasts_the_crimes_of_grindelwald $7M
62 Bohemian Rhapsody https://www.rottentomatoes.com/m/bohemian_rhapsody $6.2M
81 Instant Family https://www.rottentomatoes.com/m/instant_family $5.8M
82 Green Book https://www.rottentomatoes.com/m/green_book $4M
16 Robin Hood https://www.rottentomatoes.com/m/robin_hood_2018 $3.5M
14 The Possession of Hannah Grace https://www.rottentomatoes.com/m/the_possession_of_hannah_grace $3.3M
91 Widows https://www.rottentomatoes.com/m/widows_2018 $3.1M
{'fraction_non_alphanumeric': 0.16170534813319878, 'fraction_numerical': 0.026488395560040363, 'mean_word_length': 5.37459807073955, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 20, 'lorem ipsum': 0, 'www.': 20, 'xml': 1}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '383498', 'license': 'MIT', 'n_tokens_mistral': 1567, 'n_tokens_neox': 1505, 'n_words': 256, 'path': 'python/notes/Day13. Static Page Scraping.ipynb', 'repo_name': 'mindudekim/TIL', 'size': '3970'}