<jupyter_start><jupyter_text>The pupose of this notebook is to illustrate the use of a webscraper in Python 3 to gather text based data.The tutorial will be based on the below link and will be extended to other websites to gather information of interest.https://www.dataquest.io/blog/web-scraping-tutorial-python/<jupyter_code># -*- coding: utf-8 -*-
"""
Created on Tue Oct 30 13:12:50 2018
@author: <NAME>
"""

#import libraries
import requests
from bs4 import BeautifulSoup<jupyter_output><empty_output><jupyter_text>A GET request fetches the whole HTML page.<jupyter_code>#Send GET HTML request  to get webpage content
page = requests.get("http://dataquestio.github.io/web-scraping-pages/simple.html")
#check request status
page
#check the content of the webpage
page.content<jupyter_output><empty_output><jupyter_text>Beautifulsoup helps to parse and analyse the data.<jupyter_code>#Parse/structure the webpage content using Beautifulsoup
soup = BeautifulSoup(page.content, 'html.parser')

#check object strucutre
print(soup.prettify())
#move through the object one level at a time
#Note that children returns a list generator, so we need to call the list function on it

list(soup.children)
#check the items present in the list

[type(item) for item in list(soup.children)]<jupyter_output><empty_output><jupyter_text>It is noted that there are three objects from the above:* The first is a Doctype object, which contains information about the type of the document. * The second is a NavigableString, which represents text found in the HTML document. * The final item is a Tag object, which contains other nested tags. The Tag object allows us to navigate through an HTML document, and extract other tags and text.<jupyter_code>html = list(soup.children)[2]
print(html)<jupyter_output><html>
<head>
<title>A simple example page</title>
</head>
<body>
<p>Here is some simple content for this page.</p>
</body>
</html><jupyter_text>There are two tags above, head, and body. The part of interest is the "p" tag.<jupyter_code>#Find all instances of the "p" tag

soup = BeautifulSoup(page.content, 'html.parser')
soup.find_all('p')
#Since the above command retuns a list, use get_text to extract the text object

soup.find_all('p')[0].get_text()<jupyter_output><empty_output><jupyter_text>These techniques will now be extended to construct a pandas dataframe from a weather forecast page<jupyter_code>#Download the whole weather page
weatherpage = requests.get("https://forecast.weather.gov/MapClick.php?lat=37.7772&lon=-122.4168")
weatherpage
#Get appears to be successful
#check the content of the webpage

#weatherpage.content

#There is too much output, web inspector (ctrl+shift+c) in firefox will be used to identify the element of choice
#Based on indentation a tombstone container is used

#First parse the webpage

weathersoup = BeautifulSoup(weatherpage.content, 'html.parser')

#Find the seven day forecast
seven_day = weathersoup.find(id="seven-day-forecast")

#Extract all the instances of the tombstone container from the seven_day forecast
forecast_items = seven_day.find_all(class_="tombstone-container")

#Show tonight's forecast
tonight = forecast_items[0]
print(tonight.prettify())<jupyter_output><div class="tombstone-container">
 <p class="period-name">
  Today
  <br/>
  <br/>
 </p>
 <p>
  <img alt="Today: Sunny, with a high near 72. Light and variable wind becoming west 6 to 11 mph in the afternoon. " class="forecast-icon" src="newimages/medium/few.png" title="Today: Sunny, with a high near 72. Light and variable wind becoming west 6 to 11 mph in the afternoon. "/>
 </p>
 <p class="short-desc">
  Sunny
 </p>
 <p class="temp temp-high">
  High: 72 °F
 </p>
</div><jupyter_text>From the above, there are cetain elements that can be accessed:* Short description : Sunny* temp : temp-high* Forecast item/date : Tonight<jupyter_code>#From the tonight object find the related text for each of the above.
period = tonight.find(class_="period-name").get_text()
short_desc = tonight.find(class_="short-desc").get_text()
temp = tonight.find(class_="temp").get_text()

print(period)
print(short_desc)
print(temp)
#Extract the full description from the image attribute

img = tonight.find("img")
desc = img['title']

print(desc)
#Select all the days in the forecast by selecting the whole tombstone container

period_tags = seven_day.select(".tombstone-container .period-name")
periods = [pt.get_text() for pt in period_tags]
periods
#Get all the items for weather descriptions from tombstone container
short_descs = [sd.get_text() for sd in seven_day.select(".tombstone-container .short-desc")]
temps = [t.get_text() for t in seven_day.select(".tombstone-container .temp")]
descs = [d["title"] for d in seven_day.select(".tombstone-container img")]

print(short_descs)
print(temps)
print(descs)
#Combine in pandas dataframe for future analysis

import pandas as pd
weather = pd.DataFrame({
        "period": periods, 
        "short_desc": short_descs, 
        "temp": temps, 
        "desc":descs
    })
weather<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10274649278798656, 'fraction_numerical': 0.008496344595929657, 'mean_word_length': 5.351317440401505, 'pattern_counts': {'":': 4, '<': 51, '<?xml version=': 0, '>': 51, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '487526', 'license': 'MIT', 'n_tokens_mistral': 1610, 'n_tokens_neox': 1543, 'n_words': 560, 'path': 'Webcrawler_Tutorial.ipynb', 'repo_name': 'Spurryag/Webcrawlers-', 'size': '5068'}