<jupyter_start><jupyter_text>Goal: Exploring web scrapping Author: <NAME>Python version 3.6 Content1. Sample html webpage example2. Exploring features of BeautifulSoup3. Exploring html_read in Pandas. Direct table creation 4. Exploring requests Module-get call. Authentication requirement5. Exploring urllib.request's function urlopen5. Write Pandas DataFrame in an HTML file6. Open action items (that can be done)7. References<jupyter_code>from bs4 import BeautifulSoup
from pandas import DataFrame
from pandas import read_html

from requests import get
from urllib.request import urlopen

from pandas import set_option
set_option('display.max_row', 5)
set_option('display.max_columns', 5)
set_option('display.max_colwidth', 40)<jupyter_output><empty_output><jupyter_text>Install following packages if not present!pip install lxml!pip install html5lib   1. sample HTML page used:  Table 1 in sample HTML code looks like this:  2. Exploring features of BeautifulSoup<jupyter_code># Open HTML file. Read the content.
HtmlFile = open('WebpageWithTable.html', 'r')
html_source_code = HtmlFile.read() 
HtmlFile.close()
html_source_code
soup = BeautifulSoup(html_source_code, 'html.parser') 
#Creating Beutifulsoup object. html.parser  or lxml is the html parser
#type(soup)=> bs4.BeautifulSoup 
# the soup object allows us to extract different information egarding the website/webpage
soup
htmltagAllReviewBlocks = soup.find_all('table', {"class": "table table-bordered"})
#Extract block containing tables 

tempTotalNumberOfReviews = len(htmltagAllReviewBlocks)
print('We have total number of tables found on this page:', tempTotalNumberOfReviews)
htmltagAllReviewBlocks
pgTitleName = soup.title.text
print(f"Page title is: {pgTitleName}")
# all tables
len(htmltagAllReviewBlocks)
html_table = htmltagAllReviewBlocks[0] # Extract table 1
html_table
# access body section of each table. filter out header.
tbody = htmltagAllReviewBlocks[0].find_all('tbody')
tbody
intNumOfRows = len(tbody[0].find_all('tr'))
tdAllRows = tbody[0].find_all('td')
tdAllRows
for row in tdAllRows:
    print(row.text)
DataFrame(tdAllRows) # if created dataframe directly we get. ( not desired output )<jupyter_output><empty_output><jupyter_text>Exploring Pandas (html_read) to read the HTML file and create the tables<jupyter_code>url = 'WebpageWithTable.html'
lstAllTables = read_html(url)
lstAllTables
pdTable1 = lstAllTables[0]
pdTable2 = lstAllTables[1]
pdTable3 = lstAllTables[2]
pdTable1
pdTable2
pdTable3<jupyter_output><empty_output><jupyter_text>Try to Read tables on the webpage directly  Let's read simple table directly from the website<jupyter_code>tables = read_html('https://limitlessdatascience.wordpress.com/deep-learning/')
print('Tables found:', len(tables))  
tables[0]<jupyter_output>Tables found: 1<jupyter_text>Let's ty to read more complex webppage containing multiple tables<jupyter_code>tables = read_html('https://en.wikipedia.org/wiki/Python_(programming_language)')
print('Tables found:', len(tables))  
tables[0] # Display table 1
tables[1] # Display table 2
tables[2]
tables[3]
tables[4]<jupyter_output><empty_output><jupyter_text>Exploring requests get call- Authentication requirement Reading HTML data from URL requires authentication- When the site requires authentication, the code runs into the following exception:    - raise HTTPError(req.full_url, code, msg, hdrs, fp) urllib.error.HTTPError: HTTP Error 401: UNAUTHORIZED- To read data from such URLs can use the requests module. - You can install it with pip: ! pip install requests- Use the get() method from the requests library to make a request to a website URL while providing the optional auth parameter if the site requires authentication.<jupyter_code>%time response = get('https://en.wikipedia.org/wiki/Python_(programming_language)')
print(response) # <Response [200]> shows success
# printing content
response.text
lstTables=read_html(response.text)
print('Tables found:', len(lstTables)) 
type(lstTables)
lstTables[1]<jupyter_output><empty_output><jupyter_text>To use **authentication** feature of requests pass two additional arguments in the get request: username and password as follows:get('url', 'username', 'password')  Exploring urllib.request's function urlopen- Python’s standard library is urllib, which contains tools for working with URLs. In particular, the urllib.request module contains a function called urlopen() that can be used to open a URL within a program.<jupyter_code>%time response = urlopen('https://en.wikipedia.org/wiki/Python_(programming_language)')
print(response) # <Response [200]> shows success
type(response)
# Therefore parse it first
soup = BeautifulSoup(response, 'lxml') 
response1 = soup.text
response1<jupyter_output><empty_output><jupyter_text>lstTables=read_html(response1)              This will not work in this case it displays: ValueError: No tables foundprint('Tables found:', len(lstTables))<jupyter_code>%time response = urlopen('https://en.wikipedia.org/wiki/Python_(programming_language)')
response2 = response.readlines()
response2
type(response2)<jupyter_output><empty_output><jupyter_text>lstTables=read_html(response1)           This will not work in this case it displays: ValueError: No tables foundprint('Tables found:', len(lstTables))   Write Pandas DataFrame in an HTML file- The **to_html()** takes the path of the file you want the data exported to. If you don't provide an absolute path, it would save a file relative to the current directory.<jupyter_code>#Create new HTML page and export a DataFrame to an HTML table:
lstTables[1].to_html("ExportingDataToNewHTML.html")<jupyter_output><empty_output><jupyter_text>HTML output looks like this:<jupyter_code>Output = read_html("ExportingDataToNewHTML.html")
Output[0]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09035936420179683, 'fraction_numerical': 0.010711817553559088, 'mean_word_length': 6.7913862718707945, 'pattern_counts': {'":': 1, '<': 43, '<?xml version=': 0, '>': 44, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 3}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '141623', 'license': 'MIT', 'n_tokens_mistral': 1758, 'n_tokens_neox': 1710, 'n_words': 554, 'path': 'Reading tables from HTML page/Exploring Web Scrapping -BeautifulSoup, Pandas, Table reading writing (from and to HTML).ipynb', 'repo_name': 'Yogeshnaik1190/Web-Scrapping', 'size': '5802'}