<jupyter_start><jupyter_text>Exploring the requests-html capabilities<jupyter_code># Official documentation of the requests-html package: https://requests-html.readthedocs.io/en/latest/<jupyter_output><empty_output><jupyter_text>Initial setup<jupyter_code># Loading the necessary packages
from requests_html import HTMLSession
# establish/open a session
session = HTMLSession()
# submitting a GET request
r = session.get("https://en.wikipedia.org/wiki/Association_football")
r.status_code
# The html response to the GET request is contained in the '.html' method
r.html<jupyter_output><empty_output><jupyter_text>Links<jupyter_code># We can extract all link addresses directly with '.links'
urls = r.html.links
urls
# Note that those are the relative URLs 
# To get absolute URLs we can use '.absolute_links' instead of '.links'
full_path_urls = r.html.absolute_links
full_path_urls
# An important thing to note is that these links (given by both methods) are returned in a SET, not a LIST
type(urls)<jupyter_output><empty_output><jupyter_text>Searching for elements<jupyter_code># A quick note: requests-html uses CSS selectors for searching
# We will cover them in the next section,
# but here is a more thorough look into it: https://www.w3schools.com/cssref/css_selectors.asp
# We can search for elements similarly to Beautiful Soup using the find() method
# It behaves as find_all()

# find all 'a' tags
links = r.html.find("a")
links
links[4]
# To get the raw HTML of an element use the '.html' method
links[4].html
type(links[4].html)
# To extract the text inside an element, use ".text", just like in Beautiful Soup
links[4].text
# To obtain a dictionary of the element's attributes, use '.attrs' (exactly as in Beautiful Soup)
links[10].attrs
# This package offers a couple of ways to filter tags based off text

# Choose only those tags that contain the string 'wikipedia' in their text (not in the 'href' attribute)
# Note: this is not case-sensitive
r.html.find("a", containing = "wikipedia")
# display the text of those tags
[tag.text for tag in r.html.find("a", containing = "wikipedia")]
# If we wish to find only the first element (similarly to Beautiful Soup .find()) we need to specify the 'first' parameter
r.html.find("p", first = True)<jupyter_output><empty_output><jupyter_text>Searching for text<jupyter_code># The package also offers searching text based on the parse library
# The search() method can be thought of as the opposite of str.format():
# it finds the text instead of inserting it in the specified place

# For further details see https://pypi.org/project/parse/ 
# and https://docs.python.org/3/library/string.html#format-string-syntax
# The method searches for a matching string, where '{}' is replaced by the returned text
r.html.search("known{}soccer")
# To access the text, get the first element (index 0)
r.html.search("known{}soccer")[0]
# search() finds only the shortest matching string
# To search for all matching strings use search_all()
r.html.search_all("known{}soccer")
len(r.html.search_all("known{}soccer"))
# Further details at:
# -- https://pypi.org/project/parse/
# -- https://docs.python.org/3/library/string.html#format-string-syntax<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10245265445513815, 'fraction_numerical': 0.0034150884818379386, 'mean_word_length': 5.29296875, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 7, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '612368', 'license': 'MIT', 'n_tokens_mistral': 974, 'n_tokens_neox': 951, 'n_words': 367, 'path': 'Section 10/S10_L58/Section 10 - Searching for text.ipynb', 'repo_name': 'guilhermeaugusto9/Web-Scraping-and-API-Fundamentals-in-Python', 'size': '3221'}