<jupyter_start><jupyter_text>Parsing HTML with BeautifulSoup BeautifulSoup is a Python library for extracting data from HTML or XML files. Refer to the [documentation](https://www.crummy.com/software/BeautifulSoup/bs4/doc/) for more information.<jupyter_code># pipenv shell
import requests
from bs4 import BeautifulSoup
url = "https://www.bbc.com/pidgin/topics/c2dwqd1zr92t"
response = requests.get(url)
print(dir(response))   # dir allows us to check the methods available for an object
page_html = response.content
print(page_html)
print(type(page_html))   # Without bs4, we obtain a string object which is tedious to interact with
page_soup = BeautifulSoup(page_html, "html.parser") 
print("type: ",type(page_soup))
print(f"page soup: {page_soup}")<jupyter_output><empty_output><jupyter_text>BeautifulSoup allows us to use a variety of parsers. You can check this out :)<jupyter_code>print(len(page_soup))  # we get a rich list of methods available for our BeautifulSoup object<jupyter_output>3<jupyter_text>**`TODO`**: Complete the `get_page_soup` function in `scraper.py`  Locating elements  1. Locating elements by tag<jupyter_code>all_urls = page_soup.findAll("a")
print(len(all_urls))
print(type(all_urls[0]))
print(dir(all_urls[0]))
all_urls[0].get("href")     # hrefs may be absolute
all_urls[12].get("href")     # hrefs may also be relative
all_urls<jupyter_output><empty_output><jupyter_text>2. Locating elements by tag & class To obtain the class of an element ona web page, right click the element and select `Inspect`. ![InspectElement](static/inspect_element.png)<jupyter_code>headline = page_soup.find(
        "h1", attrs={"class": "bbc-13dm3d0 e1yj3cbb0"}
        )<jupyter_output><empty_output><jupyter_text>How many pages of article are there in a category?**`TODO`**: Go to any of the category pages of BBC Pidgin. Can you retrieve the `class` of the `span` element that contains the number of total pages as text? In the picture below, for example, retrieve the `class` of the `span` that contains `100` as text. ![PageNumber](static/PageNumber.png)  What is a valid article link? Website may have a pattern for successive pages in a category. This eases our work. Check through a number of articles. Do you notice a pattern to the article URLS?<jupyter_code>for url in all_urls:
    href = url.get("href")
    
    if (href.startswith("/pidgin/tori") or \
        href.startswith("pidgin/world") or \
        href.startswith("pidgin/sport")) and href[-1].isdigit():
        print(href)<jupyter_output>/pidgin/tori-58745492
/pidgin/tori-58745492
/pidgin/tori-58745492
/pidgin/tori-58738377
/pidgin/tori-58738377
/pidgin/tori-58738377
/pidgin/tori-58709437
/pidgin/tori-58709437
/pidgin/tori-58709437
/pidgin/tori-58736254
/pidgin/tori-58736254
/pidgin/tori-58736254
/pidgin/tori-58732508
/pidgin/tori-58732508
/pidgin/tori-58732508
/pidgin/tori-58731043
/pidgin/tori-58731043
/pidgin/tori-58731043
/pidgin/tori-58731038
/pidgin/tori-58731038
/pidgin/tori-58731038<jupyter_text>valid_url = "https://www.bbc.com" + href<jupyter_code>base_url = "https://www.bbc.com"
article = requests.get(# insert a valid URL here) 
#<jupyter_output><empty_output><jupyter_text>**`TODO`**: Complete the `get_valid_urls` and `get_urls` functions in `scraper.py`  Getting article text<jupyter_code>response = requests.get("https://www.bbc.com/pidgin/sport-58110814")
page_html = response.text
page_soup = # TODO: Create a BeautifulSoup object from `page_html`<jupyter_output><empty_output><jupyter_text>Text on websites are frequently embedded within one or more `div` elements. Open up the article we requested above in your browser.`Inspect` any paragraph from the article. What do you notice about the `div` elements? What is the value of their `class` attribute?<jupyter_code>story_div = page_soup.find_all(
        "div", attrs={"class": "bbc-19j92fr e57qer20"}
        )
story_div<jupyter_output><empty_output><jupyter_text>Inside `div` elements, text is usually written in `p` elements. BeautifulSoup allows us to search the `div` element we retrieved above for elements it contains.<jupyter_code>p_elements = [div.findAll("p") for div in story_div]
p_elements
p_elements[0]
p_elements[0][0]
p_elements[0][0].text<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11215172090845235, 'fraction_numerical': 0.05057363615078436, 'mean_word_length': 5.3666169895678095, 'pattern_counts': {'":': 2, '<': 35, '<?xml version=': 0, '>': 35, 'https://': 5, 'lorem ipsum': 0, 'www.': 5, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '636649', 'license': 'MIT', 'n_tokens_mistral': 1625, 'n_tokens_neox': 1500, 'n_words': 396, 'path': '2_Webscraping_and_Data_Labelling/scraper.ipynb', 'repo_name': 'hokagedemehin/cohort7_practicals', 'size': '4271'}