<jupyter_start><jupyter_text>Useful reference: > https://lxml.de/lxmlhtml.htmlxpath = '//p'- all Ps that are children of the body xpath = '/html/body/div/p'- the Ps that are found as children of this divxpath = '/html/body/div/p[2]'- the second P that is a child of the divxpath = '/html/body/*'- this gives us all the children of the body, regardless of type"@class""@id""@href"xpath = "//p[@class='class-1']"- all the Ps, but just those of this classxpath = '//*[id="uid"]'- all elements in the html, then just those that have ID 'uid'xpath = '//div[@id="uid"]/p[2]'- get all the divs that have a UID id, then find the 2nd P in that  Start<jupyter_code># lxml library will allow us to parse the xpath

import requests
from lxml import html

from bs4 import BeautifulSoup as bs

# use the lxml module to produce a 'response'. You could create a soup here at the same time, then you could parse either way

page = "https://www.cardiff.ac.uk/journalism-media-and-culture/people/academic-staff"
req = requests.get(page)
print(req)

response = html.fromstring(req.content)
type (response)
# find the title of the page

response.xpath('/html/head/title/text()')
# find all the h2s - note the //

print(len(response.xpath('//h2')))

# and we can specify those of a certain class

print(len(response.xpath('//h2[@class="profile-title fn"]')))

# what ARE these things?
print ('***********')
print(type(response.xpath('//h2')))
print(type(response.xpath('//h2')[4]))
print(type(response.xpath('//h2[@class="profile-title fn"]')[4]))
# what methods are available to one of these list items? We can call 'dir' on ti

my_resp = response.xpath('//h2[@class="profile-title fn"]')[4]
dir(my_resp)

# we can collect the text from all the h2s of the names via their class

for x in response.xpath('//h2[@class="profile-title fn"]'):
    print(x.text_content())
# or the overall solution via the divs

for x in response.xpath('//div[@class="profile profile-student with-image vcard"]'):
    print(x.text_content())<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15433925049309666, 'fraction_numerical': 0.008382642998027613, 'mean_word_length': 4.513586956521739, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 6, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 5}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '597593', 'license': 'CC-BY-4.0', 'n_tokens_mistral': 691, 'n_tokens_neox': 676, 'n_words': 233, 'path': 'wk_09_scraping/simple_xpath_example_(Jomec).ipynb', 'repo_name': 'aodhanlutetiae/data_journalism_jomec', 'size': '2028'}