<jupyter_start><jupyter_text>Library of Congress MODS Exercise  Lesson Outline* What is XML?* Introduction to BeautifulSoup* Parse MODS XML with BeautifulSoup* Loop through directory of XML files* Find and replace in XML files* Introduction to ElementTree * Learn how to explore XML using functions, loops, XPath  What is XML?* Let's briefly desscribe what it is.* Let's look at an example.<jupyter_code>## let's view a MODS XML file
## open file, read string into xml, import BeautifulSoup, print XML using prettify, close
## more on BeautifulSouop https://www.crummy.com/software/BeautifulSoup/
file = open('MODS/lcwaN0012195.xml', 'r')
xml = file.read()
from bs4 import BeautifulSoup
print(BeautifulSoup(xml, "xml").prettify())
file.close()<jupyter_output><?xml version="1.0" encoding="utf-8"?>
<mods version="3.4" xmlns="httpss://www.loc.gov/mods/v3" xmlns:xlink="httpss://www.w3.org/1999/xlink" xmlns:xsi="httpss://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="httpss://www.loc.gov/mods/v3 httpss://www.loc.gov/standards/mods/v3/mods-3-4.xsd">
 <identifier>
  lcwaN00 USA\
 </identifier>
 <identifier invalid="yes" type="database id">
  USA\
 </identifier>
 <titleInfo>
  <title>
   Intel Dump - Blog
  </title>
 </titleInfo>
 <language>
  <languageTerm authority="iso639-2b" type="code">
   eng
  </languageTerm>
 </language>
 <physicalDescription>
  <form authority="marcform">
   electronic
  </form>
  <internetMediaType>
   text/html
  </internetMediaType>
  <digitalOrigin>
   born digital
  </digitalOrigin>
 </physicalDescription>
 <targetAudience>
  general
 </targetAudience>
 <typeOfResource>
  text
 </typeOfResource>
 <genre authority="marcgt">
  web site
 </genre>
 <originInfo>
  <place>
   <placeTerm type="text">
    United Sta[...]<jupyter_text>What is BeautifulSoup[Beautiful Soup](https://www.crummy.com/software/BeautifulSoup/bs4/doc/) is a Python package for parsing HTML and XML documents. It creates a parse tree for parsed pages that can be used to extract data from HTML, which is useful for web scraping. It's also a widely used package, so many tutorials and examples are available. For instance, the Programming Historian has a lesson:https://programminghistorian.org/en/lessons/intro-to-beautiful-soupLet's use it to parse XML...<jupyter_code>## import BeautifulSoup, open and read XML file, use BeautifulSoup to final all title text
from bs4 import BeautifulSoup
infile = open("MODS/lcwaN0012195.xml","r")
contents = infile.read()
soup = BeautifulSoup(contents,'xml')
titles = soup.find_all('title')
for title in titles:
    print(title.get_text())<jupyter_output>Intel Dump - Blog
Iraq War 2003 Web Archive
Research and Reference Services Division<jupyter_text>Exercises* Print only the first title from the array returned by 'titles = soup.find_all('title')'.* Use the script above to parse for text from other elements.* Can you use the same script to get the text for element attributes? Try 'languageTerm' and 'authority'.* Extra credit: Can you use the same script to search by a particular attribute value? Try 'languageTerm' and 'authority' again and 'iso639-2b'.  Loop through directory of XML files<jupyter_code>## import operating system interface, BeautifulSoup, set path to files, loop through directory
## conditional for .xml, join path and filename, open and read XML file with BeautifulSoup
## find all titles in file but only print first title
import os
from bs4 import BeautifulSoup

path = 'MODS'
for filename in os.listdir(path):
    if filename.endswith('.xml'): 
        fullname = os.path.join(path, filename)
        infile = open(fullname,"r")
        contents = infile.read()
        soup = BeautifulSoup(contents,'xml')
        titles = soup.find_all('title')
        print(filename, titles[0].get_text())<jupyter_output>lcwaN0009700.xml YTMND: You're the man now dog!
lcwaN0003238.xml Huffington Post
lcwaN0010888.xml Cute Overload! ;)
lcwaE0008001.xml Official Campaign Web Site - <NAME>
lcwaN0012195.xml Intel Dump - Blog
lcwaE0008338.xml Official Campaign Web Site - Danny Page
lcwaE0008846.xml Official Campaign Web Site - <NAME>
lcwaN0010226.xml Homepage | Meme Generator
lcwaE0008918.xml Official Campaign Web Site - <NAME>
lcwaN0010144.xml BuzzFeed
lcwaN0010145.xml Drudge Report
lcwaN0012178.xml Life in this Girl's Army / New Lives - Blog
lcwaE0008263.xml Official Campaign Web Site - <NAME>
lcwaN0012179.xml Sgt. Missick, A Line In The Sand - Blog
lcwaN0012184.xml Doc in the Box - Blog
lcwaN0010234.xml Slate Magazine
lcwaN0010401.xml Metafilter | Community Weblog
lcwaN0010940.xml Sri Lanka Guardian
lcwa00097019.xml PMDB : O PARTIDO DO BRASIL
lcwaN0010933.xml Tamil National Alliance (TNA) - Sri Lanka
lcwaN0010932.xml Official Campaign Web Site - Maithripala Sirisen[...]<jupyter_text>Exercises* Use the script above to return one or two more elements from each XML file.* Now save the list of elements returned as a CSV file.  Find and replace in XML files<jupyter_code>import os
from bs4 import BeautifulSoup
import re

path = 'MODS'
for filename in os.listdir(path):
    if filename.endswith('.xml'): 
        fullname = os.path.join(path, filename)
        infile = open(fullname,"r")
        contents = infile.read()
        filedata = contents.replace('http', 'https')
        outfile = open(fullname, 'w')
        outfile.write(filedata)<jupyter_output><empty_output><jupyter_text>Exercise & Question* Add 'USA' in 'physicalLocation' if it is missing from the address.* Can you use something like what we have been exploring in your work?  More on BeautifulSoup, Searching, and Modifying XMLhttps://code.tutsplus.com/tutorials/scraping-webpages-in-python-with-beautiful-soup-search-and-dom-modification--cms-28276  Intro to ElementTreeSince XML is structured, you can work with it programmatically. Python has a built in library called ElementTree which has functions that allow you to work with XML. Let's look at the Python documentation for [ElementTree](https://docs.python.org/2/library/xml.etree.elementtree.html).<jupyter_code>## import package and give it an alias
import xml.etree.ElementTree as ET
## get the root element tag
etree = ET.parse('MODS/lcwaN0012195.xml')
root = etree.getroot()
root.tag
## now try the attribute
## try looping through the children
for child in root:
    print(child.tag, child.attrib)
## iterate through all the children or subelements
for elem in root.iter():
    print(elem.tag)
    ## try to get the text from each element as well
for url in root.iter('{http://www.loc.gov/mods/v3}url'):
    print(url.text)
## try another child element and print out the tag, attrib, text
nspace = 'http://www.loc.gov/mods/v3'
ns = {'mods' : nspace}

for titleInfo in root.findall('.//mods:titleInfo/mods:title', namespaces=ns): 
    element = titleInfo
    print(element.text)
for child_of_root in root:
    print(child_of_root.tag, child_of_root.attrib)
print(root[5].tag, root[5].text)<jupyter_output>{http://www.loc.gov/mods/v3}targetAudience general
{'fraction_non_alphanumeric': 0.10333954421671206, 'fraction_numerical': 0.03153217715350437, 'mean_word_length': 4.618357487922705, 'pattern_counts': {'":': 0, '<': 52, '<?xml version=': 1, '>': 52, 'https://': 5, 'lorem ipsum': 0, 'www.': 10, 'xml': 39}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '444937', 'license': 'Apache-2.0', 'n_tokens_mistral': 2366, 'n_tokens_neox': 2174, 'n_words': 728, 'path': 'LoC-MODS-Lesson.ipynb', 'repo_name': 'libcce/lc-lesson-materials', 'size': '7017'}