<jupyter_start><jupyter_text>Dive into XML This notebook is the result of reading through [chapter 12 "XML"](https://diveintopython3.net/xml.html) of <NAME>'s "Dive into Python".This notebook as the original Book is licensed under theCreative Commons Attribution Share-Alike license ([CC-BY-SA-3.0](https://creativecommons.org/licenses/by-sa/3.0/)). ElementTree`ElementTree` from the standard library and `lxml` are the most prevalent tools in the Python world for processing XML. The ElementTree library is part of the Python standard library<jupyter_code>import xml.etree.ElementTree as etree<jupyter_output><empty_output><jupyter_text>The primary entry point for the ElementTree library is the `parse()` function, which can take a filename or a file-like object. This function parses the entire document at once. If memory is tight, there are ways to parse an XML document incrementally instead.<jupyter_code>tree = etree.parse('feed.xml')<jupyter_output><empty_output><jupyter_text>The `parse()` function returns an object which represents the entire document. This is not the root element. To get a reference to the root element, call the `getroot()` method.<jupyter_code>root = tree.getroot()<jupyter_output><empty_output><jupyter_text>In the example file `feed.xml` the root element is the feed element in the http://www.w3.org/2005/Atom namespace. The string representation of this object reinforces an important point: an XML element is a combination of its namespace and its tag name (also called the local name). Every element in this document is in the Atom namespace, so the root element is represented as {http://www.w3.org/2005/Atom}feed.<jupyter_code>root<jupyter_output><empty_output><jupyter_text>*ElementTree represents XML elements as `{namespace}localname`. You’ll see and use this format in multiple places in the ElementTree `API`.*<jupyter_code>root.tag<jupyter_output><empty_output><jupyter_text>The “length” of the root element is the number of child elements.<jupyter_code>len(root)<jupyter_output><empty_output><jupyter_text>An element can be used as an iterator to loop through all of its child elements. The list of child elements only includes `direct` children.<jupyter_code>for child in root:
    print(child)<jupyter_output><Element '{http://www.w3.org/2005/Atom}title' at 0x10b8e7e50>
<Element '{http://www.w3.org/2005/Atom}subtitle' at 0x10b8e7ef0>
<Element '{http://www.w3.org/2005/Atom}id' at 0x10b8ed040>
<Element '{http://www.w3.org/2005/Atom}updated' at 0x10b8ed0e0>
<Element '{http://www.w3.org/2005/Atom}link' at 0x10b8ed220>
<Element '{http://www.w3.org/2005/Atom}entry' at 0x10b8ed2c0>
<Element '{http://www.w3.org/2005/Atom}entry' at 0x10b8eda90>
<Element '{http://www.w3.org/2005/Atom}entry' at 0x10b8edf90><jupyter_text>Attributes Are DictonariesOnce you have a reference to a specific element, you can easily get its attributes as a Python dictionary.<jupyter_code>root.attrib
root[4]
root[4].attrib
root[3]
# The updated element has no attributes, 
# so its .attrib is just an empty dictionary.
root[3].attrib<jupyter_output><empty_output><jupyter_text>Searching For Nodes Within An XML Document findall()Each element — including the root element, but also child elements — has a `findall()` method. It finds all matching elements among the element’s children.<jupyter_code>tree
# We will need to use the namespace a lot, so we make this shortcut
namespace = '{http://www.w3.org/2005/Atom}'
root.findall(f'{namespace}entry')
root.tag
# This query returns an empty list because the root
# element 'feed' does not have any child element 'feed'
root.findall(f'{namespace}feed')
# This query only finds direct children. The author nodes are nested,
# therefore this query returns an empty list
root.findall(f'{namespace}author')<jupyter_output><empty_output><jupyter_text>For convenience, the `tree` object (returned from the `etree.parse()` function) has several methods that mirror the methods on the root element. The results are the same as if you had called the `tree.getroot().findall()` method.<jupyter_code>tree.findall(f'{namespace}entry')
tree.findall(f'{namespace}author')<jupyter_output><empty_output><jupyter_text>find()The `find()` method takes an ElementTree query returns the first matching element. This is useful for situations where you are only expecting one match, or if there are multiple matches, you only care about the first one.<jupyter_code>entries = tree.findall(f'{namespace}entry')
len(entries)
# Get the first title, secretly we know there is only one title
title_element = entries[0].find(f'{namespace}title')
title_element.text<jupyter_output><empty_output><jupyter_text>There are no elements in this entry named `foo`, so this returns `None`.<jupyter_code>foo_element = entries[0].find(f'{namespace}foo')
foo_element
type(foo_element)<jupyter_output><empty_output><jupyter_text>**Beware:** In a boolean context, ElementTree element objects will evaluate to `False` if they contain no children (i.e. if `len(element)` is 0). This means that if `element.find('...')` is not testing whether the find() method found a matching element; it’s testing whether that matching element has any child elements! To test whether the `find()` method returned an element, use if `element.find('...') is not None`.  Search for descendant elementsA query like `//{http://www.w3.org/2005/Atom}link` with the two slashes at the beginning finds any elements, regardless of nesting level.<jupyter_code>all_links = tree.findall(f'.//{namespace}link')
all_links
all_links[0].attrib
all_links[1].attrib
all_links[2].attrib
all_links[3].attrib<jupyter_output><empty_output><jupyter_text>ElementTree’s `findall()` method is a very powerful feature, but the query language can be a bit surprising. ElementTree’s query language is similar enough to XPath to do basic searching, but dissimilar enough that it may annoy you if you already know XPath.  Parsing with lxml`lxml` utilizes the popular `libxml2` parser. It provides a 100% compatible ElementTree API, then extends it with full XPath 1.0 support and a few other niceties.<jupyter_code># We will need to use the namespace a lot, so we make this shortcut
namespace = '{http://www.w3.org/2005/Atom}'
from lxml import etree
tree = etree.parse('feed.xml')
root = tree.getroot()
root.findall(f'{namespace}entry')<jupyter_output><empty_output><jupyter_text>For large XML documents `lxml` is significantly faster than the `built-in` ElementTree library. If you’re only using the ElementTree API and want to use the fastest available implementation, you can try to import `lxml` and fall back to the built-in ElementTree.<jupyter_code>try:
    from lxml import etree
except ImportError:
    import xml.etree.ElementTree as etree<jupyter_output><empty_output><jupyter_text>The following query finds all elements in the Atom namespace, anywhere in the document, that have an `href` attribute. The `//` at the beginning of the query means “elements anywhere (not just as children of the root element).” `{http://www.w3.org/2005/Atom}` means “only elements in the Atom namespace.” `*` means “elements with any local name.” And `[@href]` means “has an href attribute.”<jupyter_code>tree.findall(f'//{namespace}*[@href]')
tree.findall(f"//{namespace}*[@href='http://diveintomark.org/']")
# Using NS as name of the namespace variable is a cool idea
NS = '{http://www.w3.org/2005/Atom}'<jupyter_output><empty_output><jupyter_text>The following query searches for Atom `author` elements that have an Atom `uri` element as a child. This only returns two `author` elements, the ones in the first and second `entry`. The `author` in the last `entry` contains only a `name`, not a `uri`.<jupyter_code>tree.findall(f'//{NS}author[{NS}uri]')<jupyter_output><empty_output><jupyter_text>XPath support in lxmlTechnically an XPath expressions returns a list of nodes. (Thats what the DOM of a parsed XML document is made up of). Depending on their type, nodes can be elements, attributes, or even text content. To perform XPath queries on namespaced elements, you need to define a namespace prefix mapping. This is just a Python dictionary.<jupyter_code>NSMAP = {'atom': 'http://www.w3.org/2005/Atom'}<jupyter_output><empty_output><jupyter_text>The XPath expression searches for `category` elements (in the Atom namespace) that contain a `term` attribute with the value `accessibility`. The `/..` bit means to return the parent element of the category element you just found.So this single XPath query will find all entries with a child element of ``.In this case the `xpath()` function returns a list of ElementTree objects.<jupyter_code>entries = tree.xpath("//atom:category[@term='accessibility']/..", 
                    namespaces=NSMAP)
entries<jupyter_output><empty_output><jupyter_text>The following query returns a list that contains a string. It selects text content (`text()`) of the title element (`atom:title`) that is a child of the current element (`./`).<jupyter_code># Pick the first (and only) element from the entries list
entry = entries[0]
# It is an ElementTree object and therefore supports
entry.xpath('./atom:title/text()', namespaces=NSMAP)<jupyter_output><empty_output><jupyter_text>Generating XMLYou can create XML documents from scratch.<jupyter_code>import xml.etree.ElementTree as etree
atom_NS = '{http://www.w3.org/2005/Atom}'
w3_NS = '{http://www.w3.org/XML/1998/namespace}'<jupyter_output><empty_output><jupyter_text>To create a new element, instantiate the `Element` class. You pass the element name (namespace + local name) as the first argument. This statement creates a `feed` element in the Atom namespace. This will be our new document’s root element.To add attributes to the newly created element, pass a dictionary of attribute names and values in the `attrib` argument. Note that the attribute name should be in the standard ElementTree format, `{namespace}localname`.<jupyter_code>new_feed = etree.Element(f'{atom_NS}feed',
                        attrib={f'{w3_NS}lang': 'en'})<jupyter_output><empty_output><jupyter_text>At any time, you can serialize any element (and its children) with the ElementTree `tostring()` function.<jupyter_code>print(etree.tostring(new_feed))<jupyter_output>b'<ns0:feed xmlns:ns0="http://www.w3.org/2005/Atom" xml:lang="en" />'<jupyter_text>Default namespacesA default namespace is useful for documents — like Atom feeds — where every element is in the same namespace. The namespace is declared once and each element just needs to be declared with its local name (``, ``, ``). There is no need to use any prefixes unless you want to declare elements from another namespace.    The first snippet has an default, implicit namespace.```xml```Whereas the second, this is how `ElementTree` serializes namespaced XML elements, has an explicit namespace. This is technically accurate, but a bit cumbersome to work with.     Both `DOM`s of the serialisations in the example are identical.```xml```    `lxml` does offer fine-grained control over how namespaced elements are serialized. The built-in `ElementTree` does not.<jupyter_code># We import lxml's etree like this, to make it recognizeable 
# in the example
import lxml.etree<jupyter_output><empty_output><jupyter_text>Define a namespace mapping as a dictionary. Dictionary values are namespaces; dictionary keys are the desired prefix. Using `None` as a prefix effectively declares a default namespace.<jupyter_code>NSMAP = {None: 'http://www.w3.org/2005/Atom'}<jupyter_output><empty_output><jupyter_text>Now you can pass the `lxml`-specific `nsmap` argument when you create an element, and `lxml` will respect the namespace prefixes you’ve defined.<jupyter_code>new_feed = lxml.etree.Element('feed', nsmap=NSMAP)<jupyter_output><empty_output><jupyter_text>This serialization defines the Atom namespace as the default namespace and declares the feed element without a namespace prefix.<jupyter_code>print(lxml.etree.tounicode(new_feed))
# Aha, .tounicode() would be one way to get a string instead of 
# a byte object
print(lxml.etree.tostring(new_feed))<jupyter_output>b'<feed xmlns="http://www.w3.org/2005/Atom"/>'<jupyter_text>You can always add attributes to any element with the `set()` method. It takes two arguments: the attribute name in standard ElementTree format, then the attribute value. This method is not `lxml`-specific.<jupyter_code>new_feed.set('{http://www.w3.org/XML/1998/namespace}lang', 'en')
print(lxml.etree.tounicode(new_feed))<jupyter_output><feed xmlns="http://www.w3.org/2005/Atom" xml:lang="en"/><jupyter_text>Create child elementsInstantiate the `SubElement` class to create a child element of an existing element. The only required arguments are the parent element (`new_feed` in this case) and the new element’s name. Since this child element will inherit the namespace mapping of its parent, there is no need to redeclare the namespace or prefix here.You can also pass in an attribute dictionary. Keys are attribute names; values are attribute values.<jupyter_code>title = lxml.etree.SubElement(new_feed, 'title', 
                              attrib={'type':'html'})
print(lxml.etree.tounicode(new_feed))<jupyter_output><feed xmlns="http://www.w3.org/2005/Atom" xml:lang="en"><title type="html"/></feed><jupyter_text>Set the `.text` property to add the text content to an element.<jupyter_code>title.text = 'dive into &hellip;'
print(lxml.etree.tounicode(new_feed))
print(lxml.etree.tounicode(new_feed, pretty_print=True))<jupyter_output><feed xmlns="http://www.w3.org/2005/Atom" xml:lang="en">
  <title type="html">dive into &amp;hellip;</title>
</feed><jupyter_text>Parsing broken xml`lxml` is capable of parsing not wellformed xml documents.The parser chokes on this document, because because the `&hellip;` entity is not defined in XML.<jupyter_code>import lxml.etree
tree = lxml.etree.parse('broken-feed.xml')<jupyter_output><empty_output><jupyter_text>Instantiate the `lxml.etree.XMLParser` class to create a custom parser. It can take a number of different named arguments. Here we are using the `recover` argument, so that the XML parser will try its best to “recover” from wellformedness errors.<jupyter_code>parser = lxml.etree.XMLParser(recover=True)<jupyter_output><empty_output><jupyter_text>This works! The second argument of `parse()` is the custom parser.<jupyter_code>tree = lxml.etree.parse('broken-feed.xml', parser)<jupyter_output><empty_output><jupyter_text>The parser keeps a log of the wellformedness errors that it has encountered.<jupyter_code>parser.error_log
tree.findall('{http://www.w3.org/2005/Atom}title')<jupyter_output><empty_output><jupyter_text>The parser just dropped the undefined `&hellip;` entity. The text content of the title element becomes 'dive into '.<jupyter_code>title = tree.findall('{http://www.w3.org/2005/Atom}title')[0]
title.text<jupyter_output><empty_output><jupyter_text>As you can see from the serialization, the &hellip; entity didn’t get moved; it was simply dropped.<jupyter_code>print(lxml.etree.tounicode(tree.getroot()))<jupyter_output><feed xmlns="http://www.w3.org/2005/Atom" xml:lang="en">
  <title>dive into </title>
</feed>
{'fraction_non_alphanumeric': 0.10481792346837618, 'fraction_numerical': 0.01460577622100324, 'mean_word_length': 5.878181818181818, 'pattern_counts': {'":': 0, '<': 161, '<?xml version=': 0, '>': 161, 'https://': 2, 'lorem ipsum': 0, 'www.': 28, 'xml': 54}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '36', 'dir': 'jupyter-structured-clean-dedup', 'id': '327634', 'license': 'Apache-2.0', 'n_tokens_mistral': 4762, 'n_tokens_neox': 4573, 'n_words': 1621, 'path': '_notebooks/2021-10-11-dive_into_xml.ipynb', 'repo_name': 'staeff/fastblog', 'size': '15137'}