<jupyter_start><jupyter_text>Parsing XML with BeautifulSoup[BeautifulSoup](https://www.crummy.com/software/BeautifulSoup/bs4/doc/) is a general way to read hierarchical, tree-like structures, e.g. XML and HTML.Let's read some XML! Here's the data:<jupyter_code>xml_text = """<?xml version="1.0" ?>
<collection>
<mineral>
<name>Quartz</name>
<hardness>7</hardness>
<colour>colourless</colour>
<info>Common in continental crust</info>
</mineral>
<mineral>
<name>Olivine</name>
<hardness>6.5</hardness>
<colour>green</colour>
<info>Very common in the mantle</info>
</mineral>
</collection>
"""
from bs4 import BeautifulSoup

soup = BeautifulSoup(xml_text, 'html.parser')

print(soup.prettify())<jupyter_output><?xml version="1.0" ?>
<collection>
 <mineral>
  <name>
   Quartz
  </name>
  <hardness>
   7
  </hardness>
  <colour>
   colourless
  </colour>
  <info>
   Common in continental crust
  </info>
 </mineral>
 <mineral>
  <name>
   Olivine
  </name>
  <hardness>
   6.5
  </hardness>
  <colour>
   green
  </colour>
  <info>
   Very common in the mantle
  </info>
 </mineral>
</collection><jupyter_text>Find all of a particular field<jupyter_code>minerals = soup.find_all('name')
minerals
for mineral in minerals:
    print(mineral.get_text())<jupyter_output>Quartz
Olivine<jupyter_text>Put data in `pandas`<jupyter_code>minerals = soup.find_all('mineral')
import pandas as pd

data = []
for mineral in minerals:
    d = [x for x in filter(None, mineral.get_text().split('\n'))]
    data.append(d)
    
# I'm sure there's a better way to get these names...
df = pd.DataFrame(data, columns=['Name', 'Hardness', 'Color', 'Info'])
df<jupyter_output><empty_output><jupyter_text>SelectSometimes you'd like to be more specific about what you're selecting. Then you can use `select()`, which on a simple level does what `find_all()` can do...<jupyter_code>soup.select('info')<jupyter_output><empty_output><jupyter_text>...but you can also pass full branches: `soup.select("div[id=foo] > div > div[class=fee] > span > span > a"`<jupyter_code>soup.select('collection > mineral > info')
[i.text for i in soup.select('collection > mineral > info')]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15377532228360957, 'fraction_numerical': 0.0055248618784530384, 'mean_word_length': 5.138418079096045, 'pattern_counts': {'":': 0, '<': 65, '<?xml version=': 2, '>': 74, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 4}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '417606', 'license': 'Apache-2.0', 'n_tokens_mistral': 826, 'n_tokens_neox': 793, 'n_words': 189, 'path': 'Parsing_XML_with_BeautifulSoup.ipynb', 'repo_name': 'EvanBianco/geocomputing_demos', 'size': '2172'}