<jupyter_start><jupyter_text>Mining the social web  Workout 2. Understanding XML - CityU COM5507 201819A - Unit 2: Web data collection- 24 Oct 2018, Week 8: Mining the social web - data formats - Course Instructor: [Dr. <NAME>](www.drxinzhizhang.com)  (JOUR, Hong Kong Baptist University)   - <EMAIL>- The codes in this notebook are modified from various sources. All codes are for educational purposes only and released under the CC1.0.   XML basic: structures & finding the elements - tutorial: https://stackabuse.com/reading-and-writing-xml-files-in-python/- official tutorial: https://docs.python.org/3.7/library/xml.etree.elementtree.html  - ```Element.find()``` finds the first child with a particular tag, and Element.text accesses the element’s text content. - ```Element.get()``` accesses the element’s attributes. - ```Element.findall()``` finds only elements with a tag which are direct children of the current element. - ```Element.iter()``` creates a tree iterator with the current element as the root. The iterator iterates over this element and all elements below it, in document (depth first) order.<jupyter_code>import xml.etree.ElementTree as ET # always import this when parsing XML 
data = '''
<person>
  <name>Chuck</name>
  <phone type="intl">
     +1 734 303 4456
   </phone>
   <email hide="yes"/>
</person>
'''
tree = ET.fromstring(data)
print('Name:', tree.find('name').text)
# what happened if we print: print('Name:', tree.find('name'))? 
print('Phone:', tree.find('phone').text) 
# compare: print('Phone:', tree.find('phone').text.strip()) 
print('Attrib', tree.find('email').get('hide'))
input = '''
<stuff>
    <users>
        <user x="2">
            <id>001</id>
            <name>Chuck</name>
        </user>
        <user x="7">
            <id>009</id>
            <name>Brent</name>
        </user>
    </users>
</stuff>'''
stuff = ET.fromstring(input)
lst = stuff.findall('users/user')
print(type(lst))
print('User count:', len(lst))
for item in lst:
    print('Name', item.find('name').text)
    print('Id', item.find('id').text)
    print('Attribute', item.get("x"))<jupyter_output>Name Chuck
Id 001
Attribute 2
Name Brent
Id 009
Attribute 7<jupyter_text>Another example: an official tutorial<jupyter_code>country_data_as_string = '''
<data>
    <country name="Liechtenstein">
        <rank>1</rank>
        <year>2008</year>
        <gdppc>141100</gdppc>
        <neighbor name="Austria" direction="E"/>
        <neighbor name="Switzerland" direction="W"/>
    </country>
    <country name="Singapore">
        <rank>4</rank>
        <year>2011</year>
        <gdppc>59900</gdppc>
        <neighbor name="Malaysia" direction="N"/>
    </country>
    <country name="Panama">
        <rank>68</rank>
        <year>2011</year>
        <gdppc>13600</gdppc>
        <neighbor name="Costa Rica" direction="W"/>
        <neighbor name="Colombia" direction="E"/>
    </country>
</data>'''
import xml.etree.ElementTree as ET
tree = ET.parse('data.xml')
root = tree.getroot()

#root = ET.fromstring(country_data_as_string)
for child in root:
    print(child.tag, child.attrib)
# XML can be as paths 
root[0][1].text
for neighbor in root.iter('neighbor'):
    print(neighbor.attrib)
names = []
directions = []
for neighbor in root.iter('neighbor'):
    name = neighbor.get('name')
    direction = neighbor.get('direction') 
    #print(name, vote)
    names.append(name)
    directions.append(direction)

print(names)
print(directions)<jupyter_output>['Austria', 'Switzerland', 'Malaysia', 'Costa Rica', 'Colombia']
['E', 'W', 'N', 'W', 'E']<jupyter_text>A real example: the Hong Kong LegCo voting- Open data: https://www.legco.gov.hk/general/chinese/open-legco/cm-201819.htmln1a- Data source: https://www.legco.gov.hk/yr18-19/chinese/counmtg/voting/cm_vote_20181010.xml- The Hong Kong LegCo voting record documented on the LegCo website is a very good example of open data, as well as a good practicum field for data analytical skills.<jupyter_code>import xml.etree.ElementTree as ET  
tree = ET.parse('cm_vote_20181010.xml')  
root = tree.getroot()
root.tag
root.attrib
for child in root:
    print(child.tag, child.attrib) 
for member in root.iter('member'):
    print(member.attrib) 
names = []
votes = []
for member in root.iter('member'):
    vote = member.find('vote').text
    name = member.get('name-ch') 
    #print(name, vote)
    votes.append(vote)
    names.append(name)
import pandas as pd
df_vote = {
    'names': names,
    'votes': votes
}
pd_vote = pd.DataFrame.from_dict(df_vote) 
print(pd_vote.info())
pd_vote
pd_vote<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15382941688424717, 'fraction_numerical': 0.024151436031331592, 'mean_word_length': 3.8186582809224316, 'pattern_counts': {'":': 0, '<': 67, '<?xml version=': 0, '>': 67, 'https://': 4, 'lorem ipsum': 0, 'www.': 3, 'xml': 8}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '95588', 'license': 'MIT', 'n_tokens_mistral': 1610, 'n_tokens_neox': 1516, 'n_words': 402, 'path': 'py_notebooks/socialweb_01_XML/.ipynb_checkpoints/socialweb_02_xml-checkpoint.ipynb', 'repo_name': 'xzzhang2/201819A_cityu_com5507', 'size': '4613'}