<jupyter_start><jupyter_text>Name: <NAME>Date: 5/4/2018Version: 3.0Environment: Python 3.6.0 and Jupyter notebookLibraries used:* json (for json file, included in Anaconda Python 3.6) * re 2.2.1 (for regular expression, included in Anaconda Python 3.6)  1.IntroductionThis assignment make extraction of the data from an xml file and save the result into a csv file. The requirement can be explain as follow:1. The data structure is json. All the data should be save in a json file.2. Use only the regular expression to extract data.The detail of each section is shown below.<jupyter_code>import re
import json<jupyter_output><empty_output><jupyter_text>To open the xml file and use readlines method to present the data.<jupyter_code>xml = open('australian-sport-thesaurus-student.xml', encoding= 'UTF-8')
xml_lines = xml.readlines()
#xml_lines<jupyter_output><empty_output><jupyter_text>Use the regular expression to find out the data in exact tag of the xml file.<jupyter_code>reg_description = re.compile('(?<=<Description>)(.*?)(?=</Description>)')
reg_start = re.compile('<Description>')
reg_description_end = re.compile('</Term>')
reg_related_terms = re.compile('<RelatedTerms>')
reg_related_title = re.compile('(?<=<Title>)(.*?)(?=</Title>)')
reg_title_end = re.compile('</RelatedTerms>')
reg_relationship = re.compile('(?<=<Relationship>)(.*?)(?=</Relationship>)')<jupyter_output><empty_output><jupyter_text>After observation, the tag 'description','title' and 'related terms' are in one-level list. To find out the data inside these three tags and append in to a new list.<jupyter_code>dic = {}
dic["thesaurus"] = []
for i in range(0,len(xml_lines)):
    dic1 = {}
    c = reg_description.search(xml_lines[i])
    #to find out the lines with 'description' tag and locate them
    if reg_start.search(xml_lines[i]) != None:
        #use the 'description' location to search for it previous and next tag and values
        d = reg_related_title.search(xml_lines[i-1])
        term = reg_description_end.search(xml_lines[i+1])
        #if the data only have 'description' and 'title', append to a dictionary
        if c != None and d != None:
            if term != None:
                dic1 = {"Description": c.group(), "Title": d.group()}
            #if not, keep a null lits as the value of the 'related terms'
            else:
                dic1 = {"Description": c.group(),"RelatedTerms": [], "Title": d.group()}
            dic["thesaurus"].append(dic1)
#print(dic)<jupyter_output><empty_output><jupyter_text>Then to structure the sublist of the 'related terms'. The data of the in the 'related term' tag is with tag 'title' and 'relationship'. But it can contain more than one pair of them in one 'related term'.<jupyter_code>second_list = []
dic3 = {}
for k in range(0,len(xml_lines)):
    dic_related = {}
    dic_related["RelatedTerms"] = []
    #to locate the 'related terms' tag and find out it sub-tags
    if reg_related_terms.search(xml_lines[k]) != None:
        k += 2
        #find out the 'title' and 'relationship' content of inside 'related terms' tag
        a = reg_related_title.search(xml_lines[k])
        relationship = reg_relationship.search(xml_lines[k + 1])
        if a != None and relationship != None:
            #structure the sub-dictionary of 'related terms' directly,
            #if there is only one pair of 'title' and 'relationship'
            dic3 = {"Relationship": relationship.group(),"Title": a.group()}
            dic_related["RelatedTerms"].append(dic3)
            k += 3
            #if the 'title' and 'relationship' tag pair is more than one,
            #find out all of them and structure them in sub-dictionary
            while reg_title_end.search(xml_lines[k]) == None:
                k += 1
                b = reg_related_title.search(xml_lines[k])
                relationship = reg_relationship.search(xml_lines[k + 1])
                if b != None and relationship != None:
                    dic3 = {"Relationship": relationship.group(), "Title": b.group()}
                    dic_related["RelatedTerms"].append(dic3)
        #print(dic_related)
        second_list.append(dic_related)<jupyter_output><empty_output><jupyter_text>Afer extracting the data in one-level dictionary and sub-dictionary, to assign the 'related terms' sub-dictionary into the original null list. The aggregation of the two dictionaries makes the whole structure finished.<jupyter_code>reg = re.compile('RelatedTerms')
s= 0
for dics in dic["thesaurus"]:
    if reg.search(str(dics)) !=None:        
        dics["RelatedTerms"] = second_list[s]["RelatedTerms"]
        s += 1
dics["RelatedTerms"] = second_list[s]["RelatedTerms"]
dic<jupyter_output><empty_output><jupyter_text>Finally, to dump the data in to using json and save the data in to json file.<jupyter_code>data = json.dumps(dic,sort_keys = True)
data
with open('sport.dat.json','w') as file:
    json.dump(dic,file,sort_keys = True)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1208592652077896, 'fraction_numerical': 0.008432041758682996, 'mean_word_length': 3.722274881516588, 'pattern_counts': {'":': 9, '<': 43, '<?xml version=': 0, '>': 40, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 20}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '627869', 'license': 'MIT', 'n_tokens_mistral': 1561, 'n_tokens_neox': 1515, 'n_words': 472, 'path': 'xml_json.ipynb', 'repo_name': 'MaMickey/Data-Wrangling', 'size': '4983'}