<jupyter_start><jupyter_text>20. Read JSON Documents<jupyter_code>with open('./enwiki-country.json') as f:
    lines = f.readlines()
    for line in lines:
        data = json.loads(line)
        if data['title'] == 'United Kingdom':
            break
print(data['title'])<jupyter_output>United Kingdom<jupyter_text>21. Lines with Category Names<jupyter_code>for line in data['text'].split('\n'):
    print(line)<jupyter_output>{{About-distinguish2|the country|[[Great Britain]], its largest island whose name is also loosely applied to the whole country}}
{{Redirect|UK|other uses of "UK"|UK (disambiguation)|other uses of "United Kingdom"|United Kingdom (disambiguation)}}
{{pp-semi-indef}}
{{pp-move-indef}}
{{short description|Country in Western Europe}}
{{Use British English|date=April 2012}}
{{Use dmy dates|date=November 2019}}
{{Infobox country
| common_name               = United Kingdom
| linking_name              = the United Kingdom<!--Note: "the" required here as this entry used to create wikilinks-->
| conventional_long_name    = United Kingdom of Great Britain and Northern Ireland
| image_flag                = Flag of the United Kingdom.svg
| alt_flag = A flag featuring both cross and saltire in red, white and blue
| other_symbol = [[File:Royal Coat of Arms of the United Kingdom.svg|x100px]][[File:Royal Coat of Arms of the United Kingdom (Scotland).svg|x100px]]
| other_symbol_type = [[Royal coat of ar[...]<jupyter_text>By inspecting the output above, we can see that there are the following lines.```[[Category:United Kingdom| ]][[Category:British Islands]][[Category:Countries in Europe]][[Category:English-speaking countries and territories]][[Category:G7 nations]][[Category:Group of Eight nations]][[Category:G20 nations]][[Category:Island countries]][[Category:Northern European countries]][[Category:Former member states of the European Union]][[Category:Member states of NATO]][[Category:Member states of the Commonwealth of Nations]][[Category:Member states of the Council of Europe]][[Category:Member states of the Union for the Mediterranean]][[Category:Member states of the United Nations]][[Category:Priority articles for attention after Brexit]][[Category:Western European countries]]```We'll extract these lines wusing regular expression. The regular expression can be written as: `\[\[Category:.+]]`. References:* https://regex101.com/* https://docs.python.org/3/library/re.html<jupyter_code>categories = re.findall(r"\[\[Category:.+]]", data['text'])
categories<jupyter_output><empty_output><jupyter_text>22. Category NamesReference: https://stackoverflow.com/questions/4666973/how-to-extract-the-substring-between-two-markers<jupyter_code>pattern = '\[\[Category:(.+?)\]\]'

for category in categories:
    m = re.search(pattern, category)
    if m:
        found = m.group(1)
        print(found)<jupyter_output>United Kingdom| 
British Islands
Countries in Europe
English-speaking countries and territories
G7 nations
Group of Eight nations
G20 nations
Island countries
Northern European countries
Former member states of the European Union
Member states of NATO
Member states of the Commonwealth of Nations
Member states of the Council of Europe
Member states of the Union for the Mediterranean
Member states of the United Nations
Priority articles for attention after Brexit
Western European countries<jupyter_text>23. Sections<jupyter_code>sections = re.findall("==.+==", data['text'])
sections
section_names = []
for section in sections:
    section_level = int(len(re.findall('=', section)) / 2 - 1)
    section_name = section.replace('=', '') 
    print(section_level, section_name)<jupyter_output>1 Etymology and terminology
1 History
2 Background
2 Treaty of Union
2 From the union with Ireland to the end of the First World War
2 Between the World Wars
2 Since the Second World War
1 Geography
2 Climate
2 Administrative divisions
1 Dependencies
1 Politics
2 Government
2 Devolved administrations
2 Law and criminal justice
2 Foreign relations
2 Military
1 Economy
2 Overview
2 Science and technology
2 Transport
2 Energy
2 Water supply and sanitation
1 Demographics
2 Ethnic groups
2 Languages
2 Religion
2 Migration
2 Education
2 Health
1 Culture
2 Literature
2 Music
2 Visual art
2 Cinema
2  Cuisine 
2 Media
2 Philosophy
2 Sport
2 Symbols
2 Stereotypes
1 Historiography
1 See also
1 Notes
1 References
1 External links<jupyter_text>24. Media ReferencesBy looking at the original file, I can see parts like: `[[File:Uk topo en.jpg|thumb|upright|The United Kingdom showing hilly regions to north and west]]`. We can extract these.<jupyter_code>media_references = re.findall("\[\[File:.+\]\]", data['text'])
media_references<jupyter_output><empty_output><jupyter_text>25. InfoboxI want to extract:```{{Infobox country| common_name               = United Kingdom| linking_name              = the United Kingdom...| date_format                = {{abbr|dd|day}}/{{abbr|mm|month}}/{{abbr|yyyy|year}}{{abbr|yyyy|year}}-{{abbr|mm|month}}-{{abbr|dd|day}}&nbsp;([[Anno Domini|AD]])| electricity                = 230 V–50 Hz| drives_on                = [[Left- and right-hand traffic|left]]Except two overseas territories, [[Gibraltar]], the [[British Indian Ocean Territory]] and [[Savoy Court]] in London.| calling_code                = [[Telephone numbers in the United Kingdom|+44]][[Telephone numbers in the United KingdomTelephone numbers in Overseas Territories|Excludes most overseas territories]]| cctld                = [[.uk]]The [[.gb]] domain is also reserved for the UK, but has been little used.| today                = }}```References:* `re.MULTILINE` and `re.DOTALL` https://docs.python.org/3/library/re.html* Non-capturing group https://medium.com/@yeukhon/non-capturing-group-in-pythons-regular-expression-75c4a828a9eb* Conditional https://stackoverflow.com/questions/39222950/regular-expression-with-if-condition<jupyter_code>pattern = r'^\{\{Infobox country.*?$(.*?)^\}\}'
s = re.findall(pattern, data['text'], re.MULTILINE + re.DOTALL)[0]
s
pattern = r'^\|(.+?)\s*=\s*(.+?)(?:(?=\n\|)|(?=\n$))'
entries = re.findall(pattern, s, re.MULTILINE + re.DOTALL)
entries = [(e[0].strip(), e[1].strip()) for e in entries]
d1 = dict(entry for entry in entries)
d1<jupyter_output><empty_output><jupyter_text>26. Remove Emphasis MarkupsI believe the attributes only have `''`, but I'll try to remove `'''` and `'''''` as well.* Replace https://stackoverflow.com/questions/11475885/python-replace-regex/11475905* Repeated characters https://stackoverflow.com/questions/8843410/repeating-a-section-of-a-regular-expression<jupyter_code>test_string = """''italic''
'''bold'''
'''''both'''''
"""

def remove_emphasis(s):
    pattern = r'\'{2,5}'
    return re.sub(pattern, '', s)


# print(my_replace(test_string))

d2 = {}
for key in d1:
    d2[key] = remove_emphasis(d1[key])
    print(d2[key])<jupyter_output>United Kingdom
the United Kingdom<!--Note: "the" required here as this entry used to create wikilinks-->
United Kingdom of Great Britain and Northern Ireland
Flag of the United Kingdom.svg
A flag featuring both cross and saltire in red, white and blue
[[File:Royal Coat of Arms of the United Kingdom.svg|x100px]][[File:Royal Coat of Arms of the United Kingdom (Scotland).svg|x100px]]
[[Royal coat of arms of the United Kingdom|Royal coats of arms]]:{{#tag:ref |The coat of arms on the left is used in England, Northern Ireland, and Wales; the version on the right is used in Scotland|group=note}}
"[[God Save the Queen]]"{{#tag:ref |There is no authorised version of the national anthem as the words are a matter of tradition; only the first verse is usually sung.<ref>{{cite web |title=National Anthem |url=https://www.royal.uk/national-anthem |website=Official web site of the British Royal Family |accessdate=4 June 2016|date=15 January 2016 }}</ref> No law was passed making "God Save the Queen" [...]<jupyter_text>27. Remove Internal Links<jupyter_code>text = """
[[copy edit]]
[[Android (operating system)|Android]]
[[Frog#Locomotion]]
"""

def remove_link(s):
    pattern = r'\[\[(?:[^|]*?\|)??([^|]*?)\]\]'
    new_s = re.sub(pattern, r'\1', s)
    return new_s.strip()

# print(remove_link(text))

d3 = {}
for key in d2:
    d3[key] = remove_link(d2[key])
    print(d3[key])<jupyter_output>United Kingdom
the United Kingdom<!--Note: "the" required here as this entry used to create wikilinks-->
United Kingdom of Great Britain and Northern Ireland
Flag of the United Kingdom.svg
A flag featuring both cross and saltire in red, white and blue
x100pxx100px
Royal coats of arms:{{#tag:ref |The coat of arms on the left is used in England, Northern Ireland, and Wales; the version on the right is used in Scotland|group=note}}
"God Save the Queen"{{#tag:ref |There is no authorised version of the national anthem as the words are a matter of tradition; only the first verse is usually sung.<ref>{{cite web |title=National Anthem |url=https://www.royal.uk/national-anthem |website=Official web site of the British Royal Family |accessdate=4 June 2016|date=15 January 2016 }}</ref> No law was passed making "God Save the Queen" the official anthem. In the English tradition, such laws are not necessary; proclamation and usage are sufficient to make it the national anthem. "God Save the Queen" a[...]
{'fraction_non_alphanumeric': 0.12332757876564523, 'fraction_numerical': 0.021040138109624514, 'mean_word_length': 4.983860555196901, 'pattern_counts': {'":': 0, '<': 38, '<?xml version=': 0, '>': 38, 'https://': 10, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '524362', 'license': 'MIT', 'n_tokens_mistral': 2938, 'n_tokens_neox': 2693, 'n_words': 971, 'path': 'NLP/RegularExpression/RegularExpression.ipynb', 'repo_name': 'SMU-HCI-Lab/tutorials', 'size': '9268'}