<jupyter_start><jupyter_text>Similar to chapter 10, this chapter is also text/idea heavy. It focuses on the maintenance of linguistic data, or corpora. While this notebook includes code cells for examples of data format conversions, data extraction, and other tasks involved in managing linguistic data, it is recommended to go through the chapter in the book to gain a better understanding of the major ideas involved.https://www.nltk.org/book/ch11.html Managing Linguistic Data Corpus Structure: A Case Study NLTK includes a sample from the TIMIT corpus. You can access its documentation in the usual way, using help(nltk.corpus.timit). Print nltk.corpus.timit.fileids() to see a list of the 160 recorded utterances in the corpus sample.<jupyter_code>import nltk
nltk.corpus.timit.fileids()[:10]<jupyter_output><empty_output><jupyter_text>Each item has a phonetic transcription which can be accessed using the phones() method.<jupyter_code>phonetic = nltk.corpus.timit.phones('dr1-fvmh0/sa1')
print(phonetic)
nltk.corpus.timit.word_times('dr1-fvmh0/sa1')<jupyter_output><empty_output><jupyter_text>In addition to this text data, TIMIT includes a lexicon that provides the canonical pronunciation of every word, which can be compared with a particular utterance:<jupyter_code>timitdict = nltk.corpus.timit.transcription_dict()
timitdict['greasy'] + timitdict['wash'] + timitdict['water']
phonetic[17:30]
# Accessing demographic data of a particular ite
nltk.corpus.timit.spkrinfo('dr1-fvmh0')<jupyter_output><empty_output><jupyter_text>Acquiring Data     Obtaining Data from the Word Processor Files  Consider the following fragment of a lexical entry: "sleep [sli:p] v.i. condition of body and mind...". We can enter this in MSWord, then "Save as Web Page", then inspect the resulting HTML file:```sleep     [sli:p]     v.i.     a condition of body and mind ...```<jupyter_code>import re
legal_pos = set(['n', 'v.t.', 'v.i.', 'adj', 'det'])
pattern = re.compile(r"'font-size:11.0pt'>([a-z.]+)<")
document = open("data/dict.htm", encoding="windows-1252").read()
document<jupyter_output><empty_output><jupyter_text>We can inspect the used pos tags as follows:<jupyter_code>used_pos = set(re.findall(pattern, document))
used_pos<jupyter_output><empty_output><jupyter_text>Once we know the data is correctly formatted, we can write other programs to convert the data into a different format. Obtaining Data from Spreadsheets and DatabasesOur program might perform a linguistically motivated query which cannot be expressed in SQL, e.g. select all words that appear in example sentences for which no dictionary entry is provided. For this task, we would need to extract enough information from a record for it to be uniquely identified, along with the headwords and example sentences. Let's suppose this information was now available in a CSV file dict.csv:```"sleep","sli:p","v.i","a condition of body and mind ...""walk","wo:k","v.intr","progress by lifting and setting down each foot ...""wake","weik","intrans","cease to sleep"```We can express this query in code below:<jupyter_code>import csv
lexicon = csv.reader(open('data/dict.csv'))
pairs = [(lexeme, defn) for (lexeme, _, _, defn) in lexicon]
lexemes, defns = zip(*pairs)
defn_words = set(w for defn in defns for w in defn.split())
print(defn_words)
print(sorted(defn_words.difference(lexemes)))<jupyter_output>['...', 'a', 'and', 'body', 'by', 'cease', 'condition', 'down', 'each', 'foot', 'lifting', 'mind', 'of', 'progress', 'setting', 'to']<jupyter_text>We can use this information to enrich the lexicon.  Converting Data FormatsConsider a case where we had to have a build an inverted index using some input data. We can continue the use of the data from the above example, and consider an example where we have to construct an index that maps the words of a dictionary definition to the corresponding index.<jupyter_code>idx = nltk.Index((defn_word, lexeme) 
                 for (lexeme, defn) in pairs 
                 for defn_word in nltk.word_tokenize(defn) 
                 if len(defn_word) > 3)
idx
with open("output/dict.idx", "w") as idx_file:
     for word in sorted(idx):
            idx_words = ', '.join(idx[word])
            idx_line = "{}: {}".format(word, idx_words)
            print(idx_line, file=idx_file)<jupyter_output><empty_output><jupyter_text>Lookup by Pronunciation SimilarityFirst, we identify confusible letter sequences and map complex versions to simpler versions.<jupyter_code>mappings = [('ph', 'f'), ('ght', 't'), ('^kn', 'n'), ('qu', 'kw'),
             ('[aeiou]+', 'a'), (r'(.)\1', r'\1')]
def signature(word):
    for patt, repl in mappings: 
        word = re.sub(patt, repl, word)
    pieces = re.findall("[^aeiou]+", word)
    return "".join(char for piece in pieces for char in sorted(piece))[:8]
signature("illefent")
signature("elephant")
signature("ebsekwieous")
signature("nuculerr")<jupyter_output><empty_output><jupyter_text>Next, we create a mapping from signatures to words, for all words in our lexicon.<jupyter_code>signatures = nltk.Index((signature(w), w) for w in nltk.corpus.words.words())
print(signatures[signature("nuclerr")])<jupyter_output>['anicular', 'inocular', 'nucellar', 'nuclear', 'unicolor', 'uniocular', 'unocular']<jupyter_text>Finally, we rank the results in terms of similarity to the original word.<jupyter_code>def rank(word, wordlist):
    ranked = sorted((nltk.edit_distance(word, w), w) for w in wordlist)
    return [word for (_, word) in ranked]
def fuzzy_spell(word):
    sig = signature(word)
    if sig in signatures:
        return rank(word, signatures[sig])
    return []
fuzzy_spell("illefent")
fuzzy_spell("nuclar")
fuzzy_spell("eple")[0]<jupyter_output><empty_output><jupyter_text>In the above example, a simple program can facilitate access to lexical data in a context where the writing system of a language may not be standardized, or where users of the language may not have a good command of spellings. Working with XML The ElementTree InterfaceLet's illustrate the use of ElementTree using a collection of Shakespeare plays that have been formatted using XML.<jupyter_code># Loading the xml file
merchant_file = nltk.data.find("corpora/shakespeare/merchant.xml")
raw = open(merchant_file).read()
# Inspecting contents
print(raw[:163])
print(raw[1789:2006])<jupyter_output><TITLE>ACT I</TITLE>

<SCENE><TITLE>SCENE I.  Venice. A street.</TITLE>
<STAGEDIR>Enter ANTONIO, SALARINO, and SALANIO</STAGEDIR>

<SPEECH>
<SPEAKER>ANTONIO</SPEAKER>
<LINE>In sooth, I know not why I am so sad:</LINE><jupyter_text>As you can see above, we have accessed the XML data as string. The next step is to process the file contents as structured XML data using `ElementTree`.<jupyter_code>from xml.etree.ElementTree import ElementTree
merchant = ElementTree().parse(merchant_file)
merchant
merchant[0]
merchant[0].text
merchant.getchildren()<jupyter_output>/Users/sajal/anaconda/envs/general/lib/python3.7/site-packages/ipykernel_launcher.py:1: DeprecationWarning: This method will be removed in future versions.  Use 'list(elem)' or iteration over elem instead.
  """Entry point for launching an IPython kernel.<jupyter_text>The play consists of a title, the personae, a scene description, a subtitle, and five acts. Each act has a title and some scenes, and each scene consists of speeches which are made up of lines, a structure with four levels of nesting. Let's dig down into Act IV:<jupyter_code>merchant[-2][0].text
merchant[-2][1]
merchant[-2][1][0].text
merchant[-2][1][54]
merchant[-2][1][54][0]
merchant[-2][1][54][0].text
merchant[-2][1][54][1]
merchant[-2][1][54][1].text<jupyter_output><empty_output><jupyter_text>We can iterate over the types we are interested in (ex: ACTS) using `merchant.findall('ACT')`. Below is an example of doing tag-specific searches at every level of nesting:<jupyter_code>for i, act in enumerate(merchant.findall('ACT')):
    for j, scene in enumerate(act.findall('SCENE')):
        for k, speech in enumerate(scene.findall('SPEECH')):
            for line in speech.findall('LINE'):
                if 'music' in str(line.text):
                    print("Act %d Scene %d Speech %d: %s" % (i+1, j+1, k+1, line.text))<jupyter_output>Act 3 Scene 2 Speech 9: Let music sound while he doth make his choice;
Act 3 Scene 2 Speech 9: Fading in music: that the comparison
Act 3 Scene 2 Speech 9: And what is music then? Then music is
Act 5 Scene 1 Speech 23: And bring your music forth into the air.
Act 5 Scene 1 Speech 23: Here will we sit and let the sounds of music
Act 5 Scene 1 Speech 23: And draw her home with music.
Act 5 Scene 1 Speech 24: I am never merry when I hear sweet music.
Act 5 Scene 1 Speech 25: Or any air of music touch their ears,
Act 5 Scene 1 Speech 25: By the sweet power of music: therefore the poet
Act 5 Scene 1 Speech 25: But music for the time doth change his nature.
Act 5 Scene 1 Speech 25: The man that hath no music in himself,
Act 5 Scene 1 Speech 25: Let no such man be trusted. Mark the music.
Act 5 Scene 1 Speech 29: It is your music, madam, of the house.
Act 5 Scene 1 Speech 32: No better a musician than the wren.<jupyter_text>Instead of explicitly navigating down the hierarchy, we can also search for embedded elements. The example below examines the sequence of speakers to build a frequency distribution for speakers:<jupyter_code>from collections import Counter
speaker_seq = [s.text for s in merchant.findall("ACT/SCENE/SPEECH/SPEAKER")]
speaker_seq[:5]
speaker_freq = Counter(speaker_seq)
top5 = speaker_freq.most_common(5)
top5<jupyter_output><empty_output><jupyter_text>We can also look for patterns in which speakers follows who in the dialogues. We'll only do this for the top 5 speakers.<jupyter_code>from collections import defaultdict
abbreviate = defaultdict(lambda: 'OTH') # OTH = Other than top 5
for speaker, _ in top5:
    abbreviate[speaker] = speaker[:4]
speaker_seq2 = [abbreviate[speaker] for speaker in speaker_seq]
cfd = nltk.ConditionalFreqDist(nltk.bigrams(speaker_seq2))
cfd.tabulate()<jupyter_output>ANTO BASS GRAT  OTH PORT SHYL 
ANTO    0   11    4   11    9   12 
BASS   10    0   11   10   26   16 
GRAT    6    8    0   19    9    5 
 OTH    8   16   18  153   52   25 
PORT    7   23   13   53    0   21 
SHYL   15   15    2   26   21    0<jupyter_text>Using ElementTree for Accessing Toolbox DataWe can use the toolbox.xml() method to access a Toolbox file and load it into an elementtree object. This file contains a lexicon for the Rotokas language of Papua New Guinea.<jupyter_code>from nltk.corpus import toolbox
lexicon = toolbox.xml('rotokas.dic')<jupyter_output><empty_output><jupyter_text>There are two ways to access the contents of the lexicon object, by indexes and by paths. Indexes use the familiar syntax, thus lexicon[3] returns entry number 3.<jupyter_code>lexicon[3][0]
lexicon[3][0].tag
lexicon[3][0].text<jupyter_output><empty_output><jupyter_text>The second way to access the contents of the lexicon object uses paths. The lexicon is a series of record objects, each containing a series of field objects, such as lx and ps. We can conveniently address all of the lexemes using the path record/lx.<jupyter_code>[lexeme.text.lower() for lexeme in lexicon.findall('record/lx')][:10]<jupyter_output><empty_output><jupyter_text>Let's view the Toolbox data in XML format.<jupyter_code>import sys
from nltk.util import elementtree_indent
from xml.etree.ElementTree import ElementTree
elementtree_indent(lexicon)
tree = ElementTree(lexicon[3])
tree.write(sys.stdout, encoding="unicode")<jupyter_output><record>
    <lx>kaa</lx>
    <ps>N</ps>
    <pt>MASC</pt>
    <cl>isi</cl>
    <ge>cooking banana</ge>
    <tkp>banana bilong kukim</tkp>
    <pt>itoo</pt>
    <sf>FLORA</sf>
    <dt>12/Aug/2005</dt>
    <ex>Taeavi iria kaa isi kovopaueva kaparapasia.</ex>
    <xp>Taeavi i bin planim gaden banana bilong kukim tasol long paia.</xp>
    <xe>Taeavi planted banana in order to cook it.</xe>
  </record><jupyter_text>Working with Toolbox Data<jupyter_code>from nltk.corpus import toolbox
lexicon = toolbox.xml('rotokas.dic')<jupyter_output><empty_output><jupyter_text>Computing the average number of fields for each entry:<jupyter_code>sum(len(entry) for entry in lexicon) / len(lexicon)<jupyter_output><empty_output><jupyter_text>In this section we will discuss two tasks that arise in the context of documentary linguistics, neither of which is supported by the Toolbox software.  Adding a Field to Each Entry It is often convenient to add new fields that are derived automatically from existing ones. Such fields often facilitate search and analysis. For instance below, we define a function cv() which maps a string of consonants and vowels to the corresponding CV sequence, e.g. kakapua would map to CVCVCVV.<jupyter_code>from xml.etree.ElementTree import SubElement

def cv(s):
    s = s.lower()
    s = re.sub(r'[^a-z]',     r'_', s)
    s = re.sub(r'[aeiou]',    r'V', s)
    s = re.sub(r'[^V_]',      r'C', s)
    return (s)

def add_cv_field(entry):
    for field in entry:
        if field.tag == 'lx':
            cv_field = SubElement(entry, 'cv')
            cv_field.text = cv(field.text)
lexicon = toolbox.xml('rotokas.dic')
add_cv_field(lexicon[53])
print(nltk.toolbox.to_sfm_string(lexicon[53]))<jupyter_output>\lx kaeviro
\ps V
\pt A
\ge lift off
\ge take off
\tkp go antap
\sc MOTION
\vx 1
\nt used to describe action of plane
\dt 03/Jun/2005
\ex Pita kaeviroroe kepa kekesia oa vuripierevo kiuvu.
\xp Pita i go antap na lukim haus win i bagarapim.
\xe Peter went to look at the house that the wind destroyed.
\cv CVVCVCV<jupyter_text>Validating a Toolbox LexiconMany lexicons in Toolbox format do not conform to any particular schema. Some entries may include extra fields, or may order existing fields in a new way. Instead of manually inspecting thousands of lexical entries, we can identify frequent field sequences with the help of a `Counter`:<jupyter_code>from collections import Counter
field_sequences = Counter(':'.join(field.tag for field in entry) for entry in lexicon)
field_sequences.most_common()[:5]<jupyter_output><empty_output><jupyter_text>Now, we can use a grammar to validate entries when we iterate over them.<jupyter_code>grammar = nltk.CFG.fromstring('''
  S -> Head PS Glosses Comment Date Sem_Field Examples
  Head -> Lexeme Root
  Lexeme -> "lx"
  Root -> "rt" |
  PS -> "ps"
  Glosses -> Gloss Glosses |
  Gloss -> "ge" | "tkp" | "eng"
  Date -> "dt"
  Sem_Field -> "sf"
  Examples -> Example Ex_Pidgin Ex_English Examples |
  Example -> "ex"
  Ex_Pidgin -> "xp"
  Ex_English -> "xe"
  Comment -> "cmt" | "nt" |
  ''')
def validate_lexicon(grammar, lexicon, ignored_tags):
    rd_parser = nltk.RecursiveDescentParser(grammar)
    for entry in lexicon:
        marker_list = [field.tag for field in entry if field.tag not in ignored_tags]
        if list(rd_parser.parse(marker_list)):
            print("+", ':'.join(marker_list))
        else:
            print("-", ':'.join(marker_list))
lexicon = toolbox.xml('rotokas.dic')[10:20]
ignored_tags = ['arg', 'dcsv', 'pt', 'vx']
validate_lexicon(grammar, lexicon, ignored_tags)<jupyter_output>- lx:ps:ge:tkp:sf:nt:dt:ex:xp:xe:ex:xp:xe:ex:xp:xe
- lx:rt:ps:ge:tkp:nt:dt:ex:xp:xe:ex:xp:xe
- lx:ps:ge:tkp:nt:dt:ex:xp:xe:ex:xp:xe
- lx:ps:ge:tkp:nt:sf:dt
- lx:ps:ge:tkp:dt:cmt:ex:xp:xe:ex:xp:xe
- lx:ps:ge:ge:ge:tkp:cmt:dt:ex:xp:xe
- lx:rt:ps:ge:ge:tkp:dt
- lx:rt:ps:ge:eng:eng:eng:ge:tkp:tkp:dt:cmt:ex:xp:xe:ex:xp:xe:ex:xp:xe:ex:xp:xe:ex:xp:xe
- lx:rt:ps:ge:tkp:dt:ex:xp:xe
- lx:ps:ge:ge:tkp:dt:ex:xp:xe:ex:xp:xe
{'fraction_non_alphanumeric': 0.11063310645948375, 'fraction_numerical': 0.01528188005650443, 'mean_word_length': 4.74087725764836, 'pattern_counts': {'":': 0, '<': 131, '<?xml version=': 0, '>': 146, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 10}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '25', 'dir': 'jupyter-structured-clean-dedup', 'id': '607924', 'license': 'MIT', 'n_tokens_mistral': 5458, 'n_tokens_neox': 5143, 'n_words': 1759, 'path': '.ipynb_checkpoints/11. Managing Linguistic Data-checkpoint.ipynb', 'repo_name': 'sajal2692/working-through-nltk', 'size': '15574'}