<jupyter_start><jupyter_text>Imports and Initialisations<jupyter_code>import nltk
nltk.download('stopwords')

from nltk.corpus import brown, stopwords
import xml.etree.ElementTree as ET
import string

sents = brown.sents()
stopwords = set(stopwords.words('english'))
tree = ET.parse('../data/test/subtask1-homographic-test.xml')
puncts = string.punctuation
F = open("../data/test/subtask1-homographic-test.gold")
scores = list()
for item in F.readlines():
    ans = item.split()[1]
    scores.append(ans)
test_sents = list()
root = tree.getroot()
for text in root.getchildren():
    sent = list()
    for word in text.getchildren():
        sent.append(word.text)
    test_sents.append(sent)
def filter_sent(sent):
    sent = [w for w in sent if w.lower() not in stopwords]
    sent = [w for w in sent if w[0] not in puncts]
    return sent

def make_ordered_pairs(sent):
    ans = list()
    for index, word in enumerate(sent):
        if index == len(sent) - 1:
            break
        temp = index + 1
        while(1):
            ans.append((sent[index], sent[temp]))
            if temp == len(sent) - 1:
                break
            temp += 1
    return ans
from collections import Counter

words = Counter([])
bigrams = Counter([])

for sent in test_sents:
    sent = filter_sent(sent)
    pairs = make_ordered_pairs(sent)
    words.update(sent)
    bigrams.update(pairs)
    
print(words.most_common(10))
print(bigrams.most_common(10))<jupyter_output>[('never', 230), ('die', 201), ('OLD', 178), ('get', 101), ('good', 66), ('always', 62), ('name', 57), ('said', 57), ('Tom', 56), ('got', 51)]
[(('never', 'die'), 194), (('OLD', 'die'), 178), (('OLD', 'never'), 177), (('never', 'lose'), 40), (('die', 'lose'), 40), (('OLD', 'lose'), 38), (('never', 'get'), 36), (('die', 'get'), 34), (('OLD', 'get'), 33), (('Doctor', 'Next'), 19)]<jupyter_text>Calculating PMIThe PMI (Pointwise Mututal Information) is being calculated as:```PMI(w1, w2) = log (P(w1, w2) / (P(w1) * P(w2)))```<jupyter_code>import math

def PMI(tup):
    try:
        val = bigrams[tup] / (words[tup[0]] * words[tup[1]])
        return math.log(val)
    except:
        return 0<jupyter_output><empty_output><jupyter_text>Calculating ThresholdThe threshold for the difference in the highest PMI and is calculated using the Interquartile Range (IQR). IQR is preferred because it is able to eliminate outliers.<jupyter_code>import numpy as np
import scipy.stats as sp

IQRs = list()
for sent in test_sents:
    sent = filter_sent(sent)
    pairs = make_ordered_pairs(sent)
    PMIs = sorted([PMI(_) for _ in pairs])
    l = len(PMIs)
    if l > 0:
        IQRs.append(sp.iqr(PMIs))

IQRs = sorted(IQRs)
threshold = np.median(IQRs)
print(threshold)<jupyter_output>1.7107470400723912<jupyter_text>Testing Sentences for Pun- Converting each sentence into tokens- Stopword Removal- Generating word pairs preserving word order- Calculating PMI score and checking whether above threshold- Checking whether any word from pair has multiple sense<jupyter_code>from nltk.corpus import wordnet as wn

def has_multiple_sense(word):
    syns = Counter([_.name().split('.')[0] for _ in wn.synsets(word)])
    return syns[word] > 1

test_scores = list()
for i, sent in enumerate(test_sents):
    sent_ = sent
    sent = filter_sent(sent)
    pairs = make_ordered_pairs(sent)
    PMIs = sorted([(_, PMI(_)) for _ in pairs], key=lambda k : k[1])
    l = len(PMIs)
    if l > 0:
        if sp.iqr([v for k, v in PMIs]) > threshold:
            if has_multiple_sense(PMIs[-1][0][0]) or has_multiple_sense(PMIs[-1][0][1]):
                test_scores.append('1')
            else:
                test_scores.append('0')
        else:
            test_scores.append('0')
    else:
        test_scores.append('0')
        
print(len(test_scores))<jupyter_output>2250<jupyter_text>Calculating Precision, Recall, F1-Score```Precision = TP/TP+FPRecall = TP/TP+FNF1-Score = 2*(Recall*Precision)/(Recall+Precision)```<jupyter_code>t_scores = Counter([(scores[i], test_scores[i]) for i in range(len(scores))])

TP = t_scores[('1', '1')]
FP = t_scores[('0', '1')]
FN = t_scores[('1', '0')]

pre = TP / (TP + FP)
re = TP / (TP + FN)
f1 = 2 * (re * pre) / (re + pre)

print('precision', pre)
print('recall', re)
print('F1-score', f1)<jupyter_output>precision 0.8181818181818182
recall 0.35283136278780336
F1-score 0.4930434782608695<jupyter_text>Manually Checking Failed Cases<jupyter_code>fail_cases = []

for i in range(len(scores)):
    if(scores[i] == '1' and test_scores[i] == '0'):
        fail_cases.append(i)
#fail_cases
for i in fail_cases[0:10]:
    for j in test_sents[i]:
        print(j, end = ' ')
    print('')<jupyter_output>They hid from the gunman in a sauna where they could sweat it out . 
Wal - Mart isn ' t the only saving place ! 
Can honeybee abuse lead to a sting operation ? 
A ditch digger was entrenched in his career . 
Did you hear about the new pinata ? It ' s a huge hit . 
She was suspected of stealing a brooch but they couldn ' t pin it on her . 
' ' There ' s room for one more , ' ' Tom admitted . 
They threw a party for the inventor of the toaster . And he was toasted . 
If you ' re a gardener you might call yourself a ' plant manager ' . 
My advanced geometry class is full of squares .
{'fraction_non_alphanumeric': 0.14756582686114794, 'fraction_numerical': 0.03333964766054177, 'mean_word_length': 3.6397188049209137, 'pattern_counts': {'":': 0, '<': 20, '<?xml version=': 0, '>': 24, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 2}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '610077', 'license': 'MIT', 'n_tokens_mistral': 2034, 'n_tokens_neox': 1840, 'n_words': 535, 'path': 'src/Detecting_Homographic_Puns.ipynb', 'repo_name': 'arghyatiger/Detecting-Homographic-Puns', 'size': '5279'}