<jupyter_start><jupyter_text>Language Modeling using Ngram In this Exercise, you are going to use NLTK which is a natural language processing library for python to create a bigram language model and its variation. You will build one model for each of the following type and calculate their perplexity:- Unigram Model- Bigram Model- Bigram Model with Laplace smoothing- Bigram Model with InterpolationAs a reminder, Don't forget to shut down your instance on Gcloud when you are not using it<jupyter_code>#First we import necessary library such as math, nltk, bigram, and collections.
import math
import nltk
import io
import random
from random import shuffle
from nltk import bigrams, trigrams
from collections import Counter, defaultdict
random.seed(999)<jupyter_output><empty_output><jupyter_text>BEST2010 is a free Thai NLP dataset by NECTEC usually use as a standard benchmark for various NLP tasks includeing language modeling. BEST2010 is separated into 4 domain article, encyclopedia, news and novel. The data is already  tokenized using '|' as a separator.For example,ตาม|ที่|นางประนอม ทองจันทร์| |กับ| |ด.ช.กิตติพงษ์ แหลมผักแว่น| |และ| |ด.ญ.กาญจนา กรองแก้ว| |ป่วย|สงสัย|ติด|เชื้อ|ไข้|ขณะ|นี้|ยัง|ไม่|ดี|ขึ้น|<jupyter_code># We choose news domain as our dataset
best2010=[]
fp= io.open('BEST2010/news.txt','r',encoding='utf-8')
for i,line in enumerate(fp):
    best2010.append(line.strip()[:-1])
fp.close()
vocabuary =set()
total_word_count =0
for line in best2010:
    for word in line.split('|'):        
        vocabuary.add(word)
        total_word_count+=1
#For simplicity, we assumes that each line is a sentence.
print ('Total sentences in BEST2010 news dataset :\t'+ str(len(best2010)))
print ('Total word counts in BEST2010 news dataset :\t'+ str(total_word_count))
print ('Total vocabuary in BEST2010 news dataset :\t'+ str(len(vocabuary)))<jupyter_output>Total sentences in BEST2010 news dataset :	30969
Total word counts in BEST2010 news dataset :	1660190
Total vocabuary in BEST2010 news dataset :	35488<jupyter_text>We separate out input into 2 sets, train and test data with 70:30 ratio<jupyter_code>sentences = best2010
# The data is separated to train and test set with 70:30 ratio.
train = sentences[:int(len(sentences)*0.7)]
test = sentences[int(len(sentences)*0.7):]

# We will use 1/total word count as a default value for unknown word
unk_value = math.pow(len(vocabuary),-1)<jupyter_output><empty_output><jupyter_text>Unigram In this section, we will demonstrate how to build a unigram language model<jupyter_code>def getUnigramModel(data):
    model = defaultdict(lambda: 0)
    total_word_count =0
    for sentence in data:
        sentence +=  u'|</s>'
        for w1 in sentence.split('|'):
            model[w1] +=1.0
            total_word_count+=1
    for w1 in model:
        model[w1] = model[w1]/total_word_count
    return model
model = getUnigramModel(train)
def getLnValue(x):
    if x >0.0:
        return math.log(x)
    else:
        return math.log(unk_value)
#problability of 'นายก'
print(getLnValue(model[u'นายก']))
#for example, problability of 'นายกรัฐมนตรี' which is an unknown word is equal to
print(getLnValue(model[u'นายกรัฐมนตรี']))
#problability of 'นายก' 'ได้' 'ให้' 'สัมภาษณ์' 'กับ' 'สื่อ'
prob = getLnValue(model[u'นายก'])+getLnValue(model[u'ได้'])+ getLnValue(model[u'ให้'])+getLnValue(model[u'สัมภาษณ์'])+getLnValue(model[u'กับ'])+getLnValue(model[u'สื่อ'])
print ('Problability of a sentence', math.exp(prob))<jupyter_output>-6.551526663995246
-10.476949890150093
Problability of a sentence 2.75827124812635e-16<jupyter_text>PerplexityIn order to compare language model we need to calculate perplexity. In this task you should write a perplexity calculation code for the unigram model. The result perplexity should be around 513.97 and452.66 on train and test data.  TODO 1 Calculate perplexity<jupyter_code>def calculate_sentence_ln_prob(sentence, model):
    word = sentence.split('|')
    ln_prob = .0
    for i in range(0,len(word)-1):
        ln_prob +=(getLnValue(model[word[i]]))
    return ln_prob

def perplexity(test,model):
    ln_prob  = .0
    word_count = 0
    for sentence in test:
        sentence += u'|</s>'
        word_count += len(sentence.split('|'))
        ln_prob    += calculate_sentence_ln_prob(sentence,model)
    return math.exp(-ln_prob/word_count)
print(perplexity(train,model))
print(perplexity(test,model))<jupyter_output>513.9747870973313
452.65524668351003<jupyter_text>TODO 2 Please explain why this model give us such a high perplexity. **Your answer**:  Write your answer here.  Bigram Next, you will create a better language model than a unigram (which is not much to compare with). But first, it is very tedious to count every pair of words that occur in our corpus by ourselves. In this case, nltk provide us a simple library which will do it for us.<jupyter_code>#example of nltk usage for bigram
sentence = 'I always search google for an answer .'

print('This is how nltk generate bigram.')
for w1,w2 in bigrams(sentence.split(), pad_right=True, pad_left=True):
    print (w1,w2)
print('None is used as a start and end of sentence symbol.')<jupyter_output>This is how nltk generate bigram.
None I
I always
always search
search google
google for
for an
an answer
answer .
. None
None is used as a start and end of sentence symbol.<jupyter_text>Now, you should be able to implement a bigram model by yourself. Also, you must create a new perplexity calculation for bigram. The result perplexity should be around 58.54 and 153.36 on train and test data.  TODO 3 Write Bigram Model<jupyter_code>def getBigramModel(data):
    model = defaultdict(lambda: defaultdict(lambda: 0.0))
    for sentence in data:
        for w1, w2 in bigrams(sentence.split('|'), pad_right=True, pad_left=True): #None I go to school . None
            model[w1][w2] +=1.0
    for w1 in model:
        c_w1 = float(sum(model[w1].values()))
        for w2 in model[w1]:
            model[w1][w2] = model[w1][w2]/c_w1
    return model
model = getBigramModel(train)<jupyter_output><empty_output><jupyter_text>TODO 4 Write Perplexity for Bigram Model<jupyter_code>def perplexity(data,model):
    sumlog = 0
    numbigram = 0
    for sentence in data:
        for w1,w2 in bigrams(sentence.split('|'), pad_right=True, pad_left=True):
            sumlog += getLnValue(model[w1][w2])
            numbigram += 1
    return math.exp(-sumlog/numbigram)
print (perplexity(train,model))
print (perplexity(test, model))<jupyter_output>58.78942889762667
153.76867837686785<jupyter_text>Smoothing Usually any ngram models have a sparsity problem, which means it does not have every possible ngram of words in the dataset. Smoothing techniques can alleviate this problem. In this section, you will implement two basic smoothing methods laplace smoothing and interpolation for bigram.  TODO 5 write Bigram with Laplace smoothing (Add-One Smoothing)<jupyter_code>#Laplace Smoothing
def getBigramWithLaplaceSmoothing(data):
    model = defaultdict(lambda: defaultdict(lambda: 0.0))
    for sentence in data:
        for w1, w2 in bigrams(sentence.split('|'), pad_right=True, pad_left=True):
            model[w1][w2] += 1.0
    for w1 in model:
        c_w1 = float(sum(model[w1].values()))
        for w2 in model[w1]:
            model[w1][w2] = (model[w1][w2]+1)/(c_w1+len(vocabuary))
    return model
model = getBigramWithLaplaceSmoothing(train)
print (perplexity(train,model))
print (perplexity(test, model))<jupyter_output>1242.629211431225
1398.9466787838792<jupyter_text>TODO 6 Write Bigram with Interpolationlambda value is 0.7 for bigram, 0.25 for unigram, and 0.05 for unknown word<jupyter_code>#interpolation
def getBigramWithInterpolation(data):
    model = defaultdict(lambda: defaultdict(lambda: 0.0))
    word_count = 0
    for sentence in data:
        word_count += len(sentence.split('|'))
        for w1, w2 in bigrams(sentence.split('|'), pad_right=True, pad_left=True):
            model[w1][w2] += 1.0
    for w1 in model:
        c_w1 = float(sum(model[w1].values()))
        for w2 in model[w1]:
            model[w1][w2] = 0.7*model[w1][w2]/c_w1+0.25*c_w1/word_count+0.05*unk_value
    return model
model = getBigramWithInterpolation(train)
print (perplexity(train,model))        
print (perplexity(test,model))<jupyter_output>39.87144272053742
107.07077317720784<jupyter_text>the result perplexity on training and testing should be     1231.14, 1390.85 for Laplace smoothing    39.90, 107.27 for Interpolation  Language modeling on multiple domainsSometimes, we do not have enough data to create a language model for a new domain. In that case, we can improvised by combining several models to improve result on the new domain.In this exercise you will try to merge two language model from news and article domains to create a language model for the encyclopedia domain.<jupyter_code>encyclo_data=[]
fp= io.open('BEST2010/encyclopedia.txt','r',encoding='utf-8')
for i,line in enumerate(fp):
    encyclo_data.append(line.strip()[:-1])
fp.close()

news_data=[]
fp= io.open('BEST2010/news.txt','r',encoding='utf-8')
for i,line in enumerate(fp):
    news_data.append(line.strip()[:-1])
fp.close()

article_data=[]
fp= io.open('BEST2010/article.txt','r',encoding='utf-8')
for i,line in enumerate(fp):
    article_data.append(line.strip()[:-1])
fp.close()<jupyter_output><empty_output><jupyter_text>First, you should try to calculate perplexity of your bigram with interpolation on article data. The result perplexity should be around 466.17 For your information, a bigram model with interpolation using ariticle data to test on encyclopedia data has a perplexity of 281.24<jupyter_code>train = news_data + article_data
test = encyclo_data

vocabulary = set()
train_word_count =0
for line in train:
    for word in line.split('|'):        
        vocabulary.add(word)
        train_word_count+=1
        
unk_value = 1.0/len(vocabulary)<jupyter_output><empty_output><jupyter_text>TODO 7 Write a model that produce 260.0 or less perplexity on encyclopedia data without using data from the encyclopedia as training data. (Hint : Try to combine a model with news data and a model with article data together.)<jupyter_code>def getBigramWithInterpolation2(data):
    model = defaultdict(lambda: defaultdict(lambda: 0.0))
    word_count = 0
    for sentence in data:
        word_count += len(sentence.split('|'))
        for w1, w2 in bigrams(sentence.split('|'), pad_right=True, pad_left=True):
            model[w1][w2] += 1.0
    for w1 in model:
        unigram = float(sum(model[w1].values()))
        for w2 in model[w1]:
            model[w1][w2] = 0.7*model[w1][w2]/unigram+0.25*unigram/word_count+0.05*unk_value
    return model
model = getBigramWithInterpolation2(train)
print (perplexity(train,model))
print (perplexity(test,model))<jupyter_output>44.509959183889706
274.9606303745181
{'fraction_non_alphanumeric': 0.10512278120114044, 'fraction_numerical': 0.04800882920996965, 'mean_word_length': 4.559304703476482, 'pattern_counts': {'":': 0, '<': 47, '<?xml version=': 0, '>': 48, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '357597', 'license': 'MIT', 'n_tokens_mistral': 4019, 'n_tokens_neox': 3693, 'n_words': 1121, 'path': 'HW2/LanguageModeling.ipynb', 'repo_name': 'First4196/NLP', 'size': '10873'}