<jupyter_start><jupyter_text>使用HMM进行词性标注假设我们用NLTK自带的Brown词库进行学习* 单词集：words=w1,...wN* tags集：tags=t1...tN* P(tags|words)正比于P(ti|t{i-1})* P(wi|ti)为了找一个句子的tag,我们其实就是找的最好的一套tags,让他最能干符合给定的单词（words）  导入需要的库<jupyter_code>import nltk
# nltk.download('brown')
import sys
from nltk.corpus import brown<jupyter_output><empty_output><jupyter_text>预处理词库 这里我们需要做的预处理是：给词们加上开始和结束符号。Brown里面的句子都是自己标注好了的，长这个样子：（I,NOUN），（LOVE，VERB），（YOU，NOUN）那么我们的开始符号也和他的格式符合，我们用：（START，START）（END，END）来表示<jupyter_code>brown_tags_words=[ ]
for sent in brown.tagged_sents():
    # 先加开头
    brown_tags_words.append(("START","START"))
    # 为了省事儿，我们把tag都省略成前两个字母
    brown_tags_words.extend([ (tag[:2],word) for (word,tag) in sent])
    #加个结尾
    brown_tags_words.append(("END","END"))<jupyter_output><empty_output><jupyter_text>词统计 将词库中拥有的单词与tag之间的关系，走个简单粗暴的统计，即统计P(wi|ti)=count(wi,ti)/count(ti)你可以自己一个个的loop全部的corpus,这里直接使用NLTK给我们提供饿统计工具<jupyter_code>#conditional frequency ditribution
cfd_tagwords = nltk.ConditionalFreqDist(brown_tags_words)
# conditioal probability distribution
cpd_tagwords = nltk.ConditionalProbDist(cfd_tagwords,nltk.MLEProbDist)
# 思考，如果自己手动实现这个loop来进行统计，如何写出这个词频统计？？？
# 查看平面统计下来的结果
print("The probability of an adjective(JJ) beding 'new' is",cpd_tagwords["JJ"].prob("new"))
print("The probability of an verb(VB) beding 'duck' is",cpd_tagwords["VB"].prob("duck"))<jupyter_output>The probability of an adjective(JJ) beding 'new' is 0.01472344917632025
The probability of an verb(VB) beding 'duck' is 6.042713350943527e-05<jupyter_text>计算$P(t_i|t_{i-1})= \frac{count(t_{i-1},t_i)}{count(t_{i-1})}$这个公式跟words没有任何关系，它是隐层的马科夫链，所以我们线去除所有的tag<jupyter_code>brown_tags = [tag for (tag,word) in brown_tags_words]
# count(t{i-1},ti),bigram是将前后两个一组，连在一起
cfd_tags = nltk.ConditionalFreqDist(nltk.bigrams(brown_tags))
# P(ti|t{i-1})
cpd_tags = nltk.ConditionalProbDist(cfd_tags,nltk.MLEProbDist)
# 看效果
print("if we have just seen 'DT',the probability of 'NN' is",cpd_tags["DT"].prob("NN"))
print("if we have just seen 'VB',the probability of 'JJ' is",cpd_tags["VB"].prob("DT"))
print("if we have just seen 'VB',the probability of 'NN' is",cpd_tags["VB"].prob("NN"))<jupyter_output>if we have just seen 'DT',the probability of 'NN' is 0.5057722522030194
if we have just seen 'VB',the probability of 'JJ' is 0.016885067592065053
if we have just seen 'VB',the probability of 'NN' is 0.10970977711020183<jupyter_text>一些有趣的结果  验证问题比如，一句话，”I want to race“,一套tag,"PP VB TO VB",她们呢之间的匹配度有多高呢？其实就是：    P(START) * P(PP|START) * P(I|PP) *          P(VB|PP) * P(want|VB) *          P(TO|VB) * P(to|TO) *          P(VB|TO) * P(race|VB) *          P(END|VB)<jupyter_code>prob_tagsequence = cpd_tags["START"].prob("PP") * cpd_tagwords["PP"].prob("I") * \
  cpd_tags["PP"].prob("VB") * cpd_tagwords["VB"].prob("want") * \
  cpd_tags["VB"].prob("TO") * cpd_tagwords["TO"].prob("to") * \
  cpd_tags["TO"].prob("VB") * cpd_tagwords["VB"].prob("race") * \
  cpd_tags["VB"].prob("END")
print("The probability of the tag sequence 'START PP VB TO VB END' for 'I want to race' is:",prob_tagsequence)<jupyter_output>The probability of the tag sequence 'START PP VB TO VB END' for 'I want to race' is: 1.0817766461150474e-14<jupyter_text>Viterbi的实现如果我们手上有一句话，怎么知道最符合的tag是哪组呢？---识别问题。<jupyter_code># 首先，我们拿出所有独特的tags(即tags全集)
distinct_tags = set(brown_tags)
# 然后，随便找一句话
sentence = ["I","want","to","race"]
sentlen = len(sentence)
# 接下来，开始维特比：从1循环到句子的总长N,记为i;每次都找出以tag X为最终节点，长度为i的tag链
viterbi = [ ]
# 同时还需要一个回溯器：从1循环到句子总长N，记为i；把所有tag X前一个Tag记录下来
backpoint = [ ]
first_viterbi = { }
first_backpointer = { }
for tag in distinct_tags:
    if tag == "START": continue
    first_viterbi[tag] = cpd_tags["START"].prob(tag) * cpd_tagwords[tag].prob(sentence[0])
    first_backpointer[tag] = "START"
    
print(first_viterbi)
print(first_backpointer)
#  'NP': 'START'  'NP': 1.7319067623793952e-06
#  'NI': 'START'  'NI': 3.3324520848931064e-07
#  'NN': 'START'   'NN': 1.0580313619573935e-06
#  'PP': 'START'   'PP': 0.014930900689060006   ---这个概率最大，这个就是第一个viterbit 对应的单词是sentence[0]=I
        
# {')-': 0.0, 'CS': 0.0, 'DO': 0.0, 'RB': 0.0, 'AB': 0.0, "'": 0.0, 'BE': 0.0, ',-': 0.0, '``': 0.0, 'END': 0.0,
#  'WP': 0.0, 'AT': 0.0, '.-': 0.0, 'EX': 0.0, 'JJ': 0.0, 'TO': 0.0, '.': 0.0, 'FW': 0.0, ',': 0.0, 'AP': 0.0,
#  'CD': 0.0, '(': 0.0, 'HV': 0.0, 'WR': 0.0, 'WD': 0.0, 'CC': 0.0, ':-': 0.0, 'NP': 1.7319067623793952e-06, 'MD': 0.0, 'RN': 0.0,
#  'OD': 0.0, 'VB': 0.0, 'UH': 0.0, ':': 0.0, '*-': 0.0, 'NI': 3.3324520848931064e-07, 'NR': 0.0, 'WQ': 0.0, 'QL': 0.0, ')': 0.0, 
#  '(-': 0.0, '--': 0.0, 'NN': 1.0580313619573935e-06, 'IN': 0.0, '*': 0.0, 'PN': 0.0, 'RP': 0.0, "''": 0.0, 'PP': 0.014930900689060006, 'DT': 0.0}
# {')-': 'START', 'CS': 'START', 'DO': 'START', 'RB': 'START', 'AB': 'START', "'": 'START', 'BE': 'START', ',-': 'START', '``': 'START', 'END': 'START', 
#  'WP': 'START', 'AT': 'START', '.-': 'START', 'EX': 'START', 'JJ': 'START', 'TO': 'START', '.': 'START', 'FW': 'START', ',': 'START', 'AP': 'START',
#  'CD': 'START', '(': 'START', 'HV': 'START', 'WR': 'START', 'WD': 'START', 'CC': 'START', ':-': 'START', 'NP': 'START', 'MD': 'START', 'RN': 'START',
#  'OD': 'START', 'VB': 'START', 'UH': 'START', ':': 'START', '*-': 'START', 'NI': 'START', 'NR': 'START', 'WQ': 'START', 'QL': 'START', ')': 'START',
#  '(-': 'START', '--': 'START', 'NN': 'START', 'IN': 'START', '*': 'START', 'PN': 'START', 'RP': 'START', "''": 'START', 'PP': 'START', 'DT': 'START'}<jupyter_output><empty_output><jupyter_text>以上，是所有的第一个viterbi和第一个回溯点。接下来，把这些存到Viterbi和Backpointer两个变量里面<jupyter_code>viterbi.append(first_viterbi)
backpoint.append(first_backpointer)
# 查看第一个viterbi最好的tag是啥
currentbest = max(first_viterbi.keys(),key= lambda tag: first_viterbi[tag])
print("Word ","'"+sentence[0]+"'","current best two-tag sequence:",first_backpointer[currentbest],currentbest)<jupyter_output>Word  'I' current best two-tag sequence: START PP<jupyter_text>接下来我们开始通过loop来计算出所有的viterbi和backpointer.为什么自己的写的代码就不行？？？<jupyter_code># for wordindex in range(1,len(sentence)):
#     this_viterbi = { }  #  链的概率值currentbest，对应tag X
#     this_backpointer = { }  # 上一个链的概率值best_previous,对应tag Y
#     prev_viterbi = viterbi[-1]  #
     
#     for tag in distinct_tags:
#         if tag == "START": continue  # 注意这里的缩进层次问题
#         best_previous = max(prev_viterbi.keys(),key = lambda prevtag: \
#             prev_viterbi[prevtag] * cpd_tags[prevtag].prob(tag) * cpd_tagwords[tag].prob(sentence[wordindex]))
#         this_backpointer[tag] = best_previous
        
#         this_viterbi[tag] = prev_viterbi[best_previous] * cpd_tags[best_previous].prob(tag) * cpd_tags[tag].prob(sentence[wordindex])
#     # 每次找完Y,我们把目前最好的存一下
#     currentbest = max(this_viterbi.keys(), key = lambda tag: this_viterbi[tag])
#     print("Word","'" + sentence[wordindex] + "'","cunrrent best two-tag sequence:",this_backpointer[currentbest],currentbest)
    
#     # 完结 ，全部存下来
#     viterbi.append(this_viterbi)
#     backpoint.append(this_backpointer)
for wordindex in range(1, len(sentence)):
    this_viterbi = { }
    this_backpointer = { }
    prev_viterbi = viterbi[-1]
    
    for tag in distinct_tags:
        # START没有卵用的，我们要忽略
        if tag == "START": continue
        
        # 如果现在这个tag是X，现在的单词是w，
        # 我们想找前一个tag Y，并且让最好的tag sequence以Y X结尾。
        # 也就是说
        # Y要能最大化：
        # prev_viterbi[ Y ] * P(X | Y) * P( w | X)
        
        best_previous = max(prev_viterbi.keys(),
                            key = lambda prevtag: \
            prev_viterbi[ prevtag ] * cpd_tags[prevtag].prob(tag) * cpd_tagwords[tag].prob(sentence[wordindex]))
        this_backpointer[ tag ] = best_previous

        this_viterbi[ tag ] = prev_viterbi[ best_previous] * \
            cpd_tags[ best_previous ].prob(tag) * cpd_tagwords[ tag].prob(sentence[wordindex])
        
    
    # 每次找完Y 我们把目前最好的 存一下
    currbest = max(this_viterbi.keys(), key = lambda tag: this_viterbi[ tag ])
    print( "Word", "'" + sentence[ wordindex] + "'", "current best two-tag sequence:", this_backpointer[ currbest], currbest)


    # 完结
    # 全部存下来
    viterbi.append(this_viterbi)
    backpoint.append(this_backpointer)<jupyter_output>Word 'want' current best two-tag sequence: PP VB
Word 'to' current best two-tag sequence: VB TO
Word 'race' current best two-tag sequence: IN NN<jupyter_text>找END，结束：<jupyter_code># 找所有以END结尾的tag sequence
prev_viterbi = viterbi[-1]
best_previous = max(prev_viterbi.keys(),
                   key = lambda prevtag: prev_viterbi[prevtag] * cpd_tags[prevtag].prob("END"))

prob_tagsequence = prev_viterbi[best_previous] * cpd_tags[best_previous].prob("END")
best_tagsequence = ["END",best_previous]  # 我们这会儿是倒着存的，因为好的在后面
backpoint.reverse()  # 同理，这里也要倒着来过来<jupyter_output><empty_output><jupyter_text>最终： 回溯所有的回溯点。此时，最好的tag就是backpointer里面的current best<jupyter_code>current_best_tag = best_previous
for bp in backpoint:
    best_tagsequence.append(bp[current_best_tag])
    current_best_tag = bp[current_best_tag]
# 显示结果
best_tagsequence.reverse()
print("The sentence was:",end=" ")
for w in sentence: print(w,end=" ")
print("\n")
print("The best tag sequence is:",end=" ")
for t in best_tagsequence: print(t,end= " ")
print("\n")
print("The probility of the best tag sequence is:",prob_tagsequence)<jupyter_output>The sentence was: I want to race 

The best tag sequence is: START PP VB IN NN END 

The probility of the best tag sequence is: 5.71772824864617e-14
{'fraction_non_alphanumeric': 0.18753986816925367, 'fraction_numerical': 0.0413565809058048, 'mean_word_length': 5.037869062901155, 'pattern_counts': {'":': 7, '<': 35, '<?xml version=': 0, '>': 35, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '34591', 'license': 'Apache-2.0', 'n_tokens_mistral': 4518, 'n_tokens_neox': 4381, 'n_words': 685, 'path': 'MachineLearning_9/13_NBandHMM/HMM_tags_words.ipynb', 'repo_name': 'zengqingxue/ML_cases', 'size': '9406'}