<jupyter_start><jupyter_text>POS part-of speech 词性标注 任务描述给定一个字符序列，标注出每个字符的词性$$S = w_1,w_2.w_3,w_4,w_5$$$$Z=z_1,z_2,z_3,z_4,z_5$$ 解决方法现根据标注好的词性表进行训练，之后根据输入的字符序列，标注出各个字符对应的词性$$S'=w_1',w_2',w_3',w_4',w_5'$$$$Z'=?$$ 理论基础**Noisy Channel Model**$$\begin{align}P(Z|S) &\propto P(S|Z)P(Z)\\&=P(w_1w_2w_3...w_n|z_1z_2z_3...z_n)P(z_1z_2z_3...z_n)\\&=\prod_{i=1}^nP(w_i|z_i)\cdot P(z_1)P(z_2|z_1)...P(z_n|z_{n-1})\end{align}$$其中$\prod_{i=1}^nP(w_i|z_i)$被称为是条件概率，$P(z_1)P(z_2|z_1)...P(z_n|z_{n-1}$是语言模型。$$\begin{align}\hat Z &= argmaxP(Z|S)\\&=argmax\prod_{i=1}^nP(w_i|z_i)\cdot P(z_1)\prod_{j=2}^nP(z_j|z_{j-1})\\&\propto log(\prod_{i=1}^nP(w_i|z_i)\cdot P(z_1)\prod_{j=2}^nP(z_j|z_{j-1}))\\&=argmax\sum_{i=1}^nlog(w_i|z_i)+logP(z_1)+\sum_{j=2}^nlogP(z_j|z_{j-1})\end{align}$$为了求出$\hat Z$则需要计算上述公式，我们需要计算$\sum_{i=1}^nlog(w_i|z_i)$，$logP(z_1)$，$\sum_{j=2}^nlogP(z_j|z_{j-1})$，分别记为$A, \pi, B$<jupyter_code>tag2id, id2tag = {}, {}# 词性字典，词性转id，id转词性
word2id, id2word = {}, {}# 单词id字典，id单词字典
for line in open('traindata.txt','r',encoding='utf-8'):# 读取文件，生成相关字典
    items = line.split('/')
    word, tag = items[0],items[1].strip()
    
    if word not in word2id:
        word2id[word] = len(word2id)
        id2word[len(id2word)] = word
    if tag not in tag2id:
        tag2id[tag] = len(tag2id)
        id2tag[len(id2tag)] = tag
M = len(word2id) # M:词典的大小，# of words in dictionary
N = len(tag2id) # N：词性的种类个数，# of tags in tag dictionary
print(M,N)
# print(word2id, id2word)
# print(tag2id, id2tag)<jupyter_output>18978 54<jupyter_text>构建$A, B, \pi$<jupyter_code>import numpy as np
pi = np.zeros(N) # 初始化pi向量，表示出现在第一个位置的各个词性的概率
A = np.zeros((N,M)) # A[i][j]表示在第i个词性下单词是第j个的概率 ，发射矩阵
B = np.zeros((N,N)) # B[i][j]表示从第j个词性转移到第i个词性的概率，转移矩阵
# 根据训练数据，统计相关的数据，然后标准化，得到相应的A, B, pi
prev_tag = ""
for line in open("traindata.txt","r",encoding="utf-8"):
    items = line.split("/")
    wordID, tagID = word2id[items[0]],tag2id[items[1].strip()]
    if prev_tag == "":
        pi[tagID] += 1
        A[tagID][wordID] += 1
        prev_tag = tagID
    else:
        A[tagID][wordID] += 1
        B[tag2id[prev_tag]][tagID] += 1
    if items[0] == ".":
        prev_tag = ""
    else:
        prev_tag = items[1].strip()
# normalization
pi = pi/sum(pi)
for i in range(N):
    A[i] /= sum(A[i])
    B[i] /= sum(B[i])
print(A[0])
print(pi)
def log(v):# 计算log函数值
    if v == 0:
        return np.log(v+0.0001)
    else:
        return np.log(v)<jupyter_output><empty_output><jupyter_text>使用viterbi算法计算<jupyter_code>def viterbi(x, pi, A, B):
    x = [word2id[word] for word in x.split(" ")]#对输入的句子进行切分
    T = len(x)
    
    dp = np.zeros((T,N))
    for j in range(N):
        dp[0][j] = log(pi[j])+log(A[j][x[0]])# 矩阵第一列初始化
    ptr = np.array([[0 for x in range(N)] for y in range(T)])
    for i in range(1, T): # 每个单词
        for j in range(N): # 每个词性
            dp[i][j] = -99999
            for k in range(N): # 从每个k到达j
                score = dp[i-1][k] + log(B[k][j]) + log(A[j][x[i]])
                if score > dp[i][j]:
                    dp[i][j] = score
                    ptr[i][j] = k
    # decoding:把做好的路径打印出来
    best_seq = [0]*T
    # step1: 找出对应于最后一个单词的词性
    best_seq[T-1] = np.argmax(dp[T-1])
    for i in range(T-2, -1, -1):
        best_seq[i] = ptr[i+1][best_seq[i+1]]
    # 将词性序列打印出来
    for i in range(len(best_seq)):
        print(id2tag[best_seq[i]])
x = "Social security number , passport number and details about the services provided for the payment"
viterbi(x,pi,A,B)<jupyter_output>DT
NN
NN
,
DT
NN
CC
NNS
IN
DT
NNS
VBN
IN
DT
NN
{'fraction_non_alphanumeric': 0.21240441801189464, 'fraction_numerical': 0.033984706881903144, 'mean_word_length': 3.604954367666232, 'pattern_counts': {'":': 2, '<': 11, '<?xml version=': 0, '>': 12, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '607463', 'license': 'Apache-2.0', 'n_tokens_mistral': 1910, 'n_tokens_neox': 1976, 'n_words': 257, 'path': 'Part-Of-Speech/词性标注实战.ipynb', 'repo_name': 'CHNcreater/NLP_crash_course', 'size': '3531'}