<jupyter_start><jupyter_text>**Naive Bayes on Text 实现基于朴素贝叶斯的文本分类**- 贝叶斯公式:  - $P(y|x) = \frac{P(x|y)P(y)}{P(x)}$- 在text data上应用朴素贝叶斯方法  - 朴素贝叶斯：假定各特征对结果的影响是相互独立的  - 数据    - 预处理      - 分词      - 去停用词      - 建立词典，每个满足一定阈值（出现的频率）的词会被写入词典，最后词典的key value是单词和index    - BOW词袋模型，用向量表示词语      - 建立BOW        - 初始化，以doc数量为纵轴，vocab的数量（词典长度）横轴，用scipy.sparse的lil_matrix建表data_matrix        - 对每个doc的每个word，根据dict查每个word对应vocab的index，并在横轴标记1        - 遍历完毕，表中每一行是一个doc，每一行为1的是该doc中出现的所有word在dict中的位置        - 将data_matrix转化为scipy.sparse的tocsr转换为csr matrix存储      - BOW矩阵的shape是|N|\*|V|, N是doc的数量，V是vocab的数量    - 对标签数据做one-hot处理，最后标签矩阵的shape是|N|\*|K|, K是标签数量  - 文本贝叶斯    - 求P(y):       - 先求label的frequency: np.sum(label,axis=0,keepdim=True),累加每行算出每个标签出现的频数      - 再求label的probability: label_freq / np.sum(label_freq,axis=0,keepdim=False)，归一化，每个标签的频数除以频数总数得到概率。      - 最后P(y)为(|k|,)因为keepdim为False    - 求P(x|y):      - 先求对词v出现在标签k类文本中的frequency: train_bow.transpose().dot(label)，即(K,N)\*(N,V)      - 再求probability: frequency / np.sum(frequency,axis=0,keepdim=False)，除数是行归一化的frequency代表标签i出现词语的总数，所以结果矩阵中的ij代表**词i在j标签文本中出现的概率**，这是likelihood P(x|y)    - 但文本贝叶斯在$P(y|x) = \frac{P(x|y)P(y)}{P(x)}$即预测阶段的x是一系列词语，即整个doc，不是单个词语或特征，所以这里应该写为：      - $P(y|x) = \frac{\sum_{i\in doc} P(x_i|y)P(y)}{P(x)} = \frac{P(y,x_1,x_2,...,x_n)}{P(x)}$      - 更细致地说，这里的doc属于Vocabulary但同一个词可能出现多次。故应该写为：      - $P(y|x) = \frac{\sum_{v\in V} P(x_v|y)^{c(x_v)}P(y)}{P(x)}$，其中$c(x_v)$代表v单词在该doc中出现的频数，因为同一个单词可能多次是出现所以得乘方，且所有的词都可能出现所以得求和所有词语。        - 计算该式，使用log将变成${\sum_{v\in V}c(x_v)logP(x_v|y)}+logP(y)$，我们想到$c(x_v)$代表v单词在该doc中出现的频数，即BOW的定义，所以该式即BOW矩阵乘以logP(x|y)矩阵。      - 根据定义，这个结果就是对|N|个doc的预测类别，即结果是(|N|,|K|)      - 最后np.argmax(axis=1)，计算出类别即可。    - 简化计算      - 忽略p(x)      - 用log将乘转化为加<jupyter_code>import numpy as np
from scipy import sparse
from collections import Counter
from sklearn.metrics import accuracy_score
import nltk
from nltk.corpus import stopwords
import string
import pandas as pd<jupyter_output><empty_output><jupyter_text>Collect stop words.<jupyter_code>stopwords = set(stopwords.words("english"))<jupyter_output><empty_output><jupyter_text>Data loader Define function to load document id, raw text and labels from the input csv file.The input csv file (data/train.csv or data/test.csv) has the following 3 columns:1. id: document id2. text: document raw text3. label: document label (data/train.csv: one of the values in {1,2,3,4,5}; data/test.csv: -1)<jupyter_code>def load_data(file_name):
    """
    :param file_name: a file name, type: str
    return a list of ids, a list of documents, a list of labels
    """
    df = pd.read_csv(file_name)

    return df['id'], df["text"], df['label']<jupyter_output><empty_output><jupyter_text>Define function to load document labels from the input csv file.The input csv file (data/answer.csv) has the following 2 columns:1. id: document id2. label: document label (one of the values in {1,2,3,4,5})<jupyter_code>def load_labels(file_name):
    """
    :param file_name: a file name, type: str
    return a list of labels
    """
    return pd.read_csv(file_name)['label']<jupyter_output><empty_output><jupyter_text>Feature Extractor Define tokenization function.<jupyter_code>def tokenize(text):
    """
    :param text: a doc with multiple sentences, type: str
    return a word list, type: list
    e.g.
    Input: 'Text mining is to identify useful information.'
    Output: ['Text', 'mining', 'is', 'to', 'identify', 'useful', 'information', '.']
    """
    return nltk.word_tokenize(text)<jupyter_output><empty_output><jupyter_text>Define function for filtering stop words.<jupyter_code>def filter_stopwords(tokens):
    """
    :param tokens: a list of tokens, type: list
    return a list of filtered tokens, type: list
    e.g.
    Input: ['text', 'mine', 'is', 'to', 'identifi', 'use', 'inform', '.']
    Output: ['text', 'mine', 'identifi', 'use', 'inform', '.']
    """
    ### equivalent code
    # results = list()
    # for token in tokens:
    #     if token not in stopwords and not token.isnumeric():
    #         results.append(token)
    # return results

    return [token for token in tokens if token not in stopwords and not token.isnumeric()]<jupyter_output><empty_output><jupyter_text>Define function for building the Bag Of Word (BOW) representations of documents.  Documentation of scipy lil matrix: https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.lil_matrix.htmlDocumentation of scipy csr matrix: https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.csr_matrix.html<jupyter_code>def get_bagofwords(data, vocab_dict):
    '''
    :param data: a list of tokenized documents, type: list
    :param vocab_dict: a mapping from words to indices, type: dict
    return a BOW matrix in Compressed Sparse Row matrix format, type: scipy.sparse.csr_matrix
    '''
    
    '''
    The BOW matrix is first constructed using Row-based list of lists sparse matrix (LIL) format.
    LIL is a convenient format for constructing sparse matrices, as it supports flexible slicing, 
    and it is efficient to change to the matrix sparsity structure.
    '''
    
    data_matrix = sparse.lil_matrix((len(data), len(vocab_dict)))

    for i, doc in enumerate(data):
        for word in doc:
            word_idx = vocab_dict.get(word, -1)
            if word_idx != -1:
                data_matrix[i, word_idx] += 1
                
    '''
    After constructing the BOW matrix on all input documents, we convert the matrix to Compressed Sparse 
    Row (CSR) format for fast arithmetic and matrix vector operations.
    '''
    data_matrix = data_matrix.tocsr()
    
    return data_matrix<jupyter_output><empty_output><jupyter_text>Data pre-processing Load document ids, raw texts, and labels from the train and test sets.<jupyter_code>train_file = "data/train.csv"
test_file = "data/test.csv"
ans_file = "data/answer.csv"


train_ids, train_texts, train_labels = load_data(train_file)
test_ids, test_texts, _ = load_data(test_file)
test_labels = load_labels(ans_file)



print("Size of train set: {}".format(len(train_ids)))
print("Size of test set: {}".format(len(test_ids)))<jupyter_output>Size of train set: 2000
Size of test set: 400<jupyter_text>Tokenize the raw texts in the train and test sets.<jupyter_code>train_tokens = [tokenize(text) for text in train_texts] 
test_tokens = [tokenize(text) for text in test_texts]<jupyter_output><empty_output><jupyter_text>Remove stop words from the tokenized texts.<jupyter_code>train_tokens = [filter_stopwords(tokens) for tokens in train_tokens]
test_tokens = [filter_stopwords(tokens) for tokens in test_tokens]<jupyter_output><empty_output><jupyter_text>Build a vocabulary (i.e., a mapping from words to indices) on the train set.<jupyter_code># use a set data structure to hold all words appearing in the train set
vocab = set()

for i, doc in enumerate(train_tokens):# enumerate over each document in the train set
    # enumerate over each word in the document
    for word in doc:
        # if this word has been added into the set before, 
        # then it will be ignored, otherwise, it will be 
        # added into the set.
        vocab.add(word)
        
# create a dictionary from the set of words, where the
# keys are word strings and the values are numerical indices
vocab_dict = dict(zip(vocab, range(len(vocab))))
print('Size of vocab: ', len(vocab_dict))<jupyter_output>Size of vocab:  16120<jupyter_text>Build the BOW matrices from the tokenized texts in train and test sets respectively, using the vocabulary and the get_bagofwords function defined above<jupyter_code>train_data_matrix = get_bagofwords(train_tokens, vocab_dict)
test_data_matrix = get_bagofwords(test_tokens, vocab_dict)
print('Type of train_data_matrix: ', type(train_data_matrix))
print('Type of test_data_matrix: ', type(test_data_matrix))
print('Shape of train_data_matrix:', train_data_matrix.shape)
print('Shape of test_data_matrix:', test_data_matrix.shape)<jupyter_output>Type of train_data_matrix:  <class 'scipy.sparse.csr.csr_matrix'>
Type of test_data_matrix:  <class 'scipy.sparse.csr.csr_matrix'>
Shape of train_data_matrix: (2000, 16120)
Shape of test_data_matrix: (400, 16120)<jupyter_text>Naive Bayes Define the following symbols:N_train = size of the train setN_test = size of the test setV = vocabulary sizeK = number of classesAll indices of tensors are 0-based<jupyter_code># get the size of the train set 
N_train = train_data_matrix.shape[0]

# get the size of the test set 
N_test = test_data_matrix.shape[0]

# get the vocabulary size
V = len(vocab_dict)

# get the number of classes
K = max(train_labels)

print('N_train: ', N_train)
print('N_test: ', N_test)
print('V: ', V)
print('K: ', K)<jupyter_output>N_train:  2000
N_test:  400
V:  16120
K:  5<jupyter_text>Define a utility function to normalize (with/without laplace smoothing) an input tensor over the first dimension.<jupyter_code>def normalize(P, smoothing_prior=0):
    """
    e.g.
    Input: [1,2,1,2,4]
    Output: [0.1,0.2,0.1,0.2,0.4] (without laplace smoothing) or 
    [0.1333,0.2,0.1333,0.2,0.3333] (with laplace smoothing and the smoothing prior is 1)
    """
    
    # get the size of the first dimension
    N = P.shape[0]
    
    # sum the tensor over the first dimension
    # setting axis = 0 means the summation is performed over the first dimension
    # setting keepdims=True means the reduced axes (i.e., the 0-th axis this case) 
    # are left in the result as dimensions with size one. With this option, the 
    # result will broadcast correctly against the input array.
    
    norm = np.sum(P, axis=0, keepdims=True)
    
    # perform the normalization by dividing the input tensor by the norm,
    # and add smoothing prior in both the numerator and the denominator.
    return (P + smoothing_prior) / (norm + smoothing_prior*N)<jupyter_output><empty_output><jupyter_text>Define a utility function to compute the accuracy score given the ground truth labels and predictions.<jupyter_code>def evaluate(y_true, y_pre):
    acc = accuracy_score(y_true, y_pre)
    return acc<jupyter_output><empty_output><jupyter_text>Training Given:1. the training labels (1-d array of shape (N_train,));2. the BOW matrix of training documents (scipy.sparse.csr_matrix of shape (N_train,V)),the training of Naive Bayes classifier is to compute the following two probabilities:1. prior: P(y) (an 1-d array with shape (K,), where the entry at position [l] is the is the prior probability of label l+1);2. likelihood:  P(x|y) (a matrix with shape (V,K), where the entry at position [i,l] is the probability of word i in the documents of label l+1)<jupyter_code># create a matrix with shape (N_train,K), where the entry at
# the position (i,j) is 1  
# iff the (i+1)-th document belongs to (j+1)-th 
# class, otherwise it is 0

data_label_onehot_matrix = np.zeros((N_train, K))

for i, l in enumerate(train_labels):
    # the (i+1)-th document has label l, so we 
    # set the entry at the position [i,l-1] to 
    # be 1
    data_label_onehot_matrix[i, l-1] = 1
    
    
print('data_label_onehot_matrix.shape: ', data_label_onehot_matrix.shape)<jupyter_output>data_label_onehot_matrix.shape:  (2000, 5)<jupyter_text>Check the the labels of the first three documents in the train set and the first three rows of data_label_onehot_matrix.<jupyter_code>print('The labels of the first three documents')
print(train_labels.values[:3])
print()
print('The first three rows of data_label_onehot_matrix')
print(data_label_onehot_matrix[:3])<jupyter_output>The labels of the first three documents
[4 2 3]

The first three rows of data_label_onehot_matrix
[[0. 0. 0. 1. 0.]
 [0. 1. 0. 0. 0.]
 [0. 0. 1. 0. 0.]]<jupyter_text>Compute the frequencies of all labels in the train set by row-wise summation.Set axis = 0 so that the summation is across rows of the data_label_onehot_matrix.Set keepdims = False so that we can get an 1-d array of shape (K,) after the summation.<jupyter_code>label_freq = np.sum(data_label_onehot_matrix, axis=0, keepdims=False)
print(label_freq.shape)<jupyter_output>(5,)<jupyter_text>Check the frequencies of all labels in the train set.<jupyter_code>print('Label\tFrequency')
for l, f in enumerate(label_freq):
    print('{}\t{}'.format(l+1,f))<jupyter_output>Label	Frequency
1	173.0
2	188.0
3	243.0
4	593.0
5	803.0<jupyter_text>Compute P(y) by normalizing the label frequencies with laplace smoothing, where the smoothing prior = 1.<jupyter_code>P_y = normalize(label_freq, smoothing_prior=1)
print('P_y.shape: ', P_y.shape)
print('Label\tPrior probability')
for l, p in enumerate(P_y):
    print('{}\t{}'.format(l+1,p))<jupyter_output>Label	Prior probability
1	0.08678304239401496
2	0.0942643391521197
3	0.12169576059850375
4	0.29625935162094763
5	0.400997506234414<jupyter_text>Build a matrix word_freq of shape (V,K), where word_freq[i,j] is the frequency of word i in the documents of label (j+1).<jupyter_code>print('train_data_matrix.shape: ', train_data_matrix.shape)#(N_train,V)
print('train_data_matrix.transpose().shape: ', train_data_matrix.transpose().shape)#(V,N_train)
print('data_label_onehot_matrix.shape: ', data_label_onehot_matrix.shape)#(N_train,K)
word_freq = train_data_matrix.transpose().dot(data_label_onehot_matrix)
print(word_freq.shape)<jupyter_output>(16120, 5)<jupyter_text>word_freq[i,j] = the dot product of the following 2 vectors:1. the i-th row of train_data_matrix.transpose(): 2. the j-th column of data_label_onehot_matrix The i-th row of train_data_matrix.transpose() is the frequncies of word i in all documents in the train set (i.e., train_data_matrix.transpose()[i,k] is the frequency of word i in (k+1)-th document). The j-th column of data_label_onehot_matrix is a vector indicating whether each document in the train set has label (j+1) (i.e., data_label_onehot_matrix[k,j] = 1 if the (k+1)-th document has label (j+1), otherwise it is data_label_onehot_matrix[k,j] = 0) So the dot product of these two vectors is to sum over the frequencies of word i in all the train documents of label (j+1), which is the frequency of word i in the documents of label (j+1). Normalize the word_freq matrix over the rows (i.e., across all words in the vocabulary for each label) to get P(x|y) (a matrix with shape (V,K), where the entry at position [i,l] is the probability of word i in the documents of label l+1). The normalization is with laplace smoothing, where the smoothing prior = 1.<jupyter_code>P_xy = normalize(word_freq,smoothing_prior=1)
print('P_xy.shape', P_xy.shape)<jupyter_output>P_xy.shape (16120, 5)<jupyter_text>Check the probabilities of the first three word in the vocabulary in documents of every label.<jupyter_code>print('P_xy[:3, :]: ')
P_xy[:3, :]<jupyter_output>P_xy[:3, :]:<jupyter_text>Prediction Given:1. a BOW matrix : scipy.sparse.csr_matrix of shape (N,V) (N = N_train or N = N_test);2. prior: P(y);3. likelihood:  P(x|y),the prediction of Naive Bayes classifier is to compute the following array:1. pred: an 1-d array of shape (N,), where the entry at position i is the predicted label for the i-th document in the given BOW matrix. We first compute the joint probability $P(Y=y_i, \mathbf{D}) = P(Y=y_i) \prod_{j}^{V}{P(x_j|Y=y_i)^{c(x_j)}}$, where $\mathbf{D}$ is the data,  using the following equation:$P(Y=y_i) \prod_{j}^{V}{P(x_j|Y=y_i)^{c(x_j)}} = \exp(\log(P(Y=y_i)) + \sum_{j}^{V}{c(x_j)\log(P(x_j|Y=y_i))})$,where $c(x_j)$ is the total count of word $x_j$ in $\mathbf{D}$. Since both the exponential and logarithmic function are monotonically increasing, we use them to convert the multiplication into summation in order to speed up computation. 1. Compute $\log(P(Y=y_i))$, to enable the later opeartions with matrix, we use np.expand_dims function to insert a new dimension with size 1 into the first dimension of the array np.log(P_y)<jupyter_code>log_P_y = np.expand_dims(np.log(P_y), axis=0)
print("log_P_y.shape: ",log_P_y.shape)<jupyter_output>log_P_y.shape:  (1, 5)<jupyter_text>2. Compute $\log(P(x_j|Y=y_i))$<jupyter_code>log_P_xy = np.log(P_xy)
print("log_P_xy.shape: ",log_P_xy.shape)<jupyter_output>log_P_xy.shape:  (16120, 5)<jupyter_text>3. Compute $\sum_{j}^{V}{c(x_j)\log(P(x_j|Y=y_i))}$ using the dot product between the BOW matrix and log_P_xy.<jupyter_code>train_log_P_dy = train_data_matrix.dot(log_P_xy)
test_log_P_dy = test_data_matrix.dot(log_P_xy)
print("train_log_P_dy.shape: ", train_log_P_dy.shape)
print("test_log_P_dy.shape: ", test_log_P_dy.shape)<jupyter_output>train_log_P_dy.shape:  (2000, 5)
test_log_P_dy.shape:  (400, 5)<jupyter_text>4. Compute $\log(P(Y=y_i)) + \sum_{j}^{V}{c(x_j)\log(P(x_j|Y=y_i))}$<jupyter_code>train_log_P = log_P_y + train_log_P_dy
test_log_P = log_P_y + test_log_P_dy
print("train_log_P.shape: ", train_log_P.shape)
print("test_log_P.shape: ", test_log_P.shape)<jupyter_output>train_log_P.shape:  (2000, 5)
test_log_P.shape:  (400, 5)<jupyter_text>Since $P(Y=y_i |\mathbf{D}) \propto P(Y=y_i, \mathbf{D})$, we directly use the log joint probabilities computed above to get labels for every document by choosing the maximum probability.<jupyter_code># we add 1 because labels strat from 1
train_pred = np.argmax(train_log_P, axis=1) + 1
test_pred = np.argmax(test_log_P, axis=1) + 1
print("train_pred.shape: ", train_pred.shape)
print("test_pred.shape: ", test_pred.shape)<jupyter_output>train_pred.shape:  (2000,)
test_pred.shape:  (400,)<jupyter_text>Evaluation<jupyter_code>train_acc= evaluate(train_labels, train_pred)
print("Train Accuracy: {}".format(train_acc))

test_acc= evaluate(test_labels, test_pred)
print("Test Accuracy: {}".format(test_acc))<jupyter_output>Train Accuracy: 0.8565
Test Accuracy: 0.5225
{'fraction_non_alphanumeric': 0.11707482608449749, 'fraction_numerical': 0.0209829760760138, 'mean_word_length': 4.993898305084746, 'pattern_counts': {'":': 0, '<': 101, '<?xml version=': 0, '>': 101, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '29', 'dir': 'jupyter-structured-clean-dedup', 'id': '345195', 'license': 'MIT', 'n_tokens_mistral': 6951, 'n_tokens_neox': 6769, 'n_words': 1737, 'path': 'nlp_tasks/text_classification/naiveBayes.ipynb', 'repo_name': 'Jackthebighead/nlp-uplearning', 'size': '17681'}