<jupyter_start><jupyter_text>Naive Bayes$$ \begin{split} \mathop{argmax}_{c_k}p(y=c_k|x) &= \mathop{argmax}_{c_k}p(y=c_k)p(x|y=c_k) \\& \left( due to: p(y=c_k|x) = \frac{p(y=c_k)p(x|y=c_k)}{p(x)} \right) \\&= \mathop{argmax}_{c_k}p(y=c_k)\prod_jp(x^{(j)}|y=c_k) \end{split} $$Use Maximum Likelihood Estimate(MLE) to evaluate $ p(y=c_k)$ and $ p(x^{(j)}|y=c_k) $ in datasets.$$ \hat{p}(y=c_k) = \frac{\sum_i I(y_i=c_k)}{N} \\\hat{p}(x^{(j)}=a_j|y=c_k) = \frac{\sum_i I(x_i^{(j)}=a_j,y=c_k)}{I(y_i=c_k)}$$Bayesian estimation add $ \lambda $ on numerator and denominator in MLE.  Naive Bayes in Scikit-learnClassifiers: GaussianNB, MultinomialNB, BernoulliNB Documents ClassificationUse TF-IDF(Term Frequency and Inverse Document Frequency) of term in documents as feature$$ TF-IDF = TF*IDF \\TF(t) = \frac {\text{Number of times term t appears in a document}}{\text{Total number of terms in the document}}\\IDF(t) = log_e\frac {\text{Total number of documents}}{\text{Number of documents with term t in it + 1}} $$Bag of Words  TfidfVectorizersklearn.feature_extraction.text.TfidfVectorizer(stop_words, token_pattern, max_df)<jupyter_code>from sklearn.feature_extraction.text import TfidfVectorizer
vect = TfidfVectorizer()
documents=[
    'my dog has flea problems help please',
    'maybe not take him to dog park stupid',
    'my dalmation is so cute I love him',
    'stop posting stupid worthless garbage',
    'mr licks ate my steak how to stop him',
    'quit buying worthlsess dog food stupid',
]
targets=[0,1,0,1,0,1] # 0 normal, 1 insult
tf_matrix = vect.fit_transform(documents)
# all unique words
words = vect.get_feature_names()
print(len(words), words)
# words id
print(len(vect.vocabulary_), vect.vocabulary_)
tfidf = tf_matrix.toarray()
print(tfidf.shape, tfidf[0])<jupyter_output>(6, 32) [0.         0.         0.         0.         0.2836157  0.40966432
 0.         0.         0.40966432 0.40966432 0.         0.
 0.         0.         0.         0.         0.         0.2836157
 0.         0.         0.40966432 0.         0.40966432 0.
 0.         0.         0.         0.         0.         0.
 0.         0.        ]<jupyter_text>CountVectorizer<jupyter_code>from sklearn.feature_extraction.text import CountVectorizer
c_vect = CountVectorizer()
c_matrix = c_vect.fit_transform(documents)
print(c_vect.get_feature_names())
c_matrix.toarray()
# default ngram_range is (1, 1), token_pattern=’(?u)\b\w\w+\b’
c_vect_ngram = CountVectorizer(ngram_range=(1, 2))
c_matrix_ngram = c_vect_ngram.fit_transform(documents)
print(c_vect_ngram.get_feature_names())<jupyter_output>['ate', 'ate my', 'buying', 'buying worthlsess', 'cute', 'cute love', 'dalmation', 'dalmation is', 'dog', 'dog food', 'dog has', 'dog park', 'flea', 'flea problems', 'food', 'food stupid', 'garbage', 'has', 'has flea', 'help', 'help please', 'him', 'him to', 'how', 'how to', 'is', 'is so', 'licks', 'licks ate', 'love', 'love him', 'maybe', 'maybe not', 'mr', 'mr licks', 'my', 'my dalmation', 'my dog', 'my steak', 'not', 'not take', 'park', 'park stupid', 'please', 'posting', 'posting stupid', 'problems', 'problems help', 'quit', 'quit buying', 'so', 'so cute', 'steak', 'steak how', 'stop', 'stop him', 'stop posting', 'stupid', 'stupid worthless', 'take', 'take him', 'to', 'to dog', 'to stop', 'worthless', 'worthless garbage', 'worthlsess', 'worthlsess dog']<jupyter_text>MultinomialNB<jupyter_code>from sklearn.naive_bayes import MultinomialNB
clf = MultinomialNB(alpha=0.001).fit(tf_matrix, targets)
test_vect = TfidfVectorizer(vocabulary=vect.vocabulary_)
test_features = test_vect.fit_transform([documents[3]])
predicted_labels = clf.predict(test_features)
from sklearn import metrics
print(metrics.accuracy_score([targets[3]], predicted_labels))<jupyter_output>1.0
{'fraction_non_alphanumeric': 0.16764080765143463, 'fraction_numerical': 0.02948990435706695, 'mean_word_length': 4.512445095168375, 'pattern_counts': {'":': 0, '<': 10, '<?xml version=': 0, '>': 10, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '5029', 'license': 'MIT', 'n_tokens_mistral': 1515, 'n_tokens_neox': 1424, 'n_words': 345, 'path': 'ML/NaiveBayes.ipynb', 'repo_name': 'Hacky-DH/learn', 'size': '3764'}