<jupyter_start><jupyter_text>Naive BayesThe following is an explanation of Naive Bayes (8.7) in Ertel's Artificial Intelligence.  Naive Bayes classThe code is adapted from [ML from Scratch Naive Bayes](https://github.com/python-engineer/MLfromscratch/blob/master/mlfromscratch/naivebayes.py) with the accompanying [video](https://www.youtube.com/watch?v=BqUmKsfSWho).** Note that I fixed minor bugs in the calculation of the pdf below.$$ P(y|X) = \large \frac{P(X|y) \cdot P(y)}{P(X)} $$- P(y|X): the posterior- P(X|y): likelihood - class conditional probability- P(X): evidence- P(y): prior probabilityYou can calculate the posterior conditional probabilities for all classes and the one with the highest probability is the predicted class. In other words,$$ y = argmax_yP(y|X) $$But given that each class variable is independent of each other (which is why it is called the "naive Bayes"), it is turned into:$$ = argmax_y\frac{P(x_1|y)P(x2|y)...P(x_n|y) \cdot P(y)}{P(X)} $$The formula can be simplied further by dropping P(X) because it is a constant factor for all probabilities. $$ = argmax_y P(x_1|y)P(x2|y)...P(x_n|y) \cdot P(y) $$The probabilities are all between zero and one so multiplying fractions will make the values approach to zero or in some cases becomes a zero probability. To avoid this problem, we want to use the log probabilities instead:$$ = argmax_y log(P(x_1|y)) + log(P(x2|y))...log(P(x_n|y)) + log(P(y)) $$So this is what we are going to use to code. Prior probabilities$ P(y) $: frequency of the class sample in the data Class conditional probabilityFor class conditional probabilities, let's use the Gaussian distribution (i.e., the normal distribution):$$ P(x_i|y) = \frac{1}{\sqrt {2\pi\sigma_y^2}} \cdot exp(-\frac{(x_i - \mu_i)^2}{2\sigma_y^2}) $$<jupyter_code>import numpy as np
from scipy.stats import norm

class NaiveBayes:

    def fit(self, X, y):
        n_samples, n_features = X.shape
        self._classes = np.unique(y)
        n_classes = len(self._classes)

        # calculate mean, var, and prior for each class
        self._mean = np.zeros((n_classes, n_features), dtype=np.float64)
        self._var = np.zeros((n_classes, n_features), dtype=np.float64)
        self._priors =  np.zeros(n_classes, dtype=np.float64)

        for idx, c in enumerate(self._classes):
          X_c = X[y==c]
          self._mean[idx, :] = X_c.mean(axis=0)
          self._var[idx, :] = X_c.var(axis=0)
          self._priors[idx] = float(X_c.shape[0]) / float(n_samples)
          print("priors", idx, c, self._priors[idx])

    def predict(self, X):
        y_pred = [self._predict(x) for x in X]
        return np.array(y_pred)

    def _predict(self, x):
        posteriors = []

        # calculate posterior probability for each class
        for idx, c in enumerate(self._classes):
            prior = np.log(self._priors[idx])
            posterior = np.sum(np.log(self._pdf(idx, x)))
            posterior = prior + posterior
            posteriors.append(posterior)
            
        # return class with highest posterior probability
        return self._classes[np.argmax(posteriors)]
            

    # probability density function
    # NB; the original code had minor bugs which I fixed below.
    #   the original code was
    #     numerator = np.exp(- (x-mean)**2 / (2 * var))
    #     denominator = np.sqrt(2 * np.pi * var)
    #
    #   Note that var was not squared.
    def _pdf(self, class_idx, x):
        mean = self._mean[class_idx]
        var = self._var[class_idx]

        # This is just an implementation of the normal density function.
        # The results are identical with the standard library function
        #
        numerator = np.exp(- (x-mean)**2 / (2 * var**2))
        denominator = np.sqrt(2 * np.pi * var**2)
        class_cps = numerator / denominator
        #
        # class_cps = norm.pdf(x, loc=mean, scale=var)

        # print("class_conditional probabilities", class_cps)
        # print("  norm.pdf", norm.pdf(x, loc=mean, scale=var))

        return class_cps<jupyter_output><empty_output><jupyter_text>![image.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAJwAAAAkCAYAAAB8IVNVAAAFZ0lEQVR4Ae2aPc7UPBSF7xKgpkGsgJ4CxAY+lkBJBxUFDQg6qm8FgMQCYAewA9gB7AB2AHpeckYnlvMmTpyZZMZXsuz45/r+HN/rzCSi0ZgF7kbEy4j4HhEPxiZf4PjtiHgcEY8i4sMF6l9dZQwKfW2A6yzRrwAahxICcGp3Xa2aa4E9Ao7DcmOuwgPr4JfjSR82alTJAnsDHAAg1a1BRDUHHe1n3Ubev8beF8Nzb4ATANZwEKB61TEmhX7q7rnvEyCusffF8Nwb4AQIHKSLPX0ll3vAxHwofSEQf3jzMqXSTW/VUgvsDXAOEL34fE6MAEgATlo0jf6htc5f81td0QJ7A5wiECYAWKRBAU5Ra8w86AwJmP4G6vy7aa2qaYG9Aw6QARzqqRd76cw6XkAU7bDrmnfEnt9K3nxQrGR+<KEY>FTkSuQmCC)  TestUsing the NaiveClass, we can run against the skiing example in 8.4. We can see that it correctly classified all examples except one.<jupyter_code>import matplotlib.pyplot as plt

X = np.array([
      [6., 1.],
      [7., 3.],
      [8., 2.],    
      [9., 0.],
      [8., 4.],
      [8., 6.],
      [9., 2.],    
      [9., 5.]
    ])
y = np.array([0, 0, 0, 0, 1, 1, 1, 1])
X.shape
model_nb = NaiveBayes()
model_nb.fit(X, y)
predictions = model_nb.predict(X)
predictions
def accuracy(y_true, y_pred):
    accuracy = np.sum(y_true == y_pred) / len(y_true)
    return accuracy

accuracy(y, predictions)<jupyter_output><empty_output><jupyter_text>Naive Bayes for XOR?Would the naive Bayes model be able to solve the XOR problem?<jupyter_code>X_xor = np.array([[1., 1.],
                  [1., 0.],
                  [0., 1.],
                  [0., 0.]])
y_xor = np.array([0., 1., 1., 0.])

model_xor = NaiveBayes()
model_xor.fit(X_xor, y_xor)
predictions = model_xor.predict(X_xor)
predictions<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12183460244377546, 'fraction_numerical': 0.02620860633964937, 'mean_word_length': 3.3613899613899614, 'pattern_counts': {'":': 0, '<': 14, '<?xml version=': 0, '>': 14, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '485572', 'license': 'MIT', 'n_tokens_mistral': 2170, 'n_tokens_neox': 2065, 'n_words': 513, 'path': '08.7.naive_bayes.ipynb', 'repo_name': 'changsin/AI', 'size': '7081'}