<commit_before>---
layout:     post
title:      "Naive Bayes and Logistics Regression"
date:       2018-06-17 12:00:00
author:     "GwanSiu"
catalog: true
tags:
    - Statistics and Bayesian Analysis
---

## 1. Naive Bayes Algorithm

Naive bayes algorithm is supervised learning algorithm, it has two formulations for different inputs, such as **discrete-valued inputs** and **continuous-valued inputs.** MAP is usually used in parameters estimation of naive bayes. 

In statistics machine learning, our goal is to inference class for new sample, and the posterior distribution is proportional to the likelihood multiplied by prior. $P(Y\arrowvert X_{new}, X_{train})=\frac{P(X_{new}\arrowvert X_{train} Y)P(Y)}{P(X)}$. In Naives Bayes, the assumption is conditional independence: 

$$
\begin{equation}
(\forall i,j,k)P(X=x_{i}\arrowvert Y=y_{j}, Z=z_{k})=P(X=x_{i}\arrowvert Z=z_{k})
\end{equation}
$$

Now, let we see more detail about naives bayes classifier. 

In general, we assume that Y is any discrete-valued variable, and attributes $X_{1},...,X_{n}$ are any discrete or realvalued attributes. Our goal is to train a classifier that will output the probability
distribution over possible values of Y, for each new instance X that we ask it to classify. 

The expression for the probability that Y will take on its kth possible
value, according to Bayes rule, is：

$$
\begin{equation}
P(Y=y_{k}\arrowvert X_{1},...,X_{n})=\frac{P(X_{1},...,X_{n}\arrowvert Y=y_{k})}{\sum_{j}P(Y=y_{j})P(X_{1},...,X_{n}\arrowvert Y=y_{j})}
\end{equation}
$$

where the sum is taken over all possible values y j of Y. Now, we hold the assumption of conditional independence. The formulation can be rewrite as:

$$
\begin{equation}
P(Y=y_{k}\arrowvert X_{1},...,X_{n})=\frac{\prod_{i}^{n}P(X_{i}\arrowvert Y=y_{k})P(Y=y_{k})}{\sum_{j}P(Y=y_{j})\prod_{i}^{n}P(X_{i}\arrowvert Y=y_{j})}
\end{equation}
$$

Equation(3) is the fundamental formulation of naive bayes algorithm for naives bayes algorithm.  Given a
new instance $X_{new} = X_{1},...,X_{n}$, this equation shows how to calculate the probability that $Y$ will take on any given value, given the observed attribute values of $X$ new and given the distributions $P(Y)$ and $P(X_{i}\arrowvert Y)$ estimated from the training data. If we are interested only in the most probable value of Y, then we have the Naive Bayes classification rule(MAP):

$$\begin{equation}
Y\leftarrow \arg\max_{y_{k}} \frac{\prod_{i}^{n}P(X_{i}\arrowvert Y=y_{k})}{\sum_{j}P(Y=y_{j})\prod_{i}^{n}P(X_{i}\arrowvert Y=y_{j})}
\end{equation}
$$

usually, we consider denominator as constant, and equation above can be simplified as:

$$\begin{equation}
Y\leftarrow \arg\max_{y_{k}} P(Y=y_{k})\prod_{i}^{n}P(X_{i}\arrowvert Y=y_{k})
\end{equation}
$$

### 1.1 Naive Bayes for Discrete-Valued Inputs

Assume the n input attributes $X_{i}$ each take on $J$ possible discrete values, and $Y$ is a discrete variable taking on $K$ possible discrete values, then our learning task is to estimage two sets of parameters. The first is:

$$
\begin{equation}
\theta_{ijk} \equiv P(X_{i}=x_{ij}\arrowvert Y=y_{k})
\end{equation}
$$

for each input attribute $X_{i}$, each of its possible values $x_{ij}$, and each of the possible values $y_{k}$ of $Y$. Therefore, there will be nJK such parameters, and note only $n(J-1)K$ of these are independent, given that they must satisfy $\sum_{j}\theta_{ijk}=1$ for each pair of $i,k$ values.

The second parameter set is prior probability over $Y$:

$$
\begin{equation}
\pi_{k} \equiv P(Y=y_{k})
\end{equation}
$$
Note there are $K$ parameters, $(K-1)$ of which are independent.

**Estimation of** $\theta$

Givening a set of training examples D, the MLE for $\theta_{ijk}$ are given by:

$$
\begin{equation}
\tilde{\theta}_{ijk} = \tilde{P}(X_{i}=x_{ij}\arrowvert Y=y_{k}) = \frac{#D{X_{i}=x_{ij}\wedge Y=y_{k}}}{#D{Y=y_{k}}}
\end{equation}
$$

where the $#D{x}$ operator returns the number of elements in the set $D$ that satisfy property $x$.<commit_msg>Naive Bayes<commit_after>---
layout:     post
title:      "Naive Bayes and Logistics Regression"
date:       2018-06-17 12:00:00
author:     "GwanSiu"
catalog: true
tags:
    - Statistics and Bayesian Analysis
---

## 1. Naive Bayes Algorithm

Naive bayes algorithm is supervised learning algorithm, it has two formulations for different inputs, such as **discrete-valued inputs** and **continuous-valued inputs.** MAP is usually used in parameters estimation of naive bayes. 

In statistics machine learning, our goal is to inference class for new sample, and the posterior distribution is proportional to the likelihood multiplied by prior. $P(Y\arrowvert X_{new}, X_{train})=\frac{P(X_{new}\arrowvert X_{train} Y)P(Y)}{P(X)}$. In Naives Bayes, the assumption is conditional independence: 

$$
\begin{equation}
(\forall i,j,k)P(X=x_{i}\arrowvert Y=y_{j}, Z=z_{k})=P(X=x_{i}\arrowvert Z=z_{k})
\end{equation}
$$

Now, let we see more detail about naives bayes classifier. 

In general, we assume that Y is any discrete-valued variable, and attributes $X_{1},...,X_{n}$ are any discrete or realvalued attributes. Our goal is to train a classifier that will output the probability
distribution over possible values of Y, for each new instance X that we ask it to classify. 

The expression for the probability that Y will take on its kth possible
value, according to Bayes rule, is：

$$
\begin{equation}
P(Y=y_{k}\arrowvert X_{1},...,X_{n})=\frac{P(X_{1},...,X_{n}\arrowvert Y=y_{k})}{\sum_{j}P(Y=y_{j})P(X_{1},...,X_{n}\arrowvert Y=y_{j})}
\end{equation}
$$

where the sum is taken over all possible values y j of Y. Now, we hold the assumption of conditional independence. The formulation can be rewrite as:

$$
\begin{equation}
P(Y=y_{k}\arrowvert X_{1},...,X_{n})=\frac{\prod_{i}^{n}P(X_{i}\arrowvert Y=y_{k})P(Y=y_{k})}{\sum_{j}P(Y=y_{j})\prod_{i}^{n}P(X_{i}\arrowvert Y=y_{j})}
\end{equation}
$$

Equation(3) is the fundamental formulation of naive bayes algorithm for naives bayes algorithm.  Given a
new instance $X_{new} = X_{1},...,X_{n}$, this equation shows how to calculate the probability that $Y$ will take on any given value, given the observed attribute values of $X$ new and given the distributions $P(Y)$ and $P(X_{i}\arrowvert Y)$ estimated from the training data. If we are interested only in the most probable value of Y, then we have the Naive Bayes classification rule(MAP):

$$\begin{equation}
Y\leftarrow \arg\max_{y_{k}} \frac{\prod_{i}^{n}P(X_{i}\arrowvert Y=y_{k})}{\sum_{j}P(Y=y_{j})\prod_{i}^{n}P(X_{i}\arrowvert Y=y_{j})}
\end{equation}
$$

usually, we consider denominator as constant, and equation above can be simplified as:

$$\begin{equation}
Y\leftarrow \arg\max_{y_{k}} P(Y=y_{k})\prod_{i}^{n}P(X_{i}\arrowvert Y=y_{k})
\end{equation}
$$

### 1.1 Naive Bayes for Discrete-Valued Inputs

Assume the n input attributes $X_{i}$ each take on $J$ possible discrete values, and $Y$ is a discrete variable taking on $K$ possible discrete values, then our learning task is to estimage two sets of parameters. The first is:

$$
\begin{equation}
\theta_{ijk} \equiv P(X_{i}=x_{ij}\arrowvert Y=y_{k})
\end{equation}
$$

for each input attribute $X_{i}$, each of its possible values $x_{ij}$, and each of the possible values $y_{k}$ of $Y$. Therefore, there will be nJK such parameters, and note only $n(J-1)K$ of these are independent, given that they must satisfy $\sum_{j}\theta_{ijk}=1$ for each pair of $i,k$ values.

The second parameter set is prior probability over $Y$:

$$
\begin{equation}
\pi_{k} \equiv P(Y=y_{k})
\end{equation}
$$
Note there are $K$ parameters, $(K-1)$ of which are independent.

**Estimation of** $\theta$

Givening a set of training examples D, the MLE for $\theta_{ijk}$ are given by:

$$
\begin{equation}
\tilde{\theta}_{ijk} = \tilde{P}(X_{i}=x_{ij}\arrowvert Y=y_{k}) = \frac{\#D{X_{i}=x_{ij}\wedge Y=y_{k}}}{#D{Y=y_{k}}}
\end{equation}
$$

where the $\#D{x}$ operator returns the number of elements in the set $D$ that satisfy property $x$.
{'fraction_non_alphanumeric': 0.15188608543592305, 'fraction_numerical': 0.006744941294029478, 'mean_word_length': 4.966467958271237, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4665068', 'n_tokens_mistral': 2718, 'n_tokens_neox': 2607, 'n_words': 964}