<jupyter_start><jupyter_text>Generative modelsMost **classification algorithms** fall into one of two categories:  - discriminative classifiers - generative classifiers **Discriminative classifiers** model the target variable, y, as a direct function of the predictor variables, x. Example: logistic regression uses the following model, where 𝜷 is a length-D vector of coefficients and x is a length-D vector of predictors:![image.png](attachment:d33bfaca-3d8b-4d7d-a376-6c4b67f3a7ab.png)**Generative classifiers** instead view the predictors as being generated according to their class — i.e., they see x as a function of y, rather than the other way around. They then use Bayes’ rule to get from p(x|y = k) to P(y = k|x), as explained below.Generative models can be broken down into the three following steps. Suppose we have a classification task with K unordered classes, represented by k = 1, 2, …, K.   - Estimate the prior probability that a target belongs to any given class. I.e., estimate P(y = k) for k = 1, 2, …, K.   - Estimate the density of the predictors conditional on the target belonging to each class. I.e., estimate p(x|y = k) for k = 1, 2, …, K.   - Calculate the posterior probability that the target belongs to any given class. I.e., calculate P(y = k|x), which is proportional to p(x|y = k)P(y = k) by Bayes’ rule.We then classify an observation as belonging to the class k for which the following expression is greatest:![image.png](attachment:3c132f2d-6beb-44d0-a857-3afe1e47557d.png)  Class Priors estimation for Generative classifiersLet I_nk be an indicator which equals 1 if y_n = k and 0 otherwise.![image.png](attachment:cf42fce5-b54e-4d7e-b914-9e63a9c78441.png)Our estimate of P(y = k) is just the sample fraction of the observations from class k.![image.png](attachment:07f2b086-a07e-44b4-a6e1-74a5059e16bc.png)  Data LikelihoodThe next step is to model the conditional distribution of x given y so that we can estimate this distribution’s parameters. This of course depends on the family of distributions we choose to model x. Three common approaches are detailed below.sns.<jupyter_code>import numpy as np
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_iris

data = load_iris()

X, y = data.data, data.target
X_train, X_test, Y_train, Y_test = train_test_split(X, y, test_size=0.1)<jupyter_output><empty_output><jupyter_text>Linear Discriminative Analysis (LDA)In LDA, we assume the following distribution for x![image.png](attachment:38c14b1b-ec13-4bd9-bc1b-bd0293fb9b80.png)for k = 1, 2, …, K. Note that each class has the same covariance matrix but a unique mean vector.<jupyter_code>from sklearn.discriminant_analysis import LinearDiscriminantAnalysis

clf = LinearDiscriminantAnalysis()
clf.fit(X_train, Y_train)

y_pred = clf.predict(X_train)
print("Train Accuracy ", np.mean(y_pred == Y_train))

colors = {0: 'r',1: 'g',2: 'b'}

fig, ax = plt.subplots(1, 2)
for x, y in zip(X_train, y_pred):
    ax[0].scatter(x[0],x[1],c=colors[y])
ax[0].set_xlabel("predicted")

for x, y in zip(X_train, Y_train):
    ax[1].scatter(x[0],x[1],c=colors[y])
ax[1].set_xlabel("real")

plt.show()<jupyter_output>Train Accuracy  0.9777777777777777<jupyter_text>Quadratic Discriminant Analysis (QDA)QDA looks very similar to LDA but assumes each class has its own covariance matrix. I.e.,![image.png](attachment:d3571b5a-d682-4b71-a8d3-0d805d81daed.png)<jupyter_code>from sklearn.discriminant_analysis import QuadraticDiscriminantAnalysis

clf = QuadraticDiscriminantAnalysis()
clf.fit(X_train, Y_train)

y_pred = clf.predict(X_train)
print("Train Accuracy ", np.mean(y_pred == Y_train))

colors = {0: 'r',1: 'g',2: 'b'}

fig, ax = plt.subplots(1, 2)
for x, y in zip(X_train, y_pred):
    ax[0].scatter(x[0],x[1],c=colors[y])
ax[0].set_xlabel("predicted")

for x, y in zip(X_train, Y_train):
    ax[1].scatter(x[0],x[1],c=colors[y])
ax[1].set_xlabel("real")

plt.show()<jupyter_output>Train Accuracy  0.9851851851851852<jupyter_text>Naive BayesNaive Bayes assumes the random variables within x are independent conditional on the class of the observation. That is, if x is D-dimensional,![image.png](attachment:54c8fd9e-253c-405d-887c-d563f2812526.png)<jupyter_code>from sklearn.naive_bayes import GaussianNB

clf = GaussianNB()
clf.fit(X_train, Y_train)

y_pred = clf.predict(X_train)
print("Train Accuracy ", np.mean(y_pred == Y_train))

colors = {0: 'r',1: 'g',2: 'b'}

fig, ax = plt.subplots(1, 2)
for x, y in zip(X_train, y_pred):
    ax[0].scatter(x[0],x[1],c=colors[y])
ax[0].set_xlabel("predicted")

for x, y in zip(X_train, Y_train):
    ax[1].scatter(x[0],x[1],c=colors[y])
ax[1].set_xlabel("real")

plt.show()<jupyter_output>Train Accuracy  0.9555555555555556
{'fraction_non_alphanumeric': 0.11673315137518371, 'fraction_numerical': 0.049338652110014695, 'mean_word_length': 5.56198347107438, 'pattern_counts': {'":': 0, '<': 14, '<?xml version=': 0, '>': 14, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '510093', 'license': 'MIT', 'n_tokens_mistral': 1851, 'n_tokens_neox': 1697, 'n_words': 511, 'path': 'notebooks/02-GenerativeModels.ipynb', 'repo_name': 'bcebere/statistics_python', 'size': '4763'}