<jupyter_start><jupyter_text>Fisher Discriminant Analysis First off we import necessary libraries.<jupyter_code>import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
import random

from numpy.linalg import inv, eig<jupyter_output><empty_output><jupyter_text>We define the number of points in each class, covariance matrix and mean of every class in accordance with the requirements of the exercise.<jupyter_code>n = 30
cov = np.array([[1, 0], [0, 1]])
means = {'A': np.array([-1, 1]), 'B': np.array([2, 4]), 'C': np.array([-2, 2])}<jupyter_output><empty_output><jupyter_text>We generate the data from multivariate normal distribution and insert it into the dataframe<jupyter_code>data = pd.DataFrame(index=range(n * len(means)), columns=['x', 'y', 'label'])
for i, (label, mean) in enumerate(means.items()):
    data.loc[i*n:((i+1)*n)-1, ['x', 'y']] = np.random.multivariate_normal(mean, cov, n)
    data.loc[i*n:((i+1)*n)-1, ['label']] = label<jupyter_output><empty_output><jupyter_text>Next we define a function that plots the points from the dataframe.<jupyter_code>def plot_points(data):
    fig = plt.figure(figsize=(8, 6))
    sns.set_style('white')
    sns.set_palette('muted')
    sns.scatterplot(data=data, x='x', y='y', hue='label', legend=False, edgecolor='black').set(xlim=(-5, 5), ylim=(-2, 6))
plot_points(data)<jupyter_output><empty_output><jupyter_text>Determination of optimal direction Before we move on to computing of optimal direction, we should update means of the classes and covariance matrix corresponding to the generated points.<jupyter_code>means = {label: np.array(np.mean(data.loc[data['label'] == label])) for label in means.keys()}
cov = {label: np.cov(data.loc[data['label'] == label][['x', 'y']].values.T.astype(float)) for label in means.keys()}<jupyter_output><empty_output><jupyter_text>We compute a mean of all the points without distinguishing classes. Next we compute between class covariance matrix based on the equation:\begin{equation}\boldsymbol{B} = \frac{1}{g - 1}\sum_{k=1}^{g}n_{k}(\boldsymbol{m_{k}} - \boldsymbol{m})(\boldsymbol{m_{k}} - \boldsymbol{m})^{T}\end{equation}and within class covariance matrix based on the equation:\begin{equation}\boldsymbol{W} = \frac{1}{n - g}\sum_{k=1}^{g}(n_{k} - 1)\boldsymbol{S_{k}}.\end{equation}However we can simplify the above expression into the form:\begin{equation}\boldsymbol{W} = \frac{1}{g}\sum_{k=1}^{g}\boldsymbol{S_{k}}\end{equation}because number of points is identical for every class.<jupyter_code>mean = np.mean(np.array([means[i] for i in means]), axis=0)
outer_products = [np.outer(value - mean, value - mean) for value in means.values()]
bc_cov = sum(n * outer_products) / (len(means) - 1)
wc_cov = np.mean(np.array([cov[i] for i in cov]), axis=0)<jupyter_output><empty_output><jupyter_text>We define the matrix $\boldsymbol{U}$:\begin{equation}\boldsymbol{U} = \boldsymbol{W^{-1}}\boldsymbol{B},\end{equation}determine eigenvalues and eigenvectors of that matrix and choose the eigenvector corresponding to the maximum eigenvalue.<jupyter_code>U = np.dot(inv(wc_cov), bc_cov)
eig_values, eig_vectors = eig(U)
a = eig_vectors[:, np.argmax(eig_values)]<jupyter_output><empty_output><jupyter_text>Projection of points on the line and decision boundary Having vector $\boldsymbol{a}$ that maximizes the expression:\begin{equation}\boldsymbol{J} = \frac{\boldsymbol{a^{T}Ba}}{\boldsymbol{a^{T}Wa}}\end{equation}it's possible to figure out the parameters of discriminant hyperplane.<jupyter_code>slope = a[1] / a[0]
intercept = -0.5 * np.dot(a.T, np.sum(np.array([means[i] for i in means]), axis=0))<jupyter_output><empty_output><jupyter_text>Now we can plot the line which the points will be projected on.<jupyter_code>x = np.arange(-5, 5, 0.1)
y = slope * x

plot_points(data)
sns.lineplot(x, y, color='red', linewidth=0.75);<jupyter_output><empty_output><jupyter_text>After that we define a function that projects the points on the line.<jupyter_code>def projection(data, slope):
    data_cp = data.copy()
    data_cp['x'] = (data['x'] + slope * data['y']) / (slope**2 + 1)
    data_cp['y'] = slope * data_cp['x']
    return data_cp
proj_data = projection(data, slope)<jupyter_output><empty_output><jupyter_text>Finally we can plot the projected points and the decision boundary that satisfies the equation:\begin{equation}a_{0}x + a_{1}y + b = 0,\end{equation}where $a_{0}$ and $a_{1}$ are the elements of vector $\boldsymbol{a}$ and $b$ is the intercept.<jupyter_code>plot_points(data)
sns.lineplot(x, y, color='red', linewidth=0.75)
sns.scatterplot(data=proj_data, x='x', y='y', hue='label', legend=False, marker='P', s=75, edgecolor='black')
sns.lineplot(x, -1 / slope * x - intercept / a[1], color='black', linewidth=2);<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1608668472598458, 'fraction_numerical': 0.012710981454469682, 'mean_word_length': 6.465007776049767, 'pattern_counts': {'":': 0, '<': 45, '<?xml version=': 0, '>': 45, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '167778', 'license': 'MIT', 'n_tokens_mistral': 1698, 'n_tokens_neox': 1634, 'n_words': 422, 'path': 'Exercise 0/Fisher Discriminant Analysis.ipynb', 'repo_name': 'mickuz/lsed', 'size': '4799'}