<jupyter_start><jupyter_text>Assignment 1.LDA and Logistic Regression<jupyter_code>par(family  = "Arial")
#install.packages("showtext")
library(showtext)
showtext_auto()
options(repr.plot.width=5, repr.plot.height=5)<jupyter_output><empty_output><jupyter_text>![image.png](attachment:image.png)<jupyter_code>attach(iris)
Width = iris$Sepal.Width
Lenght = iris$Petal.Length

unique(as.factor(iris$Species))
plot(iris$Sepal.Length,iris$Sepal.Width,
col=c(5,7,6)[iris$Species], pch=21,
bg=c(5,7,6)[iris$Species],
main="Width vs Length")
legend('topright',col = c(5,7,6),legend =unique(as.factor(iris$Species)),
      pch=c(19,19,19))<jupyter_output><empty_output><jupyter_text>There are three different classes observed in the scatterplot. Setosa can be perfectly classified linearly.Whereas, some of the data points of the species versicolor and virginica are mixed non linearly. So, therecould be some errors concerning the classification of the species using LDA. ![image.png](attachment:image.png)<jupyter_code>library(MASS)
resLDA = lda(Species~ Sepal.Length + Sepal.Width,data = iris)
resLDA
(resLDA$means)
resLDA$prior
table(iris$Species)
#setosas = filter(iris, Species == "setosa") #Filter iris data from setosas
setosas = subset(iris , iris$Species == 'setosa',select=c(1:2))
versicolors = subset(iris, Species == 'versicolor',select=c(1:2))
virginicas = subset(iris, Species == 'virginica',,select=c(1:2))
s2 = cov(versicolors)
s2
s1 = cov(setosas)
s1
s3 = cov(virginicas)
s3
prior = c(1/3,1/3,1/3)
prior
m1 = colMeans(setosas)
m1
m2 = colMeans(versicolors)
m2
m3 = colMeans(virginicas)
m3
df = data.frame(m1,m2,m3)
names(df) = c('setosa','versicolor','virginica')
df
paste("degree of freedom: ")
(table(iris$Species)-1 -> dof)

paste("degree of freedom: ")
(table(iris$Species)-1 -> dof)

# b
pooled_cov = (1/3) * (s1 + s2 + s3)
pooled_cov<jupyter_output><empty_output><jupyter_text>C) probabilistic model: The probability of $p(X = x | Y = k)$ is given by:$$f_k(x) = \frac{1}{(2 \pi)^{p/2} |\Sigma|^{1/2}} \exp \left( - \frac{1}{2} (x - \mu_k)^T \Sigma^{-1} (x - \mu_k) \right)$$By considering prior probability is $P(Y = k) = \pi_k$ , and taking logarithm , we will find Linear discriminant function or Linear score function:$$\delta_k(x)=x^T\Sigma^{-1}\mu_k-\frac{1}{2}\mu_{k}^{T}\Sigma^{-1}\mu_{k} + log(\pi_k)$$$$\hat{G}(x)= \text{ arg }\underset{k}{max}\delta_k(x)$$The decision boundary is the set of points in which two classes are equally probable:$$\delta_k(x) = \delta_l(x)$$We consider $ \pi_k, \ \mu_k, \ \Sigma$ by MLE as below:$$\hat \Sigma = \frac{1}{m} \sum_{i=1}^m (x^{(i)} - \mu_{y^{(i)}})(x^{(i)} - \mu_{y^{(i)}})^T$$$$ \hat \pi_k = \frac{\{i\;;\;y_i=k\}}{n}$$ ![image.png](attachment:image.png)  d)![image.png](attachment:image.png)the discriminant function for each class:we need to calculate these:![image-3.png](attachment:image-3.png)<jupyter_code>X = as.matrix(iris[,-c(3, 4, 5)])
inv_cov = solve((pooled_cov))
inv_cov
m1 = as.matrix(m1)
m2 = as.matrix(m2)
m3 = as.matrix(m3)
#line intercept
w0.1 = - 0.5*(t(m1) %*% inv_cov %*% m1) + log(prior[1]) #setosa
w0.2 = -0.5 * (t(m2) %*% inv_cov %*% m2) + log(prior[2]) #versicolor
w0.3 = -0.5 * (t(m3) %*% inv_cov %*% m3) + log(prior[3]) #virginica
round(c(w0.1,w0.2,w0.3),2)
#slope

G1 = c(round(inv_cov %*% m1,3) )
G1

G2 = c(round(inv_cov %*% m2,3) )
G2
G3 = c(round(inv_cov %*% m3,3) )
G3
# c)
# x|y = C_i, mu_i, Sigma ~ N(mu_i, Sigma)
# y|pi ~ Multinomial(pi_1, ..., pi_k)

# d)
# w_0i = -1/2 * mu_i^T * Sigma^-1 * mu_i + log(pi_i)
# w_i = Sigma^-1 * mu_i
# delta_k = x^T * w_k + w_0k

delta1 = X %*% G1 + c( w0.1)
head(delta1,3)<jupyter_output><empty_output><jupyter_text>discriminant functions ![image.png](attachment:image.png)  e) the decision boundaries: **Compute equations of decision boundaries between classes and report them**<jupyter_code># e)
# (w_1 - w_2)_l * x + (w_1 - w_2)_w * y + (w0_1 - w0_2) = 0
# (w_1 - w_3)_l * x + (w_1 - w_3)_w * y + (w0_1 - w0_3) = 0
# (w_2 - w_3)_l * x + (w_2 - w_3)_w * y + (w0_2 - w0_3) = 0ca
#first Length # second Width<jupyter_output><empty_output><jupyter_text>decision boundary for Setosa ~ Versicolor:<jupyter_code>w1 = G1- G2
w1

b.0 = w0.1 - w0.2
 b.0<jupyter_output><empty_output><jupyter_text>v e r s i c o l o r − v i r g i n i c a :<jupyter_code>w2 = G2 - G3
w2
b.2 =  w0.2 - w0.3
b.2<jupyter_output><empty_output><jupyter_text>decision boundary for Verginica ~ Setosa<jupyter_code>w3 = G3 - G1
w3
b.3 =  w0.3 - w0.1
b.3<jupyter_output><empty_output><jupyter_text>C ![image.png](attachment:image.png)<jupyter_code>discr_setosa = X %*% G1 + c(w0.1)
discr_versicolor = X %*% G2 + c(w0.2)
discr_virginica = X %*% G3 + c(w0.3)
pred = matrix(NA,n,1)
for(i in 1:n){
    
    pred[i,] = which.max(c(discr_setosa[i],discr_versicolor[i],discr_virginica[i]))
}
pred[which(pred == 1) ,] = 'setosa'
pred[which(pred == 2) ,] = 'versicolor'
pred[which(pred == 3) ,] = 'virginica'
pred = as.factor(pred)
lda_predicted = table(pred,iris$Species)
lda_predicted
missclassification_rate_lda <- 1 - (sum(diag(lda_predicted))/nrow(iris))
print(missclassification_rate_lda)

plot(X,main="prediction",pch=20,col = c(2,3,4)[iris$Species])
points(X,
col=c(2,3,4)[pred], pch=c(7,8,9)[pred],
bg=c(2,3,4)[pred])
legend('topright',col = c(2,3,4),legend =unique(as.factor(pred)),
      pch=c(7,8,9))<jupyter_output><empty_output><jupyter_text>the error rate is 0.2. the overal prediction is perfect for setosa since it is linearly seperable, however, since versicolor and virginica are not linearly seprable we can see that is where misclassification has occured.the points with 2 color are the misclassified one on the plot.![image.png](attachment:image.png)<jupyter_code>resLDA = lda(Species~ Sepal.Length + Sepal.Width,data = iris)
resLDA
temp = table(iris$Species,predict(resLDA)$class)
temp
misclass = 1 -  sum( diag(temp))/n
misclass<jupyter_output><empty_output><jupyter_text>4. Use Models reported in 2c to generate new data of this kind with the same total number of cases as in the original data (hint: use sample() and rmvnorm() from package mvtnorm). Make a scatterplot of the same kind as in step 1 but for the new data and compare it with the plots for the original and the predicted data. Conclusions?<jupyter_code>library(mvtnorm)
set.seed(12345)
predicted_iris_data = matrix(NA,nrow = 150,ncol = 3 )
colnames(predicted_iris_data) = c('sepal_length', 'sepal_width','Species')
for(i in 1:n){
    spec <- sample(unique(iris$Species), 1, replace=TRUE, prob=prior)
    if (spec == 'setosa'){
        
      predicted_iris_data[i,  ]   = c(rmvnorm(1,mean = m1,sigma = pooled_cov),'setosa')
    }
    
    else if (spec == 'versicolor' ){
       predicted_iris_data[i,  ] =  c(rmvnorm(1,mean = m2, sigma = pooled_cov),'versicolor')
    }
    else{
        
        predicted_iris_data[i,  ] = c(rmvnorm(1,mean=m3,sigma = pooled_cov),'virginica')
    }
}
predicted_iris_data = predicted_iris_data[order(predicted_iris_data[,3 ], predicted_iris_data[,2 ], 
                                                    predicted_iris_data[,1 ]),]

class =iris$Species
plot(x = predicted_iris_data[,1],
    y = predicted_iris_data[,2],col = c(2,3,4)[class],pch=19,main='simulated',
    xlab = 'Length',ylab= 'width')
legend('topright',col = c(2,3,4),legend =unique(predicted_iris_data[,3]),
      pch=20)<jupyter_output><empty_output><jupyter_text>The predicted sample looks similar to the plot obtained using original iris data. Setosa is almost same exceptfor some outliers which are present in the versicolor cluster. In the original data many points were exactly onthe line i.e (the values like 2.5,2,3). Whereas in the predicted sample, most of the samples are not on thelines as observed. ![image.png](attachment:image.png)<jupyter_code>library(nnet)
logist_model = multinom(Species ~ Sepal.Width + Sepal.Length, data = iris)
logist_model
pred.logist_model = predict(logist_model,newdata = iris, "class")
lg.table = table(pred.logist_model,iris$Species)
lg.table
misclass.lg = 1 - sum(diag(lg.table)/n)
cat('misclassification Error for logistic Regression: \n')
round(misclass.lg,3)
plot(X,main="Logistic Regression prediction",pch=20,col = c(2,3,4)[iris$Species])
points(X,
col=c(2,3,4)[pred.logist_model], pch=c(7,8,9)[pred],
bg=c(2,3,4)[pred])
legend('topright',col = c(2,3,4),legend =unique(as.factor(pred.logist_model)),
      pch=c(7,8,9))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.17107773009060562, 'fraction_numerical': 0.03457319980925131, 'mean_word_length': 5.293323330832708, 'pattern_counts': {'":': 0, '<': 51, '<?xml version=': 0, '>': 51, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '290473', 'license': 'CC0-1.0', 'n_tokens_mistral': 3599, 'n_tokens_neox': 3444, 'n_words': 723, 'path': 'LDA(Linear Discriminant Analysis)/Assignment 1.LDA and Logistic Regression.ipynb', 'repo_name': 'hodfa840/machine-learning', 'size': '8388'}