<jupyter_start><jupyter_text>Chapter 4: Classification- **Chapter 4 from the book [An Introduction to Statistical Learning](https://www.statlearning.com/).**- **By <NAME>, <NAME>, <NAME> and <NAME>.**- **Pages from $168$ to $170$**- **By [<NAME>](https://www.linkedin.com/in/mosta-ashour/)****Exercises:**- **[1.](1)**- **[2.](2)**- **[3.](3)**- **[4.](4)**- **[5.](5)**- **[6.](6)**- **[7.](7)**- **[8.](8)**- **[9.](9)**   4.7 Exercises   Conceptual   $1.$ Using a little bit of algebra, prove that $(4.2)$ is equivalent to $(4.3)$. In other words, the logistic function representation and logit representation for the logistic regression model are equivalent.>- **Answer:**$$p(x) = \frac{e^{\beta_0+\beta_1 x}}{1 + e^{\beta_0+\beta_1 x}}.  \tag{4.2}$$$\text{Let:   }$$$z = e^{\beta_0+\beta_1 x}$$$\text{So:   }$ $$p(x) = \frac{z}{1 + z}$$$$\frac{1}{p(x)} = \frac{1+z}{z} = \frac{1}{z} + \frac{z}{z}$$$$\frac{1}{p(x)} = \frac{1}{z} + 1$$$$\frac{1}{p(x)} - 1 = \frac{1}{z}$$$$\frac{1}{p(x)} - \frac{p(x)}{p(x)} = \frac{1}{z}$$$$\frac{1-p(x)}{p(x)} = \frac{1}{z}$$$$\frac{p(x)}{1-p(x)} = z$$$$\frac{p(x)}{1-p(x)} = e^{\beta_0+\beta_1 x} \tag{4.3}$$- **So $(4.2)$ is equivalent to $(4.3)$**.  $2.$ It was stated in the text that classifying an observation to the class for which $(4.12)$ is largest is equivalent to classifying an observation to the class for which $(4.13)$ is largest. Prove that this is the case. In other words, under the assumption that the observations in the kth class are drawn from a $N(\mu_k,\sigma^2)$ distribution, the Bayes’ classifier assigns an observation to the class for which the discriminant function is maximized.>- **Answer:**$$p_k(x) = \frac{\pi_k \color{red}{\frac{1}{\sqrt{2 \pi \sigma}}} \exp (- \frac{1}{2 \sigma^2} (x-\mu_k)^2)}                {\sum^{K}_{l=1} \pi_l \color{red}{\frac{1}{\sqrt{2 \pi \sigma}}} \exp(- \frac{1}{2 \sigma^2} (x-\mu_l)^2)}. \tag{4.12}$$$$p_k(x) = \frac{\pi_k \exp (- \frac{1}{2 \sigma^2} (x-\mu_k)^2)}                {\sum^{K}_{l=1} \pi_l \exp(- \frac{1}{2 \sigma^2} (x-\mu_l)^2)}$$                                $\log(p_k(x)) = \log(\pi_k \exp (- \frac{1}{2 \sigma^2} (x-\mu_k)^2)) - \log(\sum^{K}_{l=1} \pi_l \exp(- \frac{1}{2 \sigma^2} (x-\mu_l)^2))$- **$\log(\sum^{K}_{l=1} \pi_l \exp(- \frac{1}{2 \sigma^2} (x-\mu_l)^2)$ doesn't vary with k.**$$\begin{align}&= \log(\pi_k) + (- \frac{1}{2 \sigma^2} (x-\mu_k)^2)\\\\&= \log(\pi_k) + (- \frac{1}{2 \sigma^2} (x^2 -2x\mu_k + \mu_k^2))\\\\&= \log(\pi_k) - \frac{x^2}{2 \sigma^2} + \frac{\color{red}{2}x\mu_k}{\color{red}{2} \sigma^2} - \frac{\mu_k^2}{2 \sigma^2}\\\end{align}$$- $\frac{-x^2}{2 \sigma^2}$ doesn't vary with $k$.- **So $p_k(x)$ is maximized by maximizing:**$$\delta_k(x)= x.\frac{\mu_k}{\sigma^2} - \frac{\mu_k^2}{2 \sigma^2} + \log(\pi_k). \tag{4.13}$$  $3.$ This problem relates to the $QDA$ model, in which the observations within each class are drawn from a normal distribution with a class- specific mean vector and a class specific covariance matrix. We consider the simple case where $p = 1$; i.e. there is only one feature. Suppose that we have $K$ classes, and that if an observation belongs to the $k$th class then $X$ comes from a one-dimensional normal distribution, $X ∼ N(\mu_k,\sigma_{k}^{2})$. Recall that the density function for the one-dimensional normal distribution is given in $(4.11)$. Prove that in this case, the Bayes’ classifier is not linear. Argue that it is in fact quadratic. *Hint: For this problem, you should follow the arguments laid out in Section $4.4.2$, but without making the assumption that $\sigma_{1}^{2} = \dots = \sigma_{K}^{2}$.*>- **When $\sigma_{1}^{2} != \dots != \sigma_{K}^{2}$ then (4.12) would be as following:**$$p_k(x) = \frac{\pi_k \frac{1}{\sqrt{2 \pi \sigma_k}} \exp (- \frac{1}{2 \sigma_{k}^{2}} (x-\mu_k)^2)}                {\sum^{K}_{l=1} \pi_l \frac{1}{\sqrt{2 \pi \sigma_k}} \exp(- \frac{1}{2 \sigma_{k}^{2}} (x-\mu_l)^2)}$$>- **In this case, and from the previous question the term $\frac{-x^2}{2 \sigma_{k}^{2}}$ varies with $k$.**>- **Therefore we can't ignore the quadratic term $x^2$ anymore.**>- **So $p_k(x)$ is maximized by maximizing:**$$\delta_k(x)= \frac{-x^2}{2 \sigma_{k}^{2}} + x.\frac{\mu_k}{\sigma_{k}^{2}} - \frac{\mu_k^2}{2 \sigma_{k}^{2}} + \log(\pi_k)$$  $4.$ When the number of features $p$ is large, there tends to be a deterioration in the performance of $\text{KNN}$ and other local approaches that perform prediction using only observations that are near the test observation for which a prediction must be made. This phenomenon is known as the curse of dimensionality, and it ties into the fact that non-parametric approaches often perform poorly when p is large. We will now investigate this curse.**$(a)$ Suppose that we have a set of observations, each with measurements on $p = 1$ feature, $X$. We assume that $X$ is uniformly (evenly) distributed on $[0,1]$. Associated with each observation is a response value. Suppose that we wish to predict a test observation's response using only observations that are within $10$% of the range of $X$ closest to that test observation. For instance, in order to predict the response for a test observation with $X = 0.6$, we will use observations in the range $[0.55,0.65]$. On average, what fraction of the available observations will we use to make the prediction?**>**Answer:**>- **On average, the fraction would be $10\%$ for $x$ in range $[0.05, 0.95]$** **$(b)$ Now suppose that we have a set of observations, each with measurements on $p = 2$ features, $X_1$ and $X_2$. We assume that $(X_1,X_2)$ are uniformly distributed on $[0,1]\times[0,1]$. We wish to predict a test observation's response using only observations that are within 10 % of the range of $X_1$ and within 10 % of the range of $X_2$ closest to that test observation. For instance, in order to predict the response for a test observation with $X_1 = 0.6$ and $X_2 = 0.35$, we will use observations in the range $[0.55, 0.65]$ for $X_1$ and in the range $[0.3, 0.4]$ for $X_2$. On average, what fraction of the available observations will we use to make the prediction?**>**Answer:**>- **For $p=2$, on average, the fraction would be $0.10^{2} = 10^0 \% =  1\%$** **$(c)$ Now suppose that we have a set of observations on $p = 100$ features. Again the observations are uniformly distributed on each feature, and again each feature ranges in value from $0$ to $1$. We wish to predict a test observation's response using observations within the $10\%$ of each feature's range that is closest to that test observation. What fraction of the available observations will we use to make the prediction?**>**Answer:**>- **For $p=100$, on average, the fraction would be $0.10^{100}= 10^{-98}\%$** **$(d)$ Using your answers to parts $\text{(a) – (c)}$, argue that a drawback of $\text{KNN}$ when p is large is that there are very few training observations “near” any given test observation.**>- On average, the fraction would be the percentage of the observations of each feature's range over the number of futures $p$>- For $r_1 \neq \dots \neq r_n$:>  - the fraction would be $r_1\times r_2\times \dots \times r_n$>- For $r_1 = \dots = r_n$:>  - the fraction would be ${r^p}$>- **where:**>  - $r_n$ is the range of prediction for each $X_n$>  - $p$ is the number of features- As $p$ increases, the geometrically near decrease exponentially. **$(e)$ Now suppose that we wish to make a prediction for a test observation by creating a p-dimensional hypercube centered around the test observation that contains, on average, $10\%$ of the training observations. For $p = 1,2$, and $100$, what is the length of each side of the hypercube? Comment on your answer.***Note: A hypercube is a generalization of a cube to an arbitrary number of dimensions. When $p = 1$, a hypercube is simply a line segment, when $p = 2$ it is a square, and when $p = 100$ it is a 100-dimensional cube.*>- **Answer**>  - For $p-dimensional$ hypercube. the equation of length is:>  - $l^p = r$>  - **So:**>  - $l = r^{1/p}$>  - $l$ is the length of one side.- **The below plot shows a simulated example where p = 2, in order for $10\%$ of the training observations to fall within the hypercube.**<jupyter_code>def hypercube_length(p, perc):
    print(round(perc**(1/p),4))

print("With p = 1:")
hypercube_length(1, 0.1)

print("\nWith p = 2:")
hypercube_length(2, 0.1)

print("\nWith p = 100:")
hypercube_length(100, 0.1)
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

np.random.seed(1)
# creating the data
df = pd.DataFrame({'x1':np.random.rand(5000),
                   'x2':np.random.rand(5000)})
# choising the first observation
re_p = df.iloc[0]-[0.3162/2, 0.3162/2]

fig, ax = plt.subplots(figsize=(12,12))

# Display the points
ax.scatter('x1', 'x2', data=df, alpha=0.4)
ax.scatter('x1', 'x2', data=df.iloc[0], c='black')

# Create a hypercube patch
rect = plt.Rectangle(re_p, 0.3162, 0.3162, alpha=0.4, linewidth=5, edgecolor='r', facecolor='r');

# Add the patch to the Axes
ax.add_patch(rect);<jupyter_output><empty_output><jupyter_text>$5.$ We now examine the differences between $LDA$ and $QDA$.**$(a)$ If the Bayes decision boundary is linear, do we expect $LDA$ or $QDA$ to perform better on the training set? On the test set?**>- **Answer:**>  - On the **Training** set: $QDA$ should perform better because of its higher flexibility to fit noise on the data "overfitting".>  - On the **Test** set: $LDA$ should perform better as $QDA$ overfitted the training set. **$(b)$ If the Bayes decision boundary is non-linear, do we expect $LDA$ or $QDA$ to perform better on the training set? On the test set?**>- **Answer:**>  - On the **Training** set: $QDA$ should perform way better because of its higher flexibility to fit noise on the data "overfitting".>  - On the **Test** set: $QDA$ still should perform better. **$(c)$ In general, as the sample size $n$ increases, do we expect the test prediction accuracy of $QDA$ relative to $LDA$ to improve, decline, or be unchanged? Why?**>- **Answer:**>  - The test prediction accuracy would **improve**, as sample size increases, a higher flexibility model will fit the data more. **$(d)$ True or False: Even if the Bayes decision boundary for a given problem is linear, we will probably achieve a superior test error rate using $QDA$ rather than $LDA$ because $QDA$ is flexible enough to model a linear decision boundary. Justify your answer.**>- **False**. More flexible model tend to `overfit` the training set plus the noise, that will increase its test error rate.  $6.$ Suppose we collect data for a group of students in a statistics class with variables $X_1 =$ hours studied, $X_2 =$ undergrad $GPA$, and $Y =$ receive an A. We fit a logistic regression and produce estimated coefficient, $\beta^0 = −6, \beta^1 = 0.05$, $\beta^2 = 1$.**$(a)$ Estimate the probability that a student who studies for 40 h and has an undergrad $GPA$ of $3.5$ gets an A in the class.**>- For multiple logistic regression a prediction p(X) is given by $$p(X) = \frac{\exp{(\beta_0+\beta_1 X_1 + \beta_2 X_2)}}{1 + \exp{(\beta_0+\beta_1 X_1 + \beta_2 X_2)}}$$<jupyter_code>beta  = np.array([-6, 0.05, 1])
X     = np.array([1, 50, 3.5])

lr = np.exp(np.dot(beta, X)) / (1 + np.exp(np.dot(beta, X)))

print('p(X) = ' + str(np.around(lr, 4)))<jupyter_output>p(X) = 0.5<jupyter_text>**$(b)$ How many hours would the student in part $(a)$ need to study to have a $50\%$ chance of getting an A in the class?**$$p(X) = \frac{\exp{(\beta_0+\beta_1 X_1 + \beta_2 X_2)}}{1 + \exp{(\beta_0+\beta_1 X_1 + \beta_2 X_2)}}$$$$0.5 = \frac{\exp{(0.05X_1 + -2.5)}}{\color{red}{1 + \exp{(0.05X_1 + -2.5)}}}$$$$ 0.5 + \color{red}{0.5\exp{(0.05X_1 + -2.5)}} = \exp{(0.05X_1 + -2.5)}$$$$ 0.5 = 0.5\exp{(0.05X_1 + -2.5)} \tag{*2}$$$$ 1 = \exp{(0.05X_1 + -2.5)} \tag{log}$$$$ 1 = 0.05X_1 + -2.5$$$$ 0.05X_1 = 2.5$$$$ X_1 = \frac{2.5}{0.05} = 50 \text{ hours}$$  $7.$ Suppose that we wish to predict whether a given stock will issue a dividend this year ("Yes" or "No") based on $X$, last year's percent profit. We examine a large number of companies and discover that the mean value of $X$ for companies that issued a dividend was $\bar{X}= 10$, while the mean for those that didn't was $\bar{X}= 0$. In addition, the variance of $X$ for these two sets of companies was $\bar{\sigma}^2 = 36$. Finally, $80\%$ of companies issued dividends. Assuming that $X$ follows a normal distribution, predict the probability that a company will issue a dividend this year given that its percentage profit was $X = 4$ last year. Hint: Recall that the density function for a normal random variable is $f(x) = \frac{1}{\sqrt{2\pi\sigma^2}} e^{-(x-\mu)^2/2\sigma^2}$ . You will need to use Bayes' theorem. $$p_k(x) = \frac {\pi_k                \frac {1} {\sqrt{2 \pi} \sigma}                \exp(- \frac {1} {2 \sigma^2} (x - \mu_k)^2)               }               {\sum {                \pi_l                \frac {1} {\sqrt{2 \pi} \sigma}                \exp(- \frac {1} {2 \sigma^2} (x - \mu_l)^2)               }}\\       p_{yes}(x)= \frac {\pi_{yes}                \exp(- \frac {1} {2 \sigma^2}  (x - \mu_{yes})^2)               }               {\sum {                \pi_l                \exp(- \frac {1} {2 \sigma^2} (x - \mu_l)^2)               }}\\       = \frac {\pi_{yes} \exp(- \frac {1} {2 \sigma^2}  (x - \mu_{yes})^2)}               {                \pi_{yes} \exp(- \frac {1} {2 \sigma^2} (x - \mu_{yes})^2) +                \pi_{no}  \exp(- \frac {1} {2 \sigma^2}  (x - \mu_{no})^2)               }\\      = \frac {0.80 \exp(- \frac {1} {2 * 36}  (x - 10)^2)}               {                0.80 \exp(- \frac {1} {2 * 36} (x - 10)^2) +                0.20 \exp(- \frac {1} {2 * 36}  x^2)               }\\p_{yes}(4)  = \frac {0.80 \exp(- \frac {1} {2 * 36}  (4 - 10)^2)}               {                0.80 \exp(- \frac {1} {2 * 36} (4 - 10)^2) +                0.20 \exp(- \frac {1} {2 * 36}  4^2)               }        = 75.19\%$$<jupyter_code>round(0.8*np.exp(-(1/72) *36) / (0.8*np.exp(-(1/72) *36) + (0.2*np.exp(-16/72))), 4) * 100<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.18943417604985482, 'fraction_numerical': 0.0437646059060973, 'mean_word_length': 3.4076154806491887, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 53, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '436980', 'license': 'MIT', 'n_tokens_mistral': 5141, 'n_tokens_neox': 4779, 'n_words': 1818, 'path': 'Notebooks/4_7_0_Classification_Conceptual.ipynb', 'repo_name': 'MostaAshour/ISL-in-python', 'size': '14156'}