<jupyter_start><jupyter_text>COMP 562 : Inroduction to Machine Learning   Lecture 5 : Feature Scaling, Logistic Regression     <NAME>    Department of Computer Science -- UNC Chapel Hill   September 10, 2018   COMP 562 – Lecture 5Plan for today1. Review closed form linear regression, ridge regression2. Introduce sigmoid function3. Logistic regression 4. Geometric view of logistic regression -- separating hyperplanes$$\renewcommand{\xx}{\mathbf{x}}\renewcommand{\yy}{\mathbf{y}}\renewcommand{\zz}{\mathbf{z}}\renewcommand{\vv}{\mathbf{v}}\renewcommand{\loglik}{\log\mathcal{L}}\renewcommand{\likelihood}{\mathcal{L}}\renewcommand{\Data}{\textrm{Data}}\renewcommand{\given}{ | }\renewcommand{\MLE}{\textrm{MLE}}\renewcommand{\tth}{\textrm{th}}\renewcommand{\Gaussian}[2]{\mathcal{N}\left(1,2\right)}\renewcommand{\norm}[1]{\left\lVert1\right\rVert}\renewcommand{\ones}{\mathbf{1}}\renewcommand{\diag}[1]{\textrm{diag}\left( 1 \right)}\renewcommand{\sigmoid}[1]{\sigma\left(1\right)}\renewcommand{\myexp}[1]{\exp\left\{1\right\}}$$  Last time -- Closed Form Solution for Linear RegressionIn matrix form, we can rewrite this model as $$\left[\begin{array}{c} { y_1 \\ y_2 \\ \vdots \\ y_N}\end{array}\right]_{\ N \times 1} = \left[\begin{array}{ccccc} 1  & x_{11} & x_{12} & \ldots & x_{1p} \\ 1  & x_{21} & x_{22} & \ldots & x_{2p} \\ \vdots  & \vdots & \vdots & \ldots & \vdots \\ 1  & x_{N1} & x_{N2} & \ldots & x_{Np} \end{array}\right]_{\ N \times p+1} \left[\begin{array}{c} \ { \beta_0 \\ \beta_1 \\ \vdots \\ \beta_P}\end{array}\right]_{\ p+1 \times 1} \ + \left[\begin{array}{c} \ { \epsilon_1 \\ \epsilon_2 \\ \vdots \\ \epsilon_N}\end{array}\right]_{\ N \times 1}$$This can be rewritten more simply as:$$\yy = \mathbf{X} \mathbf{\beta} + \mathbf{\epsilon}$$Once we compute $\beta^{\MLE}$, we can get $(\sigma^2)^\MLE$ in closed form$$(\sigma^{2})^\MLE = \frac{1}{N} (\yy -  \mathbf{X} \beta^\MLE)^T(\yy -  \mathbf{X} \beta^\MLE)  = \frac{1}{N} \sum_{i=1}^N \left(y_i - (\beta_0^\MLE + \sum_{j=1}^p x_{i,j} \beta_j^\MLE)\right)^2$$  Last time -- RegularizationIll-posed problems have many solutionsOne way to break the ties between different solutions is to add regularizationRidge regression is adds regularization to the log-likelihood:$$\loglik(\beta\given\yy,\xx) =  -\frac{1}{2\sigma^2}\norm{\yy - \mathbf{X} \mathbf{\beta}}^2  \underbrace{-\frac{\lambda}{2}\norm{\beta}^2}_{\textrm{ridge penalty}}+ \textrm{const.}$$Using this regularization spreads the weights between correlated predictors Setting $\lambda=0$ recovers the linear regression log-likelihood  Last time -- Closed Form Solution for Ridge RegressionOur closed form solution for ridge regression$$\beta^{\MLE} = (\mathbf{X}^T\mathbf{X} + \color{red}{\lambda\sigma^2 \mathbf{I}_N})^{-1}(\mathbf{X}^T\yy)$$The only difference between the closed form of linear regression and ridge regression is the addition of the term shown in red.The bias/intercept coefficient $\beta_0$ is typically not regularized in a linear regressionUpdating our closed form solution without regularizing $\beta_0$ will yeild$$\beta^{\MLE} = \left(\mathbf{X}^T\mathbf{X} + \color{red}{\lambda\sigma^2 \left[\begin{array}{ccccc} 0  & 0 & 0 & \ldots & 0 \\ 0  & 1 & 0 & \ldots & 0 \\ \vdots  & \vdots & \vdots & \ldots & \vdots \\ 0  & 0 & 0 & \ldots & 1 \end{array}\right]_{\ N \times N}}\right)^{-1}(\mathbf{X}^T\yy)$$  Feature Scaling -- Feature Scaling* Idea: gradient ascent/descentalgorithm tends to work better if the features are on the **same scale** When contours are skewed then learning steps would take longer to converge due to oscillatory behaviour  Feature Scaling -- Centering**Center** features by removing the mean$$\begin{aligned}\mu_i &= \frac{1}{N}\sum_{k=1}^N x_{i,k}\\ \\x_{i,j} &= x_{i,j} - \mu_i\end{aligned}$$This makes each feature's mean equal to 0. Compute the mean first, then subtract it!  Feature Scaling -- Standardizing**Standardize** centered features by dividing by the standard deviation$$\begin{aligned}\sigma_i &= \sqrt{ \frac{1}{N-1}\sum_j x_{i,j}^2 }\\ \\x_{i,j}& = \frac{x_{i,j}}{\sigma_i}\end{aligned}$$Note that standardized features are first centered and then divided by their standard deviationTransform your data to a distribution that has a mean of 0 and a standard deviation of 1 (z-score)  Feature Scaling -- NormalizingAlternatively, **normalize** centered features by dividing by their norm$$\begin{aligned}r_i &= \sqrt{\sum_j x_{i,j}^2 }\\ \\x_{i,j}& = \frac{x_{i,j}}{r_i}\end{aligned}$$Note that normalized features are first centered and then divided by their normNormalization transforms your data into a range between 0 and 1 regardless of the data set size  Feature Scaling Benefits1. Centering   1. $\beta_0$ is equal to the mean of the target variable   2. Feature weights $\beta$ now tell us how much does feature's departure from mean affect the target variable 2. Standardization  1. All the features are on the same scale and their effects comparable  2. Interpretation is easier: $\beta$s tell us how much departure by single standard deviation affects the target variable  3. Normalization  1. Scale of features is the same, regardles of the size of the dataset  2. Hence weights learend on different sized datasets can be compared  3. However, their combination might be problematic -- certainly we don't trust weights learned on few samples  Classification* Tumor: Malignant / Bnign ? -- $y \in \{0,1\}$     * $y$: "Negative Class" (e.g., benign tumor)    * $y$: "Positive Class" (e.g., malignant tumor)    * Threshold classifier output $\hat{y}$ at 0.5:    * If $\hat{y}$ $\geq$ 0.5, then predict "$y$ = 1"    * If $\hat{y}$ $<$ 0.5, then predict "$y$ = 0"          ** Q: What would be the problem with using linear regression for a classification task?  **  We need to modify our linear regression model to perform the required classification taskRemember: we used linear regression to fit a predictive model of continuous variables:$$y\given\xx \sim \Gaussian{\beta_0 + \xx^T\beta}{\sigma^2}$$** Q: Gaussian distribution is not well suited for modelling discrete variables, why? What would you change? **  Classification -- Bernoulli ViewWe can model a target variable $y \in \{0,1\}$  using Bernouli distribution$$p(y=1\given\theta) = \theta$$We note that $\theta$ has to be in range $[0,1]$We cannot directly take weighted combination of features to obtain $\theta$We need a way to map $\xx^T\beta \in \mathbb{R}$ to range $[0,1]$  Sigmoid (Logistic) Function$$\sigma(z) = \frac{1}{1 + \exp(-z)}$$** Q: Argue that this function maps real line  to range $[0,1]$?  **** Q: When does it achieve $0.5$? ****  Q: Where does it achieve $0$? Where does it achieve $1$? **<jupyter_code>import numpy
import matplotlib.pyplot as plt
%matplotlib inline
x = numpy.arange(-10,10,0.1)
scales = [1.0,0.5,2.0,10.0]
labels = []
for s in scales:
    plt.plot(x,1.0/(1.0 + numpy.exp(-s*x)))
    labels.append('sigma(' + str(s) +'*x)')
plt.xlabel('x')
plt.title('Sigmoid')
plt.legend(labels,loc=4)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.16112059158134243, 'fraction_numerical': 0.022468714448236633, 'mean_word_length': 4.328030303030303, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 6, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '142489', 'license': 'MIT', 'n_tokens_mistral': 2291, 'n_tokens_neox': 2125, 'n_words': 746, 'path': 'CourseMaterial/COMP562_Lect5/COMP562_Lect5.ipynb', 'repo_name': 'SiyangJ/COMP562', 'size': '7042'}