<jupyter_start><jupyter_text>Week 1 Applications in supervised learning:* Prediction of price or click on ad (yes/no) --> Standard NN* From image to object (photo tagging) --> Convolutional NN * From audio to text trasncript (speech recognicion) --> Recurrent NN* Translation from language to language --> More advanced RNN* From image/radars to position of other cars (autonomous driving) --> Custom/Hybrid NN Types of data:* Structured data: databases* Unstructured data: audio, images, text   Week 2  Logistic regression as a NN  Binary classification Training sample $X \in \mathbb{R}^{n_X \times m}$, where $n_X$ is the number of features and $m$ is the number of observations  Logistic regression We want to estimate $\mathbb{E}[y|x] = P(y=1|x) = \hat{y} \in [0,1]$.We set $\hat{y} = \sigma(w^Tx+b)$, the *sigmoid* function $\sigma(z) = \frac{1}{1+e^{-z}}$.<jupyter_code># the sigmoid function
import matplotlib.pylab as plt
%matplotlib inline
import numpy as np
x = np.arange(-8, 8, 0.1)
f = 1 / (1 + np.exp(-x))
plt.plot(x, f)
plt.axvline(x=0, ls = "--", c = 'black')
plt.axhline(y=1, c = 'black')
plt.axhline(y=0, c = 'black')
plt.xlabel('x')
plt.ylabel('f(x)')
plt.title('Sigmoid function')
plt.show()<jupyter_output><empty_output><jupyter_text>Logistic regression cost function Loss function $L(\hat{y},y):\mathbb{R}^m \rightarrow \mathbb{R}$ to measure how good our estimate is compared to the real $y$.A quadratic function such as $\frac{1}{2}(\hat{y}-y)^2$ is not good for gradient discent.We use $L(\hat{y},y) = -(y \ln(\hat{y}) +(1-y)\ln(1-\hat{y}))$:* if $y=1 \implies L(\hat{y},y) = -\ln(\hat{y})$, therefore we minimize $L$ by choosig the $w$ and $b$ such that $\hat{y} \in (0,1)$ is as large as possible* if $y=0 \implies L(\hat{y},y) = -\ln(1-\hat{y})$, therefore we want $\hat{y} \in (0,1)$ to be as small as possibleThe cost function $J(w,b) = \frac{1}{m}\sum_{i=1}^{m} L(\hat{y}^i,y^i) = -\frac{1}{m}\sum_{i=1}^{m} \bigl[y^i \ln(\hat{y}^i) +(1-y^i)\ln(1-\hat{y}^i)\bigr]$ is convex.Note: minimizing the cost funcion is equivalent to maximizing the log-likelihood function:$\ln P(y_1,...y_m|w,b) = \ln \prod_{i=1}^m  P(y_i|x) = \sum_{i=1}^{m} \ln P(y_i|x) =  \sum_{i=1}^{m}  \ln (\hat{y}_i^{y_i}(1-\hat{y}_i)^{1-y_i})$  Gradient discent Consider $w^*,b^* = argmin_{w,b} J(w,b)$.We find $w^*$ and $b^*$ by initializing some $(w^0,b^0)$ and by iterating* $w^{t+1} = w^t - \alpha \frac{\partial J(w^t,b^t)}{\partial w^t}$, where $\alpha$ is the *learning rate*.* $b^{t+1} = b^t - \alpha \frac{\partial J(w^t,b^t)}{\partial b^t}$Because this function is convex, no matter where you initialize, you should get to the same point or roughly the same point. And what gradient descent does is it starts at that initial point and then takes a step in the steepest downhill direction.We will denote $\partial x^t = \frac{\partial J(x^t)}{\partial x^t}$.By the chain rule, we have that$\partial w^t = \frac{\partial J(w^t,b^t)}{\partial w^t} = \frac{1}{m}\sum_{i=1}^{m} \frac{\partial L(\hat{y}_i,y_i)}{\partial \hat{y}_i} \frac{\partial \hat{y}_i}{\partial z_i} \frac{\partial z_i}{\partial w^t}$, where $\hat{y}_i = \sigma(z_i)$ and $z_i = w^Tx_i+b$.By using the sigmoid function, we have that $\frac{\partial \hat{y}}{\partial z} = \frac{\partial (1+e^z)^{-1}}{\partial z} = \frac{e^{-z}}{(1+e^z)^2}$Therefore $\partial z_i =  \frac{1}{m}\sum_{i=1}^{m} \frac{\partial L(\hat{y}_i,y_i)}{\partial \hat{y}_i}\frac{e^{-z_i}}{(1+e^{z_i})^2} = \frac{1}{m}\sum_{i=1}^{m} -\big(\frac{y_i}{\hat{y}_i} - \frac{1-y_i}{1-\hat{y}_i} \big) \frac{e^{-z_i}}{(1+e^{z_i})^2} = \frac{1}{m}\sum_{i=1}^{m} \hat{y}_i - y_i$.Hence$\partial w^t = -\frac{1}{m}\sum_{i=1}^{m} (y^i-\hat{y}^i) x^i$and$\partial b^t = -\frac{1}{m}\sum_{i=1}^{m} (y^i-\hat{y}^i)$.  Week 3  Shallow Neural Network  For a single observation $i$, let $a^{[0]} = [x_1, x_2, \dots x_{n_x}]$ be the input layer.Each node $j$  of the hidden layer is computed as $a^{[1]}_j = \sigma(z^{[1]}_j) = \sigma(w^{[1]T}_jx+b^{[1]}_j)$, therefore the hidden layer is based on a matrix of weights $W^{[1]}$ of dimension $n_x \times n_j$, where $n_j$ is the number of nodes of the hidden layer, and a vector $b^{[1]}$. The output layer is computed as $\hat{y} = a^{[2]} = \sigma(z^{[2]}) = \sigma(w^{[2]T} a^{[1]} + b^{[2]})$.The function that maps from $z^{[l]}$ to $a^{[l]}$ is called *activation function*.The number of units in each layer $l$ are denoted by $n^{[l]}$. We have $n^{[0]} = n_x$ and $n^{[2]} = 1$ Consider now the case where $m>1$, now the input layer $X = a^{[0]}$ has dimension $n_x \times m$. In the hidden layer there are $n^{[1]}$ units (for each observation). We compute $a^{[1]} = \sigma(z^{[1]})$ with $z^{[1]} = W^{[1]} X + b^{[1]}$ of dimension $n^{[1]} \times m$, where $W^{[1]}$ has dimension $n^{[1]} \times n_x$ and where for each observation of the same unit we sum a constant from $b^{[1]}$.For the case with 2 features, 2 observations and 3 nodes we have:$$W^{[1]} X + b^{[1]} = \left(\begin{array}{cc} w_{11} & w_{12}\\w_{21} & w_{22}\\w_{31} & w_{32}\end{array}\right)\left(\begin{array}{cc} x^{(1)}_1 & x^{(2)}_1\\ x^{(1)}_2 & x^{(2)}_2\end{array}\right)+\left(\begin{array}{cc}b_1 & b_1\\b_2 & b_2\\b_3 & b_3\end{array}\right)=\left(\begin{array}{cc} w_{11}x^{(1)}_1+w_{12}x^{(1)}_2+b_1 & w_{11}x^{(2)}_1+w_{12}x^{(2)}_1+b_1\\w_{21}x^{(1)}_1+w_{22}x^{(1)}_2+b_2 & w_{21}x^{(2)}_1+w_{22}x^{(2)}_2+b_2\\w_{31}x^{(1)}_1+w_{32}x^{(1)}_2+b_3 & w_{31}x^{(2)}_1+w_{32}x^{(2)}_2+b_3\end{array}\right)$$ Note that in python, thanks to *broadcasting*, it is sufficient to use a vector `b1` of dimension $n^{[1]} \times 1$, since in`a1 = np.dot(W1,X)+b1`the vector `b1` will be added to each column of matrix `np.dot(W1,X)`.  Activation functions As an alternative to the sigmoid function, one can use the hyperbolic tangent function $tanh(z) = \frac{e^z-e^{-z}}{e^z+e^{-z}}$ which has zero mean and makes the learning for the next layer easier. In binary classification, the last layer can still use the sigmoid function because $\hat{y} \in (0,1)$.<jupyter_code># the hyperbolic tangent function
import matplotlib.pylab as plt
%matplotlib inline
import numpy as np
x = np.arange(-8, 8, 0.1)
f = np.tanh(x)
plt.plot(x, f)
plt.axvline(x=0, ls = "--", c = 'black')
plt.axhline(y=0, ls = "--",c = 'black')
plt.axhline(y=1, c = 'black')
plt.axhline(y=-1, c = 'black')
plt.xlabel('x')
plt.ylabel('f(x)')
plt.title('Hyperbolic tangent function')
plt.show()<jupyter_output><empty_output><jupyter_text>The problem with the above activation functions is that the derivative for $z$ very large or very small are close to zero in both cases, and this slows down gradient discent. A very common activation funcion is the *Rectified Linear Unit (ReLU)* function $a = \max\{0,z\}$, which has derivative equal to 1 when $z>0$. To avoid the negative derivative, an alternative is the *Leaky ReLU* function $a =  \max\{0.01z,z\}$<jupyter_code># ReLU and Leaky ReLU functions
import matplotlib.pylab as plt
%matplotlib inline
import numpy as np
x = np.arange(-8, 8, 0.1)

plt.figure(1)
plt.subplot(2,2,1)
f = np.maximum(0,x)
plt.plot(x, f)
plt.axvline(x=0, ls = "--", c = 'black')
plt.axhline(y=0, ls = "--",c = 'black')
plt.xlabel('x')
plt.ylabel('f(x)')
plt.title('ReLU function')
plt.subplot(2,2,2)
f = np.maximum(0.1*x,x)
plt.plot(x, f)
plt.axvline(x=0, ls = "--", c = 'black')
plt.axhline(y=0, ls = "--",c = 'black')
plt.xlabel('x')
plt.ylabel('f(x)')
plt.title('Leaky ReLU function')
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.22259313660034719, 'fraction_numerical': 0.02977700627587128, 'mean_word_length': 4.837879968823071, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 20, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '435109', 'license': 'MIT', 'n_tokens_mistral': 3071, 'n_tokens_neox': 2912, 'n_words': 850, 'path': '1 - Neural Networks and Deep Learning/1 - Notes.ipynb', 'repo_name': 'aledesa/Deep-Learning-Specialization', 'size': '7489'}