<jupyter_start><jupyter_text>DSCI 572 "lecture" 5 Lecture outline:- Video recap (15 min)- True/false questions (25 min)- Break (5 min)- Activation functions (5 min)- Deep learing software; Keras (20 min) - Counting parameters (10 min)- Wrap-up<jupyter_code>import numpy as np
import matplotlib.pyplot as plt
from matplotlib import cm
from mpl_toolkits.mplot3d import Axes3D

%matplotlib inline<jupyter_output><empty_output><jupyter_text>Dependencies:- tensorflow: `conda install tensorflow`<jupyter_code>import tensorflow as tf
from tensorflow import keras 
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense
plt.rcParams['font.size'] = 16<jupyter_output><empty_output><jupyter_text>Video recap (10 min)We define the function recursively:$$ x^{(l+1)} = h\left( W^{(l)} x^{(l)} + b^{(l)}\right) $$where $W^{(l)}$ is a matrix of parameters, $b^{(l)}$ is a vector of parameters. So what is $x^{(l)}$? * $x^{(0)}$ are the inputs * $x^{(L)}$ are the outputs, so we can say $\hat{y}=x^{(L)}$ * we refer to $L-1$ as the _number of hidden layers_<jupyter_code>def example_1_layer_nn_predict(x,W,h): # x is a vector, W is a matrix
    return h(W@x)

x = np.random.rand(5)   # d = 5
W = np.random.rand(2,5) # transforming from 5 dimensions to 2 dimensions
h = lambda x: x**2      # just an example, not a typical choice...

example_1_layer_nn_predict(x,W,h)<jupyter_output><empty_output><jupyter_text>Above, the size of the input is 5 and the size of the output is 2.<jupyter_code>def example_nn_predict(x,W,h): # x is a vector, W is a _list of matrices_
    for W_l in W:
        x = h(W_l@x)
    return x

x = np.random.rand(5)
W1 = np.random.rand(2,5)
W2 = np.random.rand(1,2)
h = lambda x: x**2
print(x)
example_nn_predict(x, [W1,W2], h)
hid = h(W1@x)
hid
h(W2@hid)<jupyter_output><empty_output><jupyter_text>Above, the size of the input is 5 and the size of the output is 1. This could now be used for regression, if we could somehow define a loss and find the $W$ matrices that minimize it... Also:  - the $W^{(l)}$ do _not_ need to be square.  - the $x^{(l)}$ for $0<l<L$ are "intermediate states"   - there are called _hidden units_ or _hidden neurons_   - the _values_ of these units are called _activations_ - we often refer to the elements of $W$ as "weights" and the elements of $b$ as "biases" - we might not apply $h$ at the last layer... more details to come. ![](https://upload.wikimedia.org/wikipedia/commons/4/46/Colored_neural_network.svg)In the diagrams above, circles are states and arrows carry weights.Important note: neural nets map from $\mathbb{R}^d\rightarrow \mathbb{R}^k$ for some arbitrary $d$ and $k$. The outputs do not have to be scalars. We will make use of this later!  Vocabulary- deep learning- (artificial) neural net(work)- NN, ANN, CNN- layers- units, neurons, activations- hidden, visible- activation function, nonlinearity- ReLU, sigmoid- backprop(agation)  True/False questions (25 min)1. Neural networks can be used for both regression and classification.2. For (fully connected) neural networks, the number of parameters $\geq$ the number of features.3. Linear regression is a special case of a neural network.4. Neural networks are non-parametric. 1. Any neural network with 3 hidden layers will have more parameters than any neural network with 2 hidden layers.2. With neural networks, we have a potentially large number of **discrete** hyperparamaters. 3. With neural networks, we have a potentially large number of **discrete** parameters. 4. Like linear regression or logistic regression, with neural networks we can interpret each feature's weight value as a measure of the feature's importance.  Break (5 min)  Activation functions (5 min) - $h$ is called the _activation function_.  - Question: why do we need $h$ at all? - Answer: if no $h$, then we are composing a bunch of linear functions, which just leaves us with a linear function. - Insight: if $h$ is nonlinear, then increasing the number of "layers" increases the complexity of the overall function.  In neural networks, we choose $h$ to be an _elementwise_ nonlinear function. i.e.$$h(x)\equiv\left[\begin{array}{c}h(x_1)\\h(x_2)\\ \vdots \\ h(x_d)  \end{array}\right]$$Activation functions tend to be continuous, but are [not always smooth or monotonic](https://arxiv.org/pdf/1710.05941.pdf).<jupyter_code>x = np.linspace(-10,10,1000)
h_sigmoid = lambda x: 1/(1+np.exp(-x))
plt.plot(x,h_sigmoid(x))
plt.ylim(-0.1,1.1);
plt.xlabel('x');
plt.ylabel('h(x)');
plt.title("Sigmoid (aka logistic) activation function");<jupyter_output><empty_output><jupyter_text>Sometimes people also use the hyperbolic tangent. It's basically the same thing but has a range of $(-1,1)$ instead of $(0,1)$.<jupyter_code>plt.plot(x, np.tanh(x));
plt.xlabel('x');
plt.ylabel('h(x)');
plt.title("tanh activation function");<jupyter_output><empty_output><jupyter_text>More recently, people use the ReLU (Rectified Linear Unit)<jupyter_code>h_relu = lambda x: np.maximum(0,x) 
plt.plot(x, h_relu(x));
plt.xlabel('x');
plt.ylabel('h(x)');
plt.title("ReLU activation function");
β = 0.5
plt.plot(x, x / (1+np.exp(-β*x)));
plt.xlabel('x');
plt.ylabel('h(x)');
plt.title("swish activation function");<jupyter_output><empty_output><jupyter_text>- Student question: why would you ever "split" a feature into multiple values? Isn't that redundant?- Answer: no, it can lead to more complex functions. Let's try a regression case with 1 feature:<jupyter_code>x = np.linspace(-1,1,1000)[None]
k = 100
W1 = np.random.randn(k,1)
b1 = np.random.randn()
W2 = np.random.randn(1,k)
b2 = np.random.randn()

hid = h_relu(W1@x+b1)
y = W2 @ hid + b2
plt.plot(np.squeeze(x), np.squeeze(y));
hid<jupyter_output><empty_output><jupyter_text>Deep learning software (20 min)- There's been a lot of software released lately to take care of this for you. - Here is some historical information, for your reference.| Name   |  Host language  | Released |  Comments | Stars on GitHub (Jan 2019) ||--------|-------------|---------------|----------|-----------------|| [Torch](http://torch.ch) | Lua | 2002 | Used at Facebook | 8k || [Theano](http://deeplearning.net/software/theano/) | Python | 2007 | From U. de Montréal, going out of fashion | 9k || [Caffe](http://caffe.berkeleyvision.org) | Executable with Python wrapper | 2014 | Designed for CNNs, by UC Berkeley | 27k| [TensorFlow](https://www.tensorflow.org) | Python | 2015 | Created by Google for both prototyping and production | 118k| [Keras](https://keras.io) | Python | 2015 | A front-end on top of Theano or TensorFlow | 37k| [PyTorch](http://pytorch.org) | Python | 2017 | Automatic differentiation through arbitrary code like Autograd | 24k || [Caffe 2](https://caffe2.ai/) | Python or C++ | 2017 | Facebook, open source, [_merged with PyTorch last year_](https://caffe2.ai/blog/2018/05/02/Caffe2_PyTorch_1_0.html) | repo archived, 8k| [TensorFlow 2.0](https://www.tensorflow.org) | Python | 2019 | Keras incorporated into TensorFlow; other improvements | 118k- The current big players are **TensorFlow** and **PyTorch**.- A lot of people I talk to say PyTorch is easier to use.  - However, I designed this course before PyTorch was created!  - This course uses TensorFlow, specifically tf.keras.   - I may change it someday, or may not, we'll see.  - TensorFlow is extremely popular, and it works for our purposes.- If interested, see [comparison of deep learning software](https://en.wikipedia.org/wiki/Comparison_of_deep_learning_software).- scikit-learn also has its [MLPRegressor](https://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPRegressor.html) and [MLPClassifier](https://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPClassifier.html).   - These were shown in the lecture video.  - They have very limited functionality and generally aren't recommended.  - Even Keras is on the user-friendly-but-less-flexible side.  - Fun fact: these classes were contributed to scikit-learn by a UBC graduate student.<jupyter_code># generate synthetic 1D data
np.random.seed(5)
N = 200
X = np.random.rand(N,1)
y = np.sin(2*X) + np.random.randn(N,1)*0.03

plt.figure()
plt.plot(X,y,'.',markersize=10);
model = Sequential()
model.add(Dense(10, input_dim=1, activation='tanh'))
model.add(Dense(15, activation='tanh'))
model.add(Dense(1, activation='linear'))

# Compile model
# This is where the magic happens!
model.compile(loss='mean_squared_error', optimizer="adam")

# Fit the model
# loss=model.evaluate(X, y,verbose=0)
# print(loss)
model.fit(X, y, epochs=1000, verbose=0)

# evaluate the model
loss = model.evaluate(X, y,verbose=0)
print(loss)<jupyter_output>0.000869073192588985<jupyter_text>Note: in scikit-learn, we use one line of Python code to set up the model and set the hyperparameters. For example:```pythonmodel = SVC(C=5, gamma=0.1)```In Keras, we use multiple lines of Python code to set up the model. For example:```pythonmodel = Sequential()model.add(Dense(10, input_dim=1, activation='tanh', kernel_initializer='lecun_uniform',))model.add(Dense(10, activation='tanh', kernel_initializer='lecun_uniform',))model.add(Dense(1, activation='linear', kernel_initializer='lecun_uniform',))```- This design decision just makes the code more human-readable.- One line of code per layer of the network.- But there's nothing fundamental about it, one could also put the architecture/hypers on one line, in a file, etc.<jupyter_code>plt.plot(X,y,'.',markersize=10,label="data")
grid = np.linspace(0,1,1000)[:,None]
plt.plot(grid, model.predict(grid),linewidth=5,label="model")
plt.legend();
# list(map(lambda x: x.shape, NN.W))<jupyter_output><empty_output><jupyter_text>(Repeat from video) we can also look at regression surfaces with random weights:<jupyter_code># random weights
model = Sequential()
model.add(Dense(50, input_dim=2, activation='tanh', kernel_initializer='lecun_uniform'))
model.add(Dense(1, activation='linear', kernel_initializer='lecun_uniform',))
model.compile(loss='mean_squared_error', optimizer='sgd')

fig = plt.figure()
ax = fig.gca(projection='3d')

n = 100
X = np.linspace(-5, 5, n)
Y = np.linspace(-5, 5, n)
X, Y = np.meshgrid(X, Y)

inputs = np.append(X.flatten()[:,None], Y.flatten()[:,None],axis=1)
outputs = model.predict(inputs)
Z = np.reshape(outputs, [n,n])

# Plot the surface.
surf = ax.plot_surface(X, Y, Z, cmap=cm.coolwarm, linewidth=0)<jupyter_output><empty_output><jupyter_text>Parameters and hyperparameters (20 min)<jupyter_code>model = Sequential()
model.add(Dense(10, input_dim=1, activation='tanh', kernel_initializer='lecun_uniform',))
model.add(Dense(5, activation='tanh', kernel_initializer='lecun_uniform',))
model.add(Dense(1, activation='linear', kernel_initializer='lecun_uniform',))
# if you install graphviz and pydot, the following will "draw" the network architecture
# however I find it more confusing than helpful
# from keras.utils.vis_utils import model_to_dot
# import graphviz
# graphviz.Source(model_to_dot(model, show_shapes=True))
model.summary()<jupyter_output>Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense (Dense)                (None, 10)                20        
_________________________________________________________________
dense_1 (Dense)              (None, 5)                 55        
_________________________________________________________________
dense_2 (Dense)              (None, 1)                 6         
=================================================================
Total params: 81
Trainable params: 81
Non-trainable params: 0
_________________________________________________________________<jupyter_text>Below: we can inspect the weights themselves, and print out their shapes:<jupyter_code>for W in model.get_weights():
    print(W.shape)<jupyter_output>(1, 10)
(10,)
(10, 5)
(5,)
(5, 1)
(1,)<jupyter_text>WARNING: Keras uses the opposite notation that we use. Here, if the shape of $W$ is $1\times 10$ that means transforming from $1$ number to $10$ numbers. In our notation, it's the other way around: if $W$ is $1 \times 10$ that means transforming from $10$ numbers to $1$ number. You'll see both notations out in the wild.- The Keras notation is more intuitive in that the shapes are $(d_\text{in},d_\text{out})$ rather than $(d_\text{out},d_\text{in}$).- Our notation is more convenient because we can write $Wx$ instead of $W^Tx$ everywhere.  - And this also makes it consistent with notation from CPSC 340.  Number of parameters- There's no agreed upon convention for the number of layers.- Let's say, for our purposes, that $x^{(0)}$ is an input and $x^{(L)}$ is an output, which means we have $L+1$ total layers and $L-1$ hidden layers.- Let $d_0,d_1,d_2,\ldots,d_L$ be the dimensionality of the layers.  - So $d_0=d$ (input layer) and $d_L=k$ (output layer). - We get to pick the dimensionality of each of the hidden layers, i.e. $d_1,\ldots,d_{L-1}$; these are hyperparameters.   In general, how many parameters do we have?- Weights: $d_0d_1 + d_1d_2 + \ldots + d_{L-1}d_L$- Biases: $d_1 + d_2 + \ldots + d_L$- Total: $$(d_0+1)d_1 + (d_1+1)d_2 + \ldots + (d_{L-1}+1)d_L$$- Again, this is just bookkeeping. But it's important to realize that  **this is potentially a lot of parameters!!**  - Example: in the 3Blue1Brown video, we had 13,002 parameters.  - Example: let $L=3,d_0=d_1=d_2=d_3=1000$. Then we have millions of parameters! - Sometimes we have billions. - So we have a million-dimensional non-convex optimization problem. That's hard. More on this next class.<jupyter_code>model = Sequential()
model.add(Dense(1000, input_dim=1000)) # d0=1000, d1=1000
model.summary()
model.add(Dense(1000)) # d2=1000
model.summary()
model.add(Dense(1000)) # d3=1000
model.summary()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14189091955677074, 'fraction_numerical': 0.02813354439487696, 'mean_word_length': 4.929607508532423, 'pattern_counts': {'":': 0, '<': 56, '<?xml version=': 0, '>': 54, 'https://': 10, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '487959', 'license': 'MIT', 'n_tokens_mistral': 4798, 'n_tokens_neox': 4447, 'n_words': 1486, 'path': 'data/text/lecture5_neural-networks.ipynb', 'repo_name': 'anasm-17/video_organizer', 'size': '13898'}