<jupyter_start><jupyter_text>Cloning the pyprobml repo<jupyter_code>!git clone https://github.com/probml/pyprobml 
%cd pyprobml/scripts<jupyter_output><empty_output><jupyter_text>Installing required software (This may take few minutes)<jupyter_code>!apt-get install octave  -qq > /dev/null
!apt-get install liboctave-dev -qq > /dev/null
%%capture
%load_ext autoreload 
%autoreload 2



DISCLAIMER = 'WARNING : Editing in VM - changes lost after reboot!!'
from google.colab import files

def interactive_script(script, i=True):
  if i:
    s = open(script).read()
    if not s.split('\n', 1)[0]=="## "+DISCLAIMER:
      open(script, 'w').write(
          f'## {DISCLAIMER}\n' + '#' * (len(DISCLAIMER) + 3) + '\n\n' + s)
    files.view(script)
    %run $script
  else:
      %run $script<jupyter_output><empty_output><jupyter_text>Figure 13.1:   (a) Illustration of the fact that the XOR function is not linearly separable, but can be separated by the two layer model using Heaviside activation functions. Adapted from Figure 10.6 of [Aur19] .  Figure(s) generated by [xor_heaviside.py](https://github.com/probml/pyprobml/blob/master/scripts/xor_heaviside.py)<jupyter_code>interactive_script("xor_heaviside.py")<jupyter_output><empty_output><jupyter_text>Figure 13.2:   (a) Illustration of how the sigmoid function is linear for inputs near 0, but saturates for large positive and negative inputs. Adapted from 11.1 of [Aur19] .  Figure(s) generated by [activationFunPlot.py](https://github.com/probml/pyprobml/blob/master/scripts/activationFunPlot.py) [activationFunPlot.py](https://github.com/probml/pyprobml/blob/master/scripts/activationFunPlot.py)<jupyter_code>interactive_script("activationFunPlot.py")
interactive_script("activationFunPlot.py")<jupyter_output><empty_output><jupyter_text>Figure 13.3:   A simple 2-layer MLP applied to the iris flower classification problem. The nodes in the hidden layers correspond to the hidden units $h_ 1,i $ and $h_ 2,j $, and the edges between them represent the weights $w_ 2,i,j $, where 2 refers to the second layer. (The first layer weight matrix corresponds to the input-to-hidden mapping.) Thus $\mathbf  z _2 = \varphi _2(\mathbf  W _2 \mathbf  z _1 + \mathbf  b _1)$, although we omit the bias terms from the figure for simplicity. From   https://bit.ly/2lDH2Up .   Figure 13.4:   Structure of the MLP used for MNIST classification. Note that $100,480=(784+1) \times 128$, and $16,512 = (128+1) \times 128$.   Figure 13.5:   Results of applying an MLP (with 2 hidden layers with 128 units and 1 output layer with 10 units) to some MNIST images (cherry picked to include some errors). Red is incorrect, blue is correct. (a) After 1 epoch of training. (b) After 2 epochs.   Figure 13.6:   Structure of the MLP used for IMDB classification. We use a vocabulary size of $V=1000$, an embedding size of $E=16$, and a hidden layer of size $16$. The embedding matrix $\mathbf  W _1$ has size $10,000 \times 16$, the hidden layer (labeled ``dense'') has a weight matrix $\mathbf  W _2$ of size $16 \times 16$ and bias $\mathbf  b _2$ of size 16 (note that $16 \times 16 + 16 = 272$), and the final layer (labeled ``dense\_1'') has a weight vector $\mathbf  w _3$ of size $16$ and a bias $b_3$ of size 1. The global average pooling layer has no free parameters.   Figure 13.7:   Illustration of an MLP with a shared ``backbone'' and two output ``heads'', one for predicting the mean and one for predicting the variance. From   https://bit.ly/39bc4XL . Used with kind permission of <NAME>.   Figure 13.8:   Illustration of predictions from an MLP fit using MLE to a 1d regression dataset with growing noise. (a) Output variance is input-dependent, as in \cref  fig:twoHeaded . (b) Mean is computed using same model as in (a), but output variance is treated as a fixed parameter $\sigma ^2$, which is estimated by MLE after training, as in \cref  sec:linregSigmaMLE .   Figure 13.9:   A decomposition of $\mathbb  R ^2$ into a finite set of linear decision regions produced by an MLP with \ensuremath  \mathrm  ReLU  \xspace activations with (a) one hidden layer of 25 hidden units and (b) two hidden layers. From Figure 1 of [MMJ19] . Used with kind permission of <NAME>.   Figure 13.10:   Illustration of two neurons connected together in a ``circuit''. The output axon of the left neuron makes a synaptic connection with the dendrites of the cell on the right Electrical charges, in the form of ion flows, allow the cells to communicate. From   https://en.wikipedia.org/wiki/Neuron . Used with kind permission of Wikipedia author BruceBlaus.   Figure 13.11:   Plot of neural network sizes over time. Models 1, 2, 3 and 4 correspond to the perceptron [Ros58] , the adaptive linear unit [BH60]  the neocognitron [K80] , and the first MLP trained by backprop [RHW86] . From Figure 1.11 of [GBC16] . Used with kind permission of Ian Goodfellow.   Figure 13.12:   A simple linear-chain feedforward model with 4 layers. Here $\mathbf  x $ is the input and $\mathbf  o $ is the output. From [Mat20] .   Figure 13.13:   An example of a computation graph with 2 (scalar) inputs and 1 (scalar) output. From [Mat20] .   Figure 13.14:   Notation for automatic differentiation at node $j$ in a computation graph. From [Mat20] .   Figure 13.15:   Illustration of various learning rate heuristics. (a) Exponential schedule. (b) Piecewise constant schedule. (c) Cosine annealing schedule (also called  \bf stochastic gradient with restarts ) from [Les17],Loshchilov2017 . (d) One-cycle schedule from [Smi18] .   Figure 13.16:   Training loss vs learning rate for a small MLP fit to FashionMNIST using vanilla SGD. (Raw loss in blue, EWMA smoothed version in orange).  Figure(s) generated by [lrfinder.py](https://github.com/probml/pyprobml/blob/master/scripts/lrfinder.py)<jupyter_code>interactive_script("lrfinder.py")<jupyter_output><empty_output><jupyter_text>Figure 13.17:   (a) The sigmoid activation function. (b) Its derivative.  Figure(s) generated by [vanishing_gradients.py](https://github.com/probml/pyprobml/blob/master/scripts/vanishing_gradients.py)<jupyter_code>interactive_script("vanishing_gradients.py")<jupyter_output><empty_output><jupyter_text>Figure 13.18:   (a) The \ensuremath  \mathrm  ReLU  \xspace activation function. (b) Its derivative.  Figure(s) generated by [vanishing_gradients.py](https://github.com/probml/pyprobml/blob/master/scripts/vanishing_gradients.py)<jupyter_code>interactive_script("vanishing_gradients.py")<jupyter_output><empty_output><jupyter_text>Figure 13.19:   (a) Illustration of a residual block. (b) Illustration of why adding residual connections can help when training a very deep model. Adapted from Figure 14.16 of [Aur19] .   Figure 13.20:   The effects of changing the hyperparameters on an MLP with one hidden layer. (a) Random functions sampled from a Gaussian prior with hyperparameters $\alpha _ 1 =5$, $\beta _1=1$, $\alpha _2=1$, $\beta _2=1$. (b) Increasing $\alpha _ 1 $ by factor of 5. (c) Increasing $\beta _ 1 $ by factor of 5. (d) Inreasing $\alpha _ 2 $ by factor of 5.  Figure(s) generated by [mlpPriorsDemo2.m](https://github.com/probml/pmtk3/blob/master/demos/mlpPriorsDemo2.m)<jupyter_code>!octave -W mlpPriorsDemo2.m >> _<jupyter_output><empty_output><jupyter_text>Figure 13.21:   (a) A deep but sparse neural network. The connections are pruned using $\ell _1$ regularization. At each level, nodes numbered 0 are clamped to 1, so their outgoing weights correspond to the offset/bias terms. (b) Predictions made by the model on the training set.  Figure(s) generated by [sparseNnetDemo.m](https://github.com/probml/pmtk3/blob/master/demos/sparseNnetDemo.m)<jupyter_code>!octave -W sparseNnetDemo.m >> _<jupyter_output><empty_output><jupyter_text>Figure 13.22:   Illustration of dropout. (a) A standard neural net with 2 hidden layers. (b) An example of a thinned net produced by applying dropout with $p_0=0.5$. Units that have been dropped out are marked with an x. From Figure 1 of [Nit+14] . Used with kind permission of <NAME>.   Figure 13.23:   (a) xor truth table. (b) Fitting a linear logistic regression classifier using degree 10 polynomial expansion. (c) Same model, but using an RBF kernel with centroids specified by the 4 black crosses.  Figure(s) generated by [logregXorDemo.m](https://github.com/probml/pmtk3/blob/master/demos/logregXorDemo.m)<jupyter_code>!octave -W logregXorDemo.m >> _<jupyter_output><empty_output><jupyter_text>Figure 13.24:   RBF basis in 1d. Left column: fitted function. Middle column: basis functions evaluated on a grid. Right column: design matrix. Top to bottom we show different bandwidths: $\tau =0.1$, $\tau =0.5$, $\tau =50$.  Figure(s) generated by [linregRbfDemo.m](https://github.com/probml/pmtk3/blob/master/demos/linregRbfDemo.m)<jupyter_code>!octave -W linregRbfDemo.m >> _<jupyter_output><empty_output><jupyter_text>Figure 13.25:   (a) Some data from a one-to-many function. (b) The responsibilities of each expert for the input domain. (c) Prediction of each expert. (d) Overeall prediction. Mean is red cross, mode is black square. Adapted from Figures 5.20 and 5.21 of [Bis06] .  Figure(s) generated by [mixexpDemoOneToMany.m](https://github.com/probml/pmtk3/blob/master/demos/mixexpDemoOneToMany.m)<jupyter_code>!octave -W mixexpDemoOneToMany.m >> _<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10190537358167416, 'fraction_numerical': 0.03125669021622779, 'mean_word_length': 4.577910447761194, 'pattern_counts': {'":': 0, '<': 52, '<?xml version=': 0, '>': 64, 'https://': 15, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '416895', 'license': 'MIT', 'n_tokens_mistral': 3211, 'n_tokens_neox': 2890, 'n_words': 1110, 'path': 'notebooks/figures/chapter13_figures.ipynb', 'repo_name': 'AFIT-CSCE623/pyprobml', 'size': '9368'}