<jupyter_start><jupyter_text>Figure 20.1:     An illustration of PCA where we project from 2d to 1d. Circles are the original data points, crosses are the reconstructions. The red star is the data mean.  Figure(s) generated by [pcaDemo2d.py](https://github.com/probml/pyprobml/blob/master/scripts/pcaDemo2d.py)<jupyter_code>#@title Setup { display-mode: "form" }
%%time
# If you run this for the first time it would take ~25/30 seconds
!git clone https://github.com/probml/pyprobml /pyprobml &> /dev/null && git clone https://github.com/Sekhen/colab_powertoys.git &> /dev/null 
!pip3 install nbimporter -qqq 
%cd -q /content/colab_powertoys 
from colab_powertoys.probml_toys import probml_toys as pmlt
%cd -q /content/

pmlt.show_and_run("/pyprobml/scripts/pcaDemo2d.py")<jupyter_output><empty_output><jupyter_text>Figure 20.2:     An illustration of PCA applied to MNIST digits from class 9. Grid points are at the 5, 25, 50, 75, 95 \% quantiles of the data distribution along each dimension. The circled points are the closest projected images to the vertices of the grid. Adapted from Figure 14.23 of [HTF09] .  Figure(s) generated by [pca_digits.py](https://github.com/probml/pyprobml/blob/master/scripts/pca_digits.py)<jupyter_code>#@title Setup { display-mode: "form" }
%%time
# If you run this for the first time it would take ~25/30 seconds
!git clone https://github.com/probml/pyprobml /pyprobml &> /dev/null && git clone https://github.com/Sekhen/colab_powertoys.git &> /dev/null 
!pip3 install nbimporter -qqq 
%cd -q /content/colab_powertoys 
from colab_powertoys.probml_toys import probml_toys as pmlt
%cd -q /content/

pmlt.show_and_run("/pyprobml/scripts/pca_digits.py")<jupyter_output><empty_output><jupyter_text>Figure 20.3:     a) Some randomly chosen $64 \times 64$ pixel images from the Olivetti face database. (b) The mean and the first three PCA components represented as images.  Figure(s) generated by [pcaImageDemo.m](https://github.com/probml/pmtk3/blob/master/demos/pcaImageDemo.m)<jupyter_code>#@title Setup { display-mode: "form" }
%%time
# If you run this for the first time it would take ~25/30 seconds
!git clone https://github.com/probml/pyprobml /pyprobml &> /dev/null && git clone https://github.com/Sekhen/colab_powertoys.git &> /dev/null 
!pip3 install nbimporter -qqq 
%cd -q /content/colab_powertoys 
from colab_powertoys.probml_toys import probml_toys as pmlt
%cd -q /content/

pmlt.show_image("/pyprobml/notebooks/figures/images/pcaImages-faces-images.png")
pmlt.show_image("/pyprobml/notebooks/figures/images/pcaImages-faces-basis.png")<jupyter_output><empty_output><jupyter_text>Figure 20.4:     Illustration of the variance of the points projected onto different 1d vectors. $v_1$ is the first principal component, which maximizes the variance of the projection. $v_2$ is the second principal component which is direction orthogonal to $v_1$. Finally $v'$ is some other vector in between $v_1$ and $v_2$. Adapted from Figure 8.7 of [Aur19] .  Figure(s) generated by [pca_projected_variance.py](https://github.com/probml/pyprobml/blob/master/scripts/pca_projected_variance.py)<jupyter_code>#@title Setup { display-mode: "form" }
%%time
# If you run this for the first time it would take ~25/30 seconds
!git clone https://github.com/probml/pyprobml /pyprobml &> /dev/null && git clone https://github.com/Sekhen/colab_powertoys.git &> /dev/null 
!pip3 install nbimporter -qqq 
%cd -q /content/colab_powertoys 
from colab_powertoys.probml_toys import probml_toys as pmlt
%cd -q /content/

pmlt.show_and_run("/pyprobml/scripts/pca_projected_variance.py")<jupyter_output><empty_output><jupyter_text>Figure 20.5:     Effect of standardization on PCA applied to the height/weight dataset. (Red=female, blue=male.) Left: PCA of raw data. Right: PCA of standardized data.  Figure(s) generated by [pcaStandardization.py](https://github.com/probml/pyprobml/blob/master/scripts/pcaStandardization.py)<jupyter_code>#@title Setup { display-mode: "form" }
%%time
# If you run this for the first time it would take ~25/30 seconds
!git clone https://github.com/probml/pyprobml /pyprobml &> /dev/null && git clone https://github.com/Sekhen/colab_powertoys.git &> /dev/null 
!pip3 install nbimporter -qqq 
%cd -q /content/colab_powertoys 
from colab_powertoys.probml_toys import probml_toys as pmlt
%cd -q /content/

pmlt.show_and_run("/pyprobml/scripts/pcaStandardization.py")<jupyter_output><empty_output><jupyter_text>Figure 20.6:     Reconstruction error on MNIST vs number of latent dimensions used by PCA. (a) Training set. (b) Test set.  Figure(s) generated by [pcaOverfitDemo.m](https://github.com/probml/pmtk3/blob/master/demos/pcaOverfitDemo.m)<jupyter_code>#@title Setup { display-mode: "form" }
%%time
# If you run this for the first time it would take ~25/30 seconds
!git clone https://github.com/probml/pyprobml /pyprobml &> /dev/null && git clone https://github.com/Sekhen/colab_powertoys.git &> /dev/null 
!pip3 install nbimporter -qqq 
%cd -q /content/colab_powertoys 
from colab_powertoys.probml_toys import probml_toys as pmlt
%cd -q /content/

pmlt.show_image("/pyprobml/notebooks/figures/images/pcaOverfitReconTrain.png")
pmlt.show_image("/pyprobml/notebooks/figures/images/pcaOverfitReconTest.png")<jupyter_output><empty_output><jupyter_text>Figure 20.7:     (a) Scree plot for training set, corresponding to \cref  fig:pcaErr (a). (b) Fraction of variance explained.  Figure(s) generated by [pcaOverfitDemo.m](https://github.com/probml/pmtk3/blob/master/demos/pcaOverfitDemo.m)<jupyter_code>#@title Setup { display-mode: "form" }
%%time
# If you run this for the first time it would take ~25/30 seconds
!git clone https://github.com/probml/pyprobml /pyprobml &> /dev/null && git clone https://github.com/Sekhen/colab_powertoys.git &> /dev/null 
!pip3 install nbimporter -qqq 
%cd -q /content/colab_powertoys 
from colab_powertoys.probml_toys import probml_toys as pmlt
%cd -q /content/

pmlt.show_image("/pyprobml/notebooks/figures/images/pcaOverfitScree.png")
pmlt.show_image("/pyprobml/notebooks/figures/images/pcaOverfitVar.png")<jupyter_output><empty_output><jupyter_text>Figure 20.8:     Profile likelihood corresponding to PCA model in \cref  fig:pcaErr (a).  Figure(s) generated by [pcaOverfitDemo.m](https://github.com/probml/pmtk3/blob/master/demos/pcaOverfitDemo.m)<jupyter_code>#@title Setup { display-mode: "form" }
%%time
# If you run this for the first time it would take ~25/30 seconds
!git clone https://github.com/probml/pyprobml /pyprobml &> /dev/null && git clone https://github.com/Sekhen/colab_powertoys.git &> /dev/null 
!pip3 install nbimporter -qqq 
%cd -q /content/colab_powertoys 
from colab_powertoys.probml_toys import probml_toys as pmlt
%cd -q /content/

pmlt.show_image("/pyprobml/notebooks/figures/images/pcaOverfitProfile.png")<jupyter_output><empty_output><jupyter_text>Figure 20.9:     Illustration of the FA generative process, where we have $L=1$ latent dimension generating $D=2$ observed dimensions; we assume $\boldsymbol  \Psi  =\sigma ^2 \mathbf  I $. The latent factor has value $z \in \mathbb  R $, sampled from $p(z)$; this gets mapped to a 2d offset $\boldsymbol  \delta  = z \mathbf  w $, where $\mathbf  w \in \mathbb  R ^2$, which gets added to $\boldsymbol  \mu  $ to define a Gaussian $p(\mathbf  x |z) = \mathcal  N (\mathbf  x |\boldsymbol  \mu  + \boldsymbol  \delta  ,\sigma ^2 \mathbf  I )$. By integrating over $z$, we ``slide'' this circular Gaussian ``spray can'' along the principal component axis $\mathbf  w $, which induces elliptical Gaussian contours in $\mathbf  x $ space centered on $\boldsymbol  \mu  $. Adapted from Figure 12.9 of [Bis06] .<jupyter_code>#@title Setup { display-mode: "form" }
%%time
# If you run this for the first time it would take ~25/30 seconds
!git clone https://github.com/probml/pyprobml /pyprobml &> /dev/null && git clone https://github.com/Sekhen/colab_powertoys.git &> /dev/null 
!pip3 install nbimporter -qqq 
%cd -q /content/colab_powertoys 
from colab_powertoys.probml_toys import probml_toys as pmlt
%cd -q /content/

pmlt.show_image("/pyprobml/notebooks/figures/images/PPCAsprayCan.png")<jupyter_output><empty_output><jupyter_text>Figure 20.10:     Illustration of EM for PCA when $D=2$ and $L=1$. Green stars are the original data points, black circles are their reconstructions. The weight vector $\mathbf  w $ is represented by blue line. (a) We start with a random initial guess of $\mathbf  w $. The E step is represented by the orthogonal projections. (b) We update the rod $\mathbf  w $ in the M step, keeping the projections onto the rod (black circles) fixed. (c) Another E step. The black circles can 'slide' along the rod, but the rod stays fixed. (d) Another M step. Adapted from Figure 12.12 of [Bis06] .  Figure(s) generated by [pcaEmStepByStep.m](https://github.com/probml/pmtk3/blob/master/demos/pcaEmStepByStep.m)<jupyter_code>#@title Setup { display-mode: "form" }
%%time
# If you run this for the first time it would take ~25/30 seconds
!git clone https://github.com/probml/pyprobml /pyprobml &> /dev/null && git clone https://github.com/Sekhen/colab_powertoys.git &> /dev/null 
!pip3 install nbimporter -qqq 
%cd -q /content/colab_powertoys 
from colab_powertoys.probml_toys import probml_toys as pmlt
%cd -q /content/

pmlt.show_image("/pyprobml/notebooks/figures/images/pcaEmStepByStepEstep1.png")
pmlt.show_image("/pyprobml/notebooks/figures/images/pcaEmStepByStepMstep1.png")
pmlt.show_image("/pyprobml/notebooks/figures/images/pcaEmStepByStepEstep2.png")
pmlt.show_image("/pyprobml/notebooks/figures/images/pcaEmStepByStepMstep2.png")<jupyter_output><empty_output><jupyter_text>Figure 20.11:     Mixture of factor analyzers as a PGM.<jupyter_code>#@title Setup { display-mode: "form" }
%%time
# If you run this for the first time it would take ~25/30 seconds
!git clone https://github.com/probml/pyprobml /pyprobml &> /dev/null && git clone https://github.com/Sekhen/colab_powertoys.git &> /dev/null 
!pip3 install nbimporter -qqq 
%cd -q /content/colab_powertoys 
from colab_powertoys.probml_toys import probml_toys as pmlt
%cd -q /content/

pmlt.show_image("/pyprobml/notebooks/figures/images/mixFAdgmC.png")<jupyter_output><empty_output><jupyter_text>Figure 20.12:     Mixture of PPCA models fit to a 2d dataset, using $L=1$ latent dimensions and $K=1$ and $K=10$ mixture components.  Figure(s) generated by [mixPpcaDemoNetlab.m](https://github.com/probml/pmtk3/blob/master/demos/mixPpcaDemoNetlab.m)<jupyter_code>#@title Setup { display-mode: "form" }
%%time
# If you run this for the first time it would take ~25/30 seconds
!git clone https://github.com/probml/pyprobml /pyprobml &> /dev/null && git clone https://github.com/Sekhen/colab_powertoys.git &> /dev/null 
!pip3 install nbimporter -qqq 
%cd -q /content/colab_powertoys 
from colab_powertoys.probml_toys import probml_toys as pmlt
%cd -q /content/

pmlt.show_image("/pyprobml/notebooks/figures/images/mixPpcaAnnulus1.png")
pmlt.show_image("/pyprobml/notebooks/figures/images/mixPpcaAnnulus10.png")<jupyter_output><empty_output><jupyter_text>Figure 20.13:     Random samples from the MixFA model fit to CelebA. From Figure 4 of [EY18] . Used with kind permission of <NAME><jupyter_code>#@title Setup { display-mode: "form" }
%%time
# If you run this for the first time it would take ~25/30 seconds
!git clone https://github.com/probml/pyprobml /pyprobml &> /dev/null && git clone https://github.com/Sekhen/colab_powertoys.git &> /dev/null 
!pip3 install nbimporter -qqq 
%cd -q /content/colab_powertoys 
from colab_powertoys.probml_toys import probml_toys as pmlt
%cd -q /content/

pmlt.show_image("/pyprobml/notebooks/figures/images/MFAGAN-samples.png")<jupyter_output><empty_output><jupyter_text>Figure 20.14:     (a) 150 synthetic 16 dimensional bit vectors. (b) The 2d embedding learned by binary PCA, fit using variational EM. We have color coded points by the identity of the true ``prototype'' that generated them. (c) Predicted probability of being on. (d) Thresholded predictions.  Figure(s) generated by [binaryFaDemoTipping.m](https://github.com/probml/pmtk3/blob/master/demos/binaryFaDemoTipping.m)<jupyter_code>#@title Setup { display-mode: "form" }
%%time
# If you run this for the first time it would take ~25/30 seconds
!git clone https://github.com/probml/pyprobml /pyprobml &> /dev/null && git clone https://github.com/Sekhen/colab_powertoys.git &> /dev/null 
!pip3 install nbimporter -qqq 
%cd -q /content/colab_powertoys 
from colab_powertoys.probml_toys import probml_toys as pmlt
%cd -q /content/

pmlt.show_image("/pyprobml/notebooks/figures/images/binaryPCAinput.png")
pmlt.show_image("/pyprobml/notebooks/figures/images/binaryPCAembedding.png")
pmlt.show_image("/pyprobml/notebooks/figures/images/binaryPCApostpred.png")
pmlt.show_image("/pyprobml/notebooks/figures/images/binaryPCArecon.png")<jupyter_output><empty_output><jupyter_text>Figure 20.15:     Gaussian latent factor models for paired data. (a) Supervised PCA. (b) Partial least squares.<jupyter_code>#@title Setup { display-mode: "form" }
%%time
# If you run this for the first time it would take ~25/30 seconds
!git clone https://github.com/probml/pyprobml /pyprobml &> /dev/null && git clone https://github.com/Sekhen/colab_powertoys.git &> /dev/null 
!pip3 install nbimporter -qqq 
%cd -q /content/colab_powertoys 
from colab_powertoys.probml_toys import probml_toys as pmlt
%cd -q /content/

pmlt.show_image("/pyprobml/notebooks/figures/images/eSPCAxy.png")
pmlt.show_image("/pyprobml/notebooks/figures/images/ePLSxy.png")<jupyter_output><empty_output><jupyter_text>Figure 20.16:     Canonical correlation analysis as a PGM.<jupyter_code>#@title Setup { display-mode: "form" }
%%time
# If you run this for the first time it would take ~25/30 seconds
!git clone https://github.com/probml/pyprobml /pyprobml &> /dev/null && git clone https://github.com/Sekhen/colab_powertoys.git &> /dev/null 
!pip3 install nbimporter -qqq 
%cd -q /content/colab_powertoys 
from colab_powertoys.probml_toys import probml_toys as pmlt
%cd -q /content/

pmlt.show_image("/pyprobml/notebooks/figures/images/eCCAxy.png")<jupyter_output><empty_output><jupyter_text>Figure 20.17:     An autoencoder with one hidden layer.<jupyter_code>#@title Setup { display-mode: "form" }
%%time
# If you run this for the first time it would take ~25/30 seconds
!git clone https://github.com/probml/pyprobml /pyprobml &> /dev/null && git clone https://github.com/Sekhen/colab_powertoys.git &> /dev/null 
!pip3 install nbimporter -qqq 
%cd -q /content/colab_powertoys 
from colab_powertoys.probml_toys import probml_toys as pmlt
%cd -q /content/

pmlt.show_image("/pyprobml/notebooks/figures/images/autoencoder.png")<jupyter_output><empty_output><jupyter_text>Figure 20.18:     Results of applying an autoencoder to the Fashion MNIST data. Top row are first 5 images from validation set. Bottom row are reconstructions. (a) MLP model (trained for 20 epochs). The encoder is an MLP with architecture 784-100-30. The decoder is the mirror image of this. (b) CNN model (trained for 5 epochs). The encoder is a CNN model with architecture Conv2D(16, 3x3, same, selu), MaxPool2D(2x2), Conv2D(32, 3x3, same, selu), MaxPool2D(2x2), Conv2D(64, 3x3, same, selu), MaxPool2D(2x2). The decoder is the mirror image of this, using transposed convolution and without the max pooling layers. Adapted from Figure 17.4 of [Aur19] .  To reproduce this figure, click the open in colab button:<jupyter_code>#@title Setup { display-mode: "form" }
%%time
# If you run this for the first time it would take ~25/30 seconds
!git clone https://github.com/probml/pyprobml /pyprobml &> /dev/null && git clone https://github.com/Sekhen/colab_powertoys.git &> /dev/null 
!pip3 install nbimporter -qqq 
%cd -q /content/colab_powertoys 
from colab_powertoys.probml_toys import probml_toys as pmlt
%cd -q /content/

pmlt.show_image("/pyprobml/notebooks/figures/images/ae_fashion_mlp_recon.png")
pmlt.show_image("/pyprobml/notebooks/figures/images/ae_fashion_cnn_recon.png")<jupyter_output><empty_output><jupyter_text>Figure 20.19:     tSNE plot of the first 2 latent dimensions of the Fashion MNIST validation set computed using an MLP-based autoencoder. Adapted from Figure 17.5 of [Aur19] .  To reproduce this figure, click the open in colab button:<jupyter_code>#@title Setup { display-mode: "form" }
%%time
# If you run this for the first time it would take ~25/30 seconds
!git clone https://github.com/probml/pyprobml /pyprobml &> /dev/null && git clone https://github.com/Sekhen/colab_powertoys.git &> /dev/null 
!pip3 install nbimporter -qqq 
%cd -q /content/colab_powertoys 
from colab_powertoys.probml_toys import probml_toys as pmlt
%cd -q /content/

pmlt.show_image("/pyprobml/notebooks/figures/images/ae-mlp-fashion-tsne.png")<jupyter_output><empty_output><jupyter_text>Figure 20.20:     Denoising autoencoder (MLP architecture) applied to some noisy Fashion MNIST images from the validation set. (a) Gaussian noise. (b) Bernoulli dropout noise. Top row: input. Bottom row: output Adapted from Figure 17.9 of [Aur19] .  To reproduce this figure, click the open in colab button:<jupyter_code>#@title Setup { display-mode: "form" }
%%time
# If you run this for the first time it would take ~25/30 seconds
!git clone https://github.com/probml/pyprobml /pyprobml &> /dev/null && git clone https://github.com/Sekhen/colab_powertoys.git &> /dev/null 
!pip3 install nbimporter -qqq 
%cd -q /content/colab_powertoys 
from colab_powertoys.probml_toys import probml_toys as pmlt
%cd -q /content/

pmlt.show_image("/pyprobml/notebooks/figures/images/ae-denoising-gaussian.png")
pmlt.show_image("/pyprobml/notebooks/figures/images/ae-denoising-dropout.png")<jupyter_output><empty_output><jupyter_text>Figure 20.21:     The residual error from a DAE, $\mathbf  e (\mathbf  x )=r( \cc@accent  "707E  \mathbf  x   )-\mathbf  x $, can learn a vector field corresponding to the score function. Arrows point towards higher probability regions. The length of the arrow is proportional to $||\mathbf  e (\mathbf  x )||$, so points near the 1d data manifold (represented by the curved line) have smaller arrows. From Figure 5 of [GY14] . Used with kind permission of <NAME>.<jupyter_code>#@title Setup { display-mode: "form" }
%%time
# If you run this for the first time it would take ~25/30 seconds
!git clone https://github.com/probml/pyprobml /pyprobml &> /dev/null && git clone https://github.com/Sekhen/colab_powertoys.git &> /dev/null 
!pip3 install nbimporter -qqq 
%cd -q /content/colab_powertoys 
from colab_powertoys.probml_toys import probml_toys as pmlt
%cd -q /content/

pmlt.show_image("/pyprobml/notebooks/figures/images/DAE.png")<jupyter_output><empty_output><jupyter_text>Figure 20.22:     Neuron activity (in the bottleneck layer) for an autoencoder applied to Fashion MNIST. We show results for three models, with different kinds of sparsity penalty: no penalty (left column), $\ell _1$ penalty (middle column), KL penalty (right column). Top row: Heatmap of 300 neuron activations (columns) across 100 examples (rows). Middle row: Histogram of activation levels derived from this heatmap. Bottom row: Histogram of the mean activation per neuron, averaged over all examples in the validation set. Adapted from Figure 17.11 of [Aur19] .  To reproduce this figure, click the open in colab button:<jupyter_code>#@title Setup { display-mode: "form" }
%%time
# If you run this for the first time it would take ~25/30 seconds
!git clone https://github.com/probml/pyprobml /pyprobml &> /dev/null && git clone https://github.com/Sekhen/colab_powertoys.git &> /dev/null 
!pip3 install nbimporter -qqq 
%cd -q /content/colab_powertoys 
from colab_powertoys.probml_toys import probml_toys as pmlt
%cd -q /content/

pmlt.show_image("/pyprobml/notebooks/figures/images/ae-sparse-noreg-heatmap.png")
pmlt.show_image("/pyprobml/notebooks/figures/images/ae-sparse-L1reg-heatmap.png")
pmlt.show_image("/pyprobml/notebooks/figures/images/ae-sparse-KLreg-heatmap.png")
pmlt.show_image("/pyprobml/notebooks/figures/images/ae-sparse-noreg-act.png")
pmlt.show_image("/pyprobml/notebooks/figures/images/ae-sparse-L1reg-act.png")
pmlt.show_image("/pyprobml/notebooks/figures/images/ae-sparse-KLreg-act.png")
pmlt.show_image("/pyprobml/notebooks/figures/images/ae-sparse-noreg-neurons.png")
pmlt.show_image("/pyprobml/notebooks/figures/images/ae-sparse-L1reg-neurons.png")
pmlt.show_image("/pyprobml/notebooks/figures/images/ae-sparse-KLreg-neurons.png")<jupyter_output><empty_output><jupyter_text>Figure 20.23:     Schematic illustration of a VAE. From a figure from   http://krasserm.github.io/2018/07/27/dfc-vae/ . Used with kind permission of <NAME>.<jupyter_code>#@title Setup { display-mode: "form" }
%%time
# If you run this for the first time it would take ~25/30 seconds
!git clone https://github.com/probml/pyprobml /pyprobml &> /dev/null && git clone https://github.com/Sekhen/colab_powertoys.git &> /dev/null 
!pip3 install nbimporter -qqq 
%cd -q /content/colab_powertoys 
from colab_powertoys.probml_toys import probml_toys as pmlt
%cd -q /content/

pmlt.show_image("/pyprobml/notebooks/figures/images/vae-krasser.png")<jupyter_output><empty_output><jupyter_text>Figure 20.24:     Comparison of reconstruction abilities of an autoencoder and VAE. Top row: Original images. Middle row: Reconstructions from a VAE. Bottom row: Reconstructions from an AE. We see that the VAE reconstructions (middle) are blurrier. Both models have the same shallow convolutional architecture (3 hidden layers, 200 latents), and are trained on identical data (20k images of size $64 \times 64$ extracted from CelebA) for the same number of epochs (20).  To reproduce this figure, click the open in colab button:<jupyter_code>#@title Setup { display-mode: "form" }
%%time
# If you run this for the first time it would take ~25/30 seconds
!git clone https://github.com/probml/pyprobml /pyprobml &> /dev/null && git clone https://github.com/Sekhen/colab_powertoys.git &> /dev/null 
!pip3 install nbimporter -qqq 
%cd -q /content/colab_powertoys 
from colab_powertoys.probml_toys import probml_toys as pmlt
%cd -q /content/

pmlt.show_image("/pyprobml/notebooks/figures/images/ae-celeba-orig.png")
pmlt.show_image("/pyprobml/notebooks/figures/images/vae-celeba-recon.png")
pmlt.show_image("/pyprobml/notebooks/figures/images/ae-celeba-recon.png")<jupyter_output><empty_output><jupyter_text>Figure 20.25:     Unconditional samples from a VAE (top row) or AE (bottom row) trained on CelebA. Both models have the same structure and both are trained for 20 epochs.  To reproduce this figure, click the open in colab button:<jupyter_code>#@title Setup { display-mode: "form" }
%%time
# If you run this for the first time it would take ~25/30 seconds
!git clone https://github.com/probml/pyprobml /pyprobml &> /dev/null && git clone https://github.com/Sekhen/colab_powertoys.git &> /dev/null 
!pip3 install nbimporter -qqq 
%cd -q /content/colab_powertoys 
from colab_powertoys.probml_toys import probml_toys as pmlt
%cd -q /content/

pmlt.show_image("/pyprobml/notebooks/figures/images/vae-celeba-samples.png")
pmlt.show_image("/pyprobml/notebooks/figures/images/ae-celeba-samples.png")<jupyter_output><empty_output><jupyter_text>Figure 20.26:     Interpolation between two real images (first and last columns) in the latent space of a VAE. Adapted from Figure 3.22 of [Dav19] .  To reproduce this figure, click the open in colab button:<jupyter_code>#@title Setup { display-mode: "form" }
%%time
# If you run this for the first time it would take ~25/30 seconds
!git clone https://github.com/probml/pyprobml /pyprobml &> /dev/null && git clone https://github.com/Sekhen/colab_powertoys.git &> /dev/null 
!pip3 install nbimporter -qqq 
%cd -q /content/colab_powertoys 
from colab_powertoys.probml_toys import probml_toys as pmlt
%cd -q /content/

pmlt.show_image("/pyprobml/notebooks/figures/images/vae-celeba-interp-gender.png")<jupyter_output><empty_output><jupyter_text>Figure 20.27:     Adding or removing the ``sunglasses'' vector to an image using a VAE. The first column is an input image, with embedding $\mathbf  z $. Subsequent columns show the decoding of $\mathbf  z + s \boldsymbol  \Delta  $, where $s \in \ -4,-3,-2,-1,0,1,2,3,4\ $ and $\boldsymbol  \Delta  = \overline  \mathbf  z  ^+ - \overline  \mathbf  z  ^-$ is the difference in the average embeddings of images of people with or without sunglasses. Adapted from Figure 3.21 of [Dav19] .  To reproduce this figure, click the open in colab button:<jupyter_code>#@title Setup { display-mode: "form" }
%%time
# If you run this for the first time it would take ~25/30 seconds
!git clone https://github.com/probml/pyprobml /pyprobml &> /dev/null && git clone https://github.com/Sekhen/colab_powertoys.git &> /dev/null 
!pip3 install nbimporter -qqq 
%cd -q /content/colab_powertoys 
from colab_powertoys.probml_toys import probml_toys as pmlt
%cd -q /content/

pmlt.show_image("/pyprobml/notebooks/figures/images/vae-celeba-glasses-scale.png")<jupyter_output><empty_output><jupyter_text>Figure 20.28:     Illustration of the tangent space and tangent vectors at two different points on a 2d curved manifold. From Figure 1 of [MM+17] . Used with kind permission of <NAME><jupyter_code>#@title Setup { display-mode: "form" }
%%time
# If you run this for the first time it would take ~25/30 seconds
!git clone https://github.com/probml/pyprobml /pyprobml &> /dev/null && git clone https://github.com/Sekhen/colab_powertoys.git &> /dev/null 
!pip3 install nbimporter -qqq 
%cd -q /content/colab_powertoys 
from colab_powertoys.probml_toys import probml_toys as pmlt
%cd -q /content/

pmlt.show_image("/pyprobml/notebooks/figures/images/tangentSpace.png")<jupyter_output><empty_output><jupyter_text>Figure 20.29:     Illustration of the image manifold. (a) An image of the digit 6 from the USPS dataset, of size $64 \times 57 = 3,648$. (b) A random sample from the space $\ 0,1\ ^ 3648 $ reshaped as an image. (c) A dataset created by rotating the original image by one degree 360 times. We project this data onto its first two principal components, to reveal the underlying 2d circular manifold. From Figure 1 of [Nei12] . Used with kind permission of <NAME><jupyter_code>#@title Setup { display-mode: "form" }
%%time
# If you run this for the first time it would take ~25/30 seconds
!git clone https://github.com/probml/pyprobml /pyprobml &> /dev/null && git clone https://github.com/Sekhen/colab_powertoys.git &> /dev/null 
!pip3 install nbimporter -qqq 
%cd -q /content/colab_powertoys 
from colab_powertoys.probml_toys import probml_toys as pmlt
%cd -q /content/

pmlt.show_image("/pyprobml/notebooks/figures/images/manifold-6-original.png")
pmlt.show_image("/pyprobml/notebooks/figures/images/manifold-6-rnd.png")
pmlt.show_image("/pyprobml/notebooks/figures/images/manifold-6-rotated.png")<jupyter_output><empty_output><jupyter_text>Figure 20.30:     Illustration of some data generated from low-dimensional manifolds. (a) The 2d Swiss-roll manifold embedded into 3d.  Figure(s) generated by [manifold_swiss_sklearn.py](https://github.com/probml/pyprobml/blob/master/scripts/manifold_swiss_sklearn.py) [manifold_digits_sklearn.py](https://github.com/probml/pyprobml/blob/master/scripts/manifold_digits_sklearn.py)<jupyter_code>#@title Setup { display-mode: "form" }
%%time
# If you run this for the first time it would take ~25/30 seconds
!git clone https://github.com/probml/pyprobml /pyprobml &> /dev/null && git clone https://github.com/Sekhen/colab_powertoys.git &> /dev/null 
!pip3 install nbimporter -qqq 
%cd -q /content/colab_powertoys 
from colab_powertoys.probml_toys import probml_toys as pmlt
%cd -q /content/

pmlt.show_and_run("/pyprobml/scripts/manifold_swiss_sklearn.py")
pmlt.show_and_run("/pyprobml/scripts/manifold_digits_sklearn.py")<jupyter_output><empty_output><jupyter_text>Figure 20.31:     Metric MDS applied to (a) Swiss roll.  Figure(s) generated by [manifold_swiss_sklearn.py](https://github.com/probml/pyprobml/blob/master/scripts/manifold_swiss_sklearn.py) [manifold_digits_sklearn.py](https://github.com/probml/pyprobml/blob/master/scripts/manifold_digits_sklearn.py)<jupyter_code>#@title Setup { display-mode: "form" }
%%time
# If you run this for the first time it would take ~25/30 seconds
!git clone https://github.com/probml/pyprobml /pyprobml &> /dev/null && git clone https://github.com/Sekhen/colab_powertoys.git &> /dev/null 
!pip3 install nbimporter -qqq 
%cd -q /content/colab_powertoys 
from colab_powertoys.probml_toys import probml_toys as pmlt
%cd -q /content/

pmlt.show_and_run("/pyprobml/scripts/manifold_swiss_sklearn.py")
pmlt.show_and_run("/pyprobml/scripts/manifold_digits_sklearn.py")<jupyter_output><empty_output><jupyter_text>Figure 20.32:     (a) If we measure distances along the manifold, we find $d(1,6) > d(1,4)$, whereas if we measure in ambient space, we find $d(1,6) [Hin13] . Used with kind permission of <NAME>.<jupyter_code>#@title Setup { display-mode: "form" }
%%time
# If you run this for the first time it would take ~25/30 seconds
!git clone https://github.com/probml/pyprobml /pyprobml &> /dev/null && git clone https://github.com/Sekhen/colab_powertoys.git &> /dev/null 
!pip3 install nbimporter -qqq 
%cd -q /content/colab_powertoys 
from colab_powertoys.probml_toys import probml_toys as pmlt
%cd -q /content/

pmlt.show_image("/pyprobml/notebooks/figures/images/hinton-isomap1.png")
pmlt.show_image("/pyprobml/notebooks/figures/images/hinton-isomap2.png")<jupyter_output><empty_output><jupyter_text>Figure 20.33:     Isomap applied to (a) Swiss roll.  Figure(s) generated by [manifold_swiss_sklearn.py](https://github.com/probml/pyprobml/blob/master/scripts/manifold_swiss_sklearn.py) [manifold_digits_sklearn.py](https://github.com/probml/pyprobml/blob/master/scripts/manifold_digits_sklearn.py)<jupyter_code>#@title Setup { display-mode: "form" }
%%time
# If you run this for the first time it would take ~25/30 seconds
!git clone https://github.com/probml/pyprobml /pyprobml &> /dev/null && git clone https://github.com/Sekhen/colab_powertoys.git &> /dev/null 
!pip3 install nbimporter -qqq 
%cd -q /content/colab_powertoys 
from colab_powertoys.probml_toys import probml_toys as pmlt
%cd -q /content/

pmlt.show_and_run("/pyprobml/scripts/manifold_swiss_sklearn.py")
pmlt.show_and_run("/pyprobml/scripts/manifold_digits_sklearn.py")<jupyter_output><empty_output><jupyter_text>Figure 20.34:     (a) Noisy version of Swiss roll data. We perturb each point by adding $\mathcal  N (0, 0.5^2)$ noise. (b) Results of Isomap applied to this data.  Figure(s) generated by [manifold_swiss_sklearn.py](https://github.com/probml/pyprobml/blob/master/scripts/manifold_swiss_sklearn.py)<jupyter_code>#@title Setup { display-mode: "form" }
%%time
# If you run this for the first time it would take ~25/30 seconds
!git clone https://github.com/probml/pyprobml /pyprobml &> /dev/null && git clone https://github.com/Sekhen/colab_powertoys.git &> /dev/null 
!pip3 install nbimporter -qqq 
%cd -q /content/colab_powertoys 
from colab_powertoys.probml_toys import probml_toys as pmlt
%cd -q /content/

pmlt.show_and_run("/pyprobml/scripts/manifold_swiss_sklearn.py")<jupyter_output><empty_output><jupyter_text>Figure 20.35:     Visualization of the first 8 kernel principal component basis functions derived from some 2d data. We use an RBF kernel with $\sigma ^2=0.1$.  Figure(s) generated by [kpcaScholkopf.m](https://github.com/probml/pmtk3/blob/master/demos/kpcaScholkopf.m)<jupyter_code>#@title Setup { display-mode: "form" }
%%time
# If you run this for the first time it would take ~25/30 seconds
!git clone https://github.com/probml/pyprobml /pyprobml &> /dev/null && git clone https://github.com/Sekhen/colab_powertoys.git &> /dev/null 
!pip3 install nbimporter -qqq 
%cd -q /content/colab_powertoys 
from colab_powertoys.probml_toys import probml_toys as pmlt
%cd -q /content/

pmlt.show_image("/pyprobml/notebooks/figures/images/kpcaScholkopfNoShade.png")<jupyter_output><empty_output><jupyter_text>Figure 20.36:     Kernel PCA applied to (a) Swiss roll.  Figure(s) generated by [manifold_swiss_sklearn.py](https://github.com/probml/pyprobml/blob/master/scripts/manifold_swiss_sklearn.py) [manifold_digits_sklearn.py](https://github.com/probml/pyprobml/blob/master/scripts/manifold_digits_sklearn.py)<jupyter_code>#@title Setup { display-mode: "form" }
%%time
# If you run this for the first time it would take ~25/30 seconds
!git clone https://github.com/probml/pyprobml /pyprobml &> /dev/null && git clone https://github.com/Sekhen/colab_powertoys.git &> /dev/null 
!pip3 install nbimporter -qqq 
%cd -q /content/colab_powertoys 
from colab_powertoys.probml_toys import probml_toys as pmlt
%cd -q /content/

pmlt.show_and_run("/pyprobml/scripts/manifold_swiss_sklearn.py")
pmlt.show_and_run("/pyprobml/scripts/manifold_digits_sklearn.py")<jupyter_output><empty_output><jupyter_text>Figure 20.37:     LLE applied to (a) Swiss roll.  Figure(s) generated by [manifold_swiss_sklearn.py](https://github.com/probml/pyprobml/blob/master/scripts/manifold_swiss_sklearn.py) [manifold_digits_sklearn.py](https://github.com/probml/pyprobml/blob/master/scripts/manifold_digits_sklearn.py)<jupyter_code>#@title Setup { display-mode: "form" }
%%time
# If you run this for the first time it would take ~25/30 seconds
!git clone https://github.com/probml/pyprobml /pyprobml &> /dev/null && git clone https://github.com/Sekhen/colab_powertoys.git &> /dev/null 
!pip3 install nbimporter -qqq 
%cd -q /content/colab_powertoys 
from colab_powertoys.probml_toys import probml_toys as pmlt
%cd -q /content/

pmlt.show_and_run("/pyprobml/scripts/manifold_swiss_sklearn.py")
pmlt.show_and_run("/pyprobml/scripts/manifold_digits_sklearn.py")<jupyter_output><empty_output><jupyter_text>Figure 20.38:     Laplacian eigenmaps applied to (a) Swiss roll.  Figure(s) generated by [manifold_swiss_sklearn.py](https://github.com/probml/pyprobml/blob/master/scripts/manifold_swiss_sklearn.py) [manifold_digits_sklearn.py](https://github.com/probml/pyprobml/blob/master/scripts/manifold_digits_sklearn.py)<jupyter_code>#@title Setup { display-mode: "form" }
%%time
# If you run this for the first time it would take ~25/30 seconds
!git clone https://github.com/probml/pyprobml /pyprobml &> /dev/null && git clone https://github.com/Sekhen/colab_powertoys.git &> /dev/null 
!pip3 install nbimporter -qqq 
%cd -q /content/colab_powertoys 
from colab_powertoys.probml_toys import probml_toys as pmlt
%cd -q /content/

pmlt.show_and_run("/pyprobml/scripts/manifold_swiss_sklearn.py")
pmlt.show_and_run("/pyprobml/scripts/manifold_digits_sklearn.py")<jupyter_output><empty_output><jupyter_text>Figure 20.39:     Illustration of the Laplacian matrix derived from an undirected graph. From   https://en.wikipedia.org/wiki/Laplacian_matrix . Used with kind permission of Wikipedia author AzaToth.<jupyter_code>#@title Setup { display-mode: "form" }
%%time
# If you run this for the first time it would take ~25/30 seconds
!git clone https://github.com/probml/pyprobml /pyprobml &> /dev/null && git clone https://github.com/Sekhen/colab_powertoys.git &> /dev/null 
!pip3 install nbimporter -qqq 
%cd -q /content/colab_powertoys 
from colab_powertoys.probml_toys import probml_toys as pmlt
%cd -q /content/

pmlt.show_image("/pyprobml/notebooks/figures/images/graphLaplacian.png")<jupyter_output><empty_output><jupyter_text>Figure 20.40:     Illustration of a (positive) function defined on a graph. From Figure 1 of [DI+13] . Used with kind permission of <NAME>.<jupyter_code>#@title Setup { display-mode: "form" }
%%time
# If you run this for the first time it would take ~25/30 seconds
!git clone https://github.com/probml/pyprobml /pyprobml &> /dev/null && git clone https://github.com/Sekhen/colab_powertoys.git &> /dev/null 
!pip3 install nbimporter -qqq 
%cd -q /content/colab_powertoys 
from colab_powertoys.probml_toys import probml_toys as pmlt
%cd -q /content/

pmlt.show_image("/pyprobml/notebooks/figures/images/graphFun.png")<jupyter_output><empty_output><jupyter_text>Figure 20.41:     tSNE applied to (a) Swiss roll.  Figure(s) generated by [manifold_swiss_sklearn.py](https://github.com/probml/pyprobml/blob/master/scripts/manifold_swiss_sklearn.py) [manifold_digits_sklearn.py](https://github.com/probml/pyprobml/blob/master/scripts/manifold_digits_sklearn.py)<jupyter_code>#@title Setup { display-mode: "form" }
%%time
# If you run this for the first time it would take ~25/30 seconds
!git clone https://github.com/probml/pyprobml /pyprobml &> /dev/null && git clone https://github.com/Sekhen/colab_powertoys.git &> /dev/null 
!pip3 install nbimporter -qqq 
%cd -q /content/colab_powertoys 
from colab_powertoys.probml_toys import probml_toys as pmlt
%cd -q /content/

pmlt.show_and_run("/pyprobml/scripts/manifold_swiss_sklearn.py")
pmlt.show_and_run("/pyprobml/scripts/manifold_digits_sklearn.py")<jupyter_output><empty_output><jupyter_text>Figure 20.42:     Illustration of the effect of changing the perplexity parameter when t-SNE is applied to some 2d data. From [MFI16] . See   http://distill.pub/2016/misread-tsne  for an animated version of these figures. Used with kind permission of <NAME>.<jupyter_code>#@title Setup { display-mode: "form" }
%%time
# If you run this for the first time it would take ~25/30 seconds
!git clone https://github.com/probml/pyprobml /pyprobml &> /dev/null && git clone https://github.com/Sekhen/colab_powertoys.git &> /dev/null 
!pip3 install nbimporter -qqq 
%cd -q /content/colab_powertoys 
from colab_powertoys.probml_toys import probml_toys as pmlt
%cd -q /content/

pmlt.show_image("/pyprobml/notebooks/figures/images/tSNE-wattenberg0.png.png")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11879667390498384, 'fraction_numerical': 0.016656956728986812, 'mean_word_length': 4.915256892230577, 'pattern_counts': {'":': 0, '<': 177, '<?xml version=': 0, '>': 262, 'https://': 112, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '42', 'dir': 'jupyter-structured-clean-dedup', 'id': '13628', 'license': 'MIT', 'n_tokens_mistral': 14247, 'n_tokens_neox': 13480, 'n_words': 3431, 'path': 'notebooks/figures/chapter20_figures.ipynb', 'repo_name': 'Junjun1guo/pyprobml', 'size': '37827'}