<jupyter_start><jupyter_text>Figure 5.1:     Illustration of a 2d function with a local minimum, global minimum, and a some saddle points. We also illustrate some trajectories through this cost landscape. From Figure 17 from [Jos+19] . Used with kind permission of <NAME>.<jupyter_code>#@title Setup { display-mode: "form" }
%%time
# If you run this for the first time it would take ~25/30 seconds
!git clone https://github.com/probml/pyprobml /pyprobml &> /dev/null && git clone https://github.com/Sekhen/colab_powertoys.git &> /dev/null 
!pip3 install nbimporter -qqq 
%cd -q /content/colab_powertoys 
from colab_powertoys.probml_toys import probml_toys as pmlt
%cd -q /content/

pmlt.show_image("/pyprobml/notebooks/figures/images/saddleScreenShot.png.png")<jupyter_output><empty_output><jupyter_text>Figure 5.2:     Illustration of constrained maximization of a nonconvex 1d function. The area between the dotted vertical lines represents the feasible set. (a) There is a unique global maximum since the function is concave within the support of the feasible set. (b) There are two global maxima, both occuring at the boundary of the feasible set. (c) In the unconstrained case, this function has no global maximum, since it is unbounded.<jupyter_code>#@title Setup { display-mode: "form" }
%%time
# If you run this for the first time it would take ~25/30 seconds
!git clone https://github.com/probml/pyprobml /pyprobml &> /dev/null && git clone https://github.com/Sekhen/colab_powertoys.git &> /dev/null 
!pip3 install nbimporter -qqq 
%cd -q /content/colab_powertoys 
from colab_powertoys.probml_toys import probml_toys as pmlt
%cd -q /content/

pmlt.show_image("/pyprobml/notebooks/figures/images/optima.png")<jupyter_output><empty_output><jupyter_text>Figure 5.3:     (a) Smooth 1d function. (b) Non-smooth 1d function. (There is a discontinuity at the origin.) From   https://scipy-lectures.org/advanced/mathematical_optimization/index.html\smooth-and-non-smooth-problems . Used with kind permission of <NAME><jupyter_code>#@title Setup { display-mode: "form" }
%%time
# If you run this for the first time it would take ~25/30 seconds
!git clone https://github.com/probml/pyprobml /pyprobml &> /dev/null && git clone https://github.com/Sekhen/colab_powertoys.git &> /dev/null 
!pip3 install nbimporter -qqq 
%cd -q /content/colab_powertoys 
from colab_powertoys.probml_toys import probml_toys as pmlt
%cd -q /content/

pmlt.show_image("/pyprobml/notebooks/figures/images/smooth-fn.png")
pmlt.show_image("/pyprobml/notebooks/figures/images/nonsmooth-fn.png")<jupyter_output><empty_output><jupyter_text>Figure 5.4:     Steepest descent on a simple convex function, starting from $(0,0)$, for 20 steps, using a fixed step size. The global minimum is at $(1,1)$. (a) $\eta =0.1$. (b) $\eta =0.6$.  Figure(s) generated by [steepestDescentDemo.m](https://github.com/probml/pmtk3/blob/master/demos/steepestDescentDemo.m)<jupyter_code>#@title Setup { display-mode: "form" }
%%time
# If you run this for the first time it would take ~25/30 seconds
!git clone https://github.com/probml/pyprobml /pyprobml &> /dev/null && git clone https://github.com/Sekhen/colab_powertoys.git &> /dev/null 
!pip3 install nbimporter -qqq 
%cd -q /content/colab_powertoys 
from colab_powertoys.probml_toys import probml_toys as pmlt
%cd -q /content/

pmlt.show_image("/pyprobml/notebooks/figures/images/aokiSteepest1.png")
pmlt.show_image("/pyprobml/notebooks/figures/images/aokiSteepest6.png")<jupyter_output><empty_output><jupyter_text>Figure 5.5:     Illustration of the effect of condition number $\kappa $ on the convergence speed of steepest descent with exact line searches. (a) Large $\kappa $. (b) Small $\kappa $. From Lecture 18 (slides 42, 46) from [Ber09] .<jupyter_code>#@title Setup { display-mode: "form" }
%%time
# If you run this for the first time it would take ~25/30 seconds
!git clone https://github.com/probml/pyprobml /pyprobml &> /dev/null && git clone https://github.com/Sekhen/colab_powertoys.git &> /dev/null 
!pip3 install nbimporter -qqq 
%cd -q /content/colab_powertoys 
from colab_powertoys.probml_toys import probml_toys as pmlt
%cd -q /content/

pmlt.show_image("/pyprobml/notebooks/figures/images/steepestDescentLargeKappa.png")
pmlt.show_image("/pyprobml/notebooks/figures/images/steepestDescentSmallKappa.png")<jupyter_output><empty_output><jupyter_text>Figure 5.6:     (a) Steepest descent on the same function as \cref  fig:aokiFixed , starting from $(0,0)$, using line search.  Figure(s) generated by [steepestDescentDemo.m](https://github.com/probml/pmtk3/blob/master/demos/steepestDescentDemo.m)<jupyter_code>#@title Setup { display-mode: "form" }
%%time
# If you run this for the first time it would take ~25/30 seconds
!git clone https://github.com/probml/pyprobml /pyprobml &> /dev/null && git clone https://github.com/Sekhen/colab_powertoys.git &> /dev/null 
!pip3 install nbimporter -qqq 
%cd -q /content/colab_powertoys 
from colab_powertoys.probml_toys import probml_toys as pmlt
%cd -q /content/

pmlt.show_image("/pyprobml/notebooks/figures/images/aokiSteepestExact.png")
pmlt.show_image("/pyprobml/notebooks/figures/images/linesearchPerp.png")<jupyter_output><empty_output><jupyter_text>Figure 5.7:     Illustration of the Nesterov update. Adapted from Figure 11.6 of [Aur19] .<jupyter_code>#@title Setup { display-mode: "form" }
%%time
# If you run this for the first time it would take ~25/30 seconds
!git clone https://github.com/probml/pyprobml /pyprobml &> /dev/null && git clone https://github.com/Sekhen/colab_powertoys.git &> /dev/null 
!pip3 install nbimporter -qqq 
%cd -q /content/colab_powertoys 
from colab_powertoys.probml_toys import probml_toys as pmlt
%cd -q /content/

pmlt.show_image("/pyprobml/notebooks/figures/images/nesterov-geron-11-6.pdf.png")<jupyter_output><empty_output><jupyter_text>Figure 5.8:     Illustration of Newton's method for minimizing a 1d function. (a) The solid curve is the function $\mathcal  L (x)$. The dotted line $\mathcal  L _ \mathrm  quad  (\theta )$ is its second order approximation at $\theta _t$. The Newton step $d_t$ is what must be added to $\theta _t$ to get to the minimum of $\mathcal  L _ \mathrm  quad  (\theta )$. Adapted from Figure 13.4 of [Van06] .  Figure(s) generated by [newtonsMethodMinQuad.m](https://github.com/probml/pmtk3/blob/master/demos/newtonsMethodMinQuad.m) [newtonsMethodNonConvex.m](https://github.com/probml/pmtk3/blob/master/demos/newtonsMethodNonConvex.m)<jupyter_code>#@title Setup { display-mode: "form" }
%%time
# If you run this for the first time it would take ~25/30 seconds
!git clone https://github.com/probml/pyprobml /pyprobml &> /dev/null && git clone https://github.com/Sekhen/colab_powertoys.git &> /dev/null 
!pip3 install nbimporter -qqq 
%cd -q /content/colab_powertoys 
from colab_powertoys.probml_toys import probml_toys as pmlt
%cd -q /content/

pmlt.show_image("/pyprobml/notebooks/figures/images/newtonsMethodMinQuadTheta.png")
pmlt.show_image("/pyprobml/notebooks/figures/images/newtonsMethodNonConvexTheta.png")
 #@title Setup { display-mode: "form" }
%%time
# If you run this for the first time it would take ~25/30 seconds
!git clone https://github.com/probml/pyprobml /pyprobml &> /dev/null && git clone https://github.com/Sekhen/colab_powertoys.git &> /dev/null 
!pip3 install nbimporter -qqq 
%cd -q /content/colab_powertoys 
from colab_powertoys.probml_toys import probml_toys as pmlt
%cd -q /content/

pmlt.show_image("/pyprobml/notebooks/figures/images/newtonsMethodMinQuadTheta.png")
pmlt.show_image("/pyprobml/notebooks/figures/images/newtonsMethodNonConvexTheta.png")<jupyter_output><empty_output><jupyter_text>Figure 5.9:     Illustration of the trust region approach. The dashed lines represents contours of the original nonconvex objective. The circles represent successive quadratic approximations. From Figure 4.2 of [Raz14] . Used with kind permission of <NAME>.<jupyter_code>#@title Setup { display-mode: "form" }
%%time
# If you run this for the first time it would take ~25/30 seconds
!git clone https://github.com/probml/pyprobml /pyprobml &> /dev/null && git clone https://github.com/Sekhen/colab_powertoys.git &> /dev/null 
!pip3 install nbimporter -qqq 
%cd -q /content/colab_powertoys 
from colab_powertoys.probml_toys import probml_toys as pmlt
%cd -q /content/

pmlt.show_image("/pyprobml/notebooks/figures/images/trustRegion.png")<jupyter_output><empty_output><jupyter_text>Figure 5.10:     Changing the mean of a Gaussian by a fixed amount (from solid to dotted curve) can have more impact when the (shared) variance is small (as in a) compared to when the variance is large (as in b). Hence the impact (in terms of prediction accuracy) of a change to $\mu $ depends on where the optimizer is in $(\mu ,\sigma )$ space. From Figure 3 of [Ant+10] , reproduced from [Val00] . Used with kind permission of <NAME>.<jupyter_code>#@title Setup { display-mode: "form" }
%%time
# If you run this for the first time it would take ~25/30 seconds
!git clone https://github.com/probml/pyprobml /pyprobml &> /dev/null && git clone https://github.com/Sekhen/colab_powertoys.git &> /dev/null 
!pip3 install nbimporter -qqq 
%cd -q /content/colab_powertoys 
from colab_powertoys.probml_toys import probml_toys as pmlt
%cd -q /content/

pmlt.show_image("/pyprobml/notebooks/figures/images/meanchange_lowvar.png")
pmlt.show_image("/pyprobml/notebooks/figures/images/meanchange_highvar.png")<jupyter_output><empty_output><jupyter_text>Figure 5.11:     Illustration of the benefits of natural gradient vs steepest descent on a 2D problem. (a) Trajectories of the two methods in parameter space (red = steepest descent, blue = NG). They both start at $(1,-1)$, bottom right location. (b) Objective vs number of iterations. (c) Gradient field in the $\boldsymbol  \theta  $ parameter space. (d) Gradient field in the whitened $\boldsymbol  \phi  = \mathbf  F ^ \frac  1  2   \boldsymbol  \theta  $ parameter space used by NG.  Figure(s) generated by [nat_grad_demo.py](https://github.com/probml/pyprobml/blob/master/scripts/nat_grad_demo.py)<jupyter_code>#@title Setup { display-mode: "form" }
%%time
# If you run this for the first time it would take ~25/30 seconds
!git clone https://github.com/probml/pyprobml /pyprobml &> /dev/null && git clone https://github.com/Sekhen/colab_powertoys.git &> /dev/null 
!pip3 install nbimporter -qqq 
%cd -q /content/colab_powertoys 
from colab_powertoys.probml_toys import probml_toys as pmlt
%cd -q /content/

pmlt.show_and_run("/pyprobml/scripts/nat_grad_demo.py")<jupyter_output><empty_output><jupyter_text>Figure 5.12:     Illustration of the LMS algorithm. Left: we start from $\boldsymbol  \theta  =(-0.5,2)$ and slowly converging to the least squares solution of $ \boldsymbol  \theta   =(1.45, 0.93)$ (red cross). Right: plot of objective function over time. Note that it does not decrease monotonically.  Figure(s) generated by [lms_demo.py](https://github.com/probml/pyprobml/blob/master/scripts/lms_demo.py)<jupyter_code>#@title Setup { display-mode: "form" }
%%time
# If you run this for the first time it would take ~25/30 seconds
!git clone https://github.com/probml/pyprobml /pyprobml &> /dev/null && git clone https://github.com/Sekhen/colab_powertoys.git &> /dev/null 
!pip3 install nbimporter -qqq 
%cd -q /content/colab_powertoys 
from colab_powertoys.probml_toys import probml_toys as pmlt
%cd -q /content/

pmlt.show_and_run("/pyprobml/scripts/lms_demo.py")<jupyter_output><empty_output><jupyter_text>Figure 5.13:     Illustration of some constrained optimization problems. Red contours are the level sets of the objective function $\mathcal  L (\boldsymbol  \theta  )$. Optimal constrained solution is the black dot, (a) Blue line is the equality constraint $h(\boldsymbol  \theta  )=0$. (b) Blue lines denote the inequality constraints $|\theta _1| + |\theta _2| \leq 1$. (Compare to \cref  fig:L2L1contours  (left).)<jupyter_code>#@title Setup { display-mode: "form" }
%%time
# If you run this for the first time it would take ~25/30 seconds
!git clone https://github.com/probml/pyprobml /pyprobml &> /dev/null && git clone https://github.com/Sekhen/colab_powertoys.git &> /dev/null 
!pip3 install nbimporter -qqq 
%cd -q /content/colab_powertoys 
from colab_powertoys.probml_toys import probml_toys as pmlt
%cd -q /content/

pmlt.show_image("/pyprobml/notebooks/figures/images/quadObjectiveLine_theta.png")
pmlt.show_image("/pyprobml/notebooks/figures/images/75 100 50 100.png")<jupyter_output><empty_output><jupyter_text>Figure 5.14:     (a) A convex polytope in 2d defined by the intersection of linear constraints. (b) Depiction of the feasible set as well as the linear objective function. The red line is a level set of the objective, and the arrow indicates the direction in which it is improving. We see that the optimal solution lies at a vertex of the polytope.<jupyter_code>#@title Setup { display-mode: "form" }
%%time
# If you run this for the first time it would take ~25/30 seconds
!git clone https://github.com/probml/pyprobml /pyprobml &> /dev/null && git clone https://github.com/Sekhen/colab_powertoys.git &> /dev/null 
!pip3 install nbimporter -qqq 
%cd -q /content/colab_powertoys 
from colab_powertoys.probml_toys import probml_toys as pmlt
%cd -q /content/

pmlt.show_image("/pyprobml/notebooks/figures/images/polytope2d.png")<jupyter_output><empty_output><jupyter_text>Figure 5.15:     Illustration of projected gradient descent. $\mathbf  w $ is the current parameter estimate, $\mathbf  w '$ is the update after a gradient step, and $P_ \mathcal  C  (\mathbf  w ')$ projects this onto the constraint set $\mathcal  C $. From   https://bit.ly/3eJ3BhZ  Used with kind permission of <NAME>.<jupyter_code>#@title Setup { display-mode: "form" }
%%time
# If you run this for the first time it would take ~25/30 seconds
!git clone https://github.com/probml/pyprobml /pyprobml &> /dev/null && git clone https://github.com/Sekhen/colab_powertoys.git &> /dev/null 
!pip3 install nbimporter -qqq 
%cd -q /content/colab_powertoys 
from colab_powertoys.probml_toys import probml_toys as pmlt
%cd -q /content/

pmlt.show_image("/pyprobml/notebooks/figures/images/proj-grad.png")<jupyter_output><empty_output><jupyter_text>Figure 5.16:     Illustration of a bound optimization algorithm. Adapted from Figure 9.14 of [Bis06] .  Figure(s) generated by [emLogLikelihoodMax.m](https://github.com/probml/pmtk3/blob/master/demos/emLogLikelihoodMax.m)<jupyter_code>#@title Setup { display-mode: "form" }
%%time
# If you run this for the first time it would take ~25/30 seconds
!git clone https://github.com/probml/pyprobml /pyprobml &> /dev/null && git clone https://github.com/Sekhen/colab_powertoys.git &> /dev/null 
!pip3 install nbimporter -qqq 
%cd -q /content/colab_powertoys 
from colab_powertoys.probml_toys import probml_toys as pmlt
%cd -q /content/

pmlt.show_image("/pyprobml/notebooks/figures/images/emLogLikelihoodMax.png")<jupyter_output><empty_output><jupyter_text>Figure 5.17:     The quadratic lower bound of an MM algorithm (solid) and the quadratic approximation of Newton's method (dashed) superimposed on an empirical density esitmate (dotted). The starting point of both algorithms is the circle. The square denotes the outcome of one MM update. The diamond denotes the outcome of one Newton update. (a) Newton's method overshoots the-se, global maximum. (b) Newton's method results in a reduction of the objective. From Figure 4 of [MC05] . Used with kind permission of <NAME>.<jupyter_code>#@title Setup { display-mode: "form" }
%%time
# If you run this for the first time it would take ~25/30 seconds
!git clone https://github.com/probml/pyprobml /pyprobml &> /dev/null && git clone https://github.com/Sekhen/colab_powertoys.git &> /dev/null 
!pip3 install nbimporter -qqq 
%cd -q /content/colab_powertoys 
from colab_powertoys.probml_toys import probml_toys as pmlt
%cd -q /content/

pmlt.show_image("/pyprobml/notebooks/figures/images/MMvsNewton.png")<jupyter_output><empty_output><jupyter_text>Figure 5.18:     Illustration of the EM for a GMM applied to the Old Faithful data. The degree of redness indicates the degree to which the point belongs to the red cluster, and similarly for blue; thus purple points have a roughly 50/50 split in their responsibilities to the two clusters. Adapted from [Bis06]  Figure 9.8.  Figure(s) generated by [mix_gauss_demo_faithful.py](https://github.com/probml/pyprobml/blob/master/scripts/mix_gauss_demo_faithful.py)<jupyter_code>#@title Setup { display-mode: "form" }
%%time
# If you run this for the first time it would take ~25/30 seconds
!git clone https://github.com/probml/pyprobml /pyprobml &> /dev/null && git clone https://github.com/Sekhen/colab_powertoys.git &> /dev/null 
!pip3 install nbimporter -qqq 
%cd -q /content/colab_powertoys 
from colab_powertoys.probml_toys import probml_toys as pmlt
%cd -q /content/

pmlt.show_and_run("/pyprobml/scripts/mix_gauss_demo_faithful.py")<jupyter_output><empty_output><jupyter_text>Figure 5.19:     (a) Illustration of how singularities can arise in the likelihood function of GMMs. Here $K=2$, but the first mixture component is a narrow spike (with $\sigma _1 \approx 0$) centered on a single data point $x_1$. Adapted from Figure 9.7 of [Bis06] .  Figure(s) generated by [mix_gauss_singularity.py](https://github.com/probml/pyprobml/blob/master/scripts/mix_gauss_singularity.py) [mixGaussMLvsMAP.m](https://github.com/probml/pmtk3/blob/master/demos/mixGaussMLvsMAP.m)<jupyter_code>#@title Setup { display-mode: "form" }
%%time
# If you run this for the first time it would take ~25/30 seconds
!git clone https://github.com/probml/pyprobml /pyprobml &> /dev/null && git clone https://github.com/Sekhen/colab_powertoys.git &> /dev/null 
!pip3 install nbimporter -qqq 
%cd -q /content/colab_powertoys 
from colab_powertoys.probml_toys import probml_toys as pmlt
%cd -q /content/

pmlt.show_and_run("/pyprobml/scripts/mix_gauss_singularity.py")
 #@title Setup { display-mode: "form" }
%%time
# If you run this for the first time it would take ~25/30 seconds
!git clone https://github.com/probml/pyprobml /pyprobml &> /dev/null && git clone https://github.com/Sekhen/colab_powertoys.git &> /dev/null 
!pip3 install nbimporter -qqq 
%cd -q /content/colab_powertoys 
from colab_powertoys.probml_toys import probml_toys as pmlt
%cd -q /content/

pmlt.show_image("/pyprobml/notebooks/figures/images/gmmSingularityB.png")
pmlt.show_image("/pyprobml/notebooks/figures/images/mixGaussMLvsMAP.png")<jupyter_output><empty_output><jupyter_text>Figure 5.20:     Left: $N=200$ data points sampled from a mixture of 2 Gaussians in 1d, with $\pi _k=0.5$, $\sigma _k=5$, $\mu _1=-10$ and $\mu _2=10$. Right: Likelihood surface $p( \mathcal  D  |\mu _1,\mu _2)$, with all other parameters set to their true values. We see the two symmetric modes, reflecting the unidentifiability of the parameters.  Figure(s) generated by [mixGaussLikSurfaceDemo.m](https://github.com/probml/pmtk3/blob/master/demos/mixGaussLikSurfaceDemo.m)<jupyter_code>#@title Setup { display-mode: "form" }
%%time
# If you run this for the first time it would take ~25/30 seconds
!git clone https://github.com/probml/pyprobml /pyprobml &> /dev/null && git clone https://github.com/Sekhen/colab_powertoys.git &> /dev/null 
!pip3 install nbimporter -qqq 
%cd -q /content/colab_powertoys 
from colab_powertoys.probml_toys import probml_toys as pmlt
%cd -q /content/

pmlt.show_image("/pyprobml/notebooks/figures/images/gmmLikSurfaceHistSigma5.png")
pmlt.show_image("/pyprobml/notebooks/figures/images/gmmLikSurfaceSurSigma5.png")<jupyter_output><empty_output><jupyter_text>Figure 5.21:     Illustration of data imputation. (a) Scatter plot of true values vs imputed values using true parameters. (b) Same as (a), but using parameters estimated with EM. We just show the first four variables, for brevity.  Figure(s) generated by [gaussImputationDemoEM.m](https://github.com/probml/pmtk3/blob/master/demos/gaussImputationDemoEM.m)<jupyter_code>#@title Setup { display-mode: "form" }
%%time
# If you run this for the first time it would take ~25/30 seconds
!git clone https://github.com/probml/pyprobml /pyprobml &> /dev/null && git clone https://github.com/Sekhen/colab_powertoys.git &> /dev/null 
!pip3 install nbimporter -qqq 
%cd -q /content/colab_powertoys 
from colab_powertoys.probml_toys import probml_toys as pmlt
%cd -q /content/

pmlt.show_image("/pyprobml/notebooks/figures/images/mvnImputeScatterTruth.png")
pmlt.show_image("/pyprobml/notebooks/figures/images/mvnImputeScatterEm.png")<jupyter_output><empty_output><jupyter_text>Figure 5.22:     Illustration of grid search (left) vs random search (right). From Figure 1 of [JY12] . Used with kind permission of <NAME>.<jupyter_code>#@title Setup { display-mode: "form" }
%%time
# If you run this for the first time it would take ~25/30 seconds
!git clone https://github.com/probml/pyprobml /pyprobml &> /dev/null && git clone https://github.com/Sekhen/colab_powertoys.git &> /dev/null 
!pip3 install nbimporter -qqq 
%cd -q /content/colab_powertoys 
from colab_powertoys.probml_toys import probml_toys as pmlt
%cd -q /content/

pmlt.show_image("/pyprobml/notebooks/figures/images/rndGrid.png")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11597950260918621, 'fraction_numerical': 0.014902919467820037, 'mean_word_length': 4.774158523344191, 'pattern_counts': {'":': 0, '<': 96, '<?xml version=': 0, '>': 144, 'https://': 62, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '22', 'dir': 'jupyter-structured-clean-dedup', 'id': '126461', 'license': 'MIT', 'n_tokens_mistral': 7764, 'n_tokens_neox': 7408, 'n_words': 2069, 'path': 'notebooks/figures/chapter5_figures.ipynb', 'repo_name': 'Junjun1guo/pyprobml', 'size': '21334'}