<jupyter_start><jupyter_text>Problem Set 3: Extra problems Learning from data [TIF285], Chalmers, Fall 2019Last revised: 11-Oct-2019 by <NAME> [<EMAIL>]  Instructions - See deadline on the course web page- This problem set is performed individually (but collaboration is encouraged) and contains a number of basic and extra problems; you can choose which and how many to work on.- See examination rules on the course web page.- Hand-in is performed through the following **two** actions:  - Upload of your solution in the form of a jupyter notebook, or python code, via Canvas.  - Answer the corresponding questions on OpenTA.    Note that the hand-in is not complete, and will not be graded, if any of those actions is not performed. Fill your personal details- Name: **Lastname, Firstname**- Personnummer: **YYMMDD-XXXX**    (civic registration number)  ProblemsThe 8 extra points of this problem set are distributed over two problems:5. Bayesian Optimization (4 extra points)6. Deep neural network python class (4 extra points)  Problem 5: (extra) Bayesian Optimization (4 extra points)<jupyter_code># KEYS
#
%matplotlib inline
import numpy as np
import scipy as sp
from matplotlib import pyplot as plt
import GPy

# Not really needed, but nicer plots
import seaborn as sns
sns.set()
sns.set_context("talk")<jupyter_output><empty_output><jupyter_text>A univariate minimization problem Try to minimize the function$$f(x) = \sin(6 x) + 0.2 x^2 - 0.7 x$$on the interval $x \in [-5,5]$.The aim is to find the position of the minimum $x^*$ to within $\pm 0.05$ under the constraint that we would like to make as few function evaluations as possible.  Task 1: Using "standard" optimization methods a. **Plot the true function and indicate the position of the minimum**<jupyter_code># Insert code here
#<jupyter_output><empty_output><jupyter_text>b. Find the minimum using `scipy.optimize.minimize` with `method='Nelder-Mead'`. * Choose the starting point randomly from a uniform pdf $U(-5,5)$. * Repeat one hundred times. **Do you always get the same minimum?*** More specifically, set the tolerance of the optimizer to `tol=0.01` and check for success by the criterion $|x^* - x^*_\mathrm{opt}| < 0.05$, where $x^*_\mathrm{opt}$ is the result from the optimizer.* Be quantitative about the average number of function evaluations that are needed per successful optimization. Compute the ratio of the total number of function evaluations number (summed over the 100 tries with different starting points) with the number of successful attempts.    *Hint*: The number of function evaluations from a `scipy.optimize.minimize` result is returned in the ``OptimizeResult`` object.<jupyter_code># Insert code here
#<jupyter_output><empty_output><jupyter_text>Task 2. Build your own BayesOpt algorithm You will now try a very different approach to minimize an objective function (this is a conventional name for the function that we are trying to minimize or maximize). The approach is known as **Bayesian optimization** and the basic idea is the following:* Sample a few random points and build a **statistical model** for the unknown function (usually a Gaussian Process model).* Make a decision which point to sample next based on a so called **acquisition function** evaluated from the statistical model. This will incorporate our current knowledge about the function including our uncertainty for its value in different regions.* Improve the statistical model using the new sample. Continue sampling new points according to the acquisition function.* If done correctly, this approach will balance **exploration** of new regions that might contain the minimum, and **exploitation** of searching the region that is currently most promising.* Very importantly, this method also works when you are dealing with **noisy objective functions**, i.e. when your "measurement" of its value at a new point in parameter space contains some random noise.Your task is to repeat the above minimization with **your own Bayesian Optimization algorithm**, that should be assembled as described below. Such algoritms are built into libraries such as `Scikit-optimize` and `GPyOpt`, but we will build our own simple version using functions from `numpy`, `scipy`, and `GPy` (for building the statistical model). The pseudo-code for BayesOpt is the following (see specific hints for your implementation at the end):1. pick starting points $\mathbf{x}^{(1)},\mathbf{x}^{(2)},\ldots \mathbf{x}^{(k)}$, where $k \geq 2$1. evaluate the objective function $f(\mathbf{x})$ to obtain $y^{(i)}=f(\mathbf{x}^{(i)})$ for $i=1,\ldots,k$1. initialize a data vector $\mathcal{D}_k = \left\{(\mathbf{x}^{(i)},y^{(i)})\right\}_{i=1}^k$1. select a statistical model for $f(\mathbf{x})$1. **For** {$n=k+1,k+2,\ldots$}   1.    select $\mathbf{x}^{(n)}$ by optimizing the acquisition function: $\mathbf{x}^{(n)} = \underset{\mathbf{x}}{\text{arg max}}\, \mathcal{A}(\mathbf{x}|\mathcal{D}_{n-1})$   1.    evaluate the objective function to obtain $y^{(n)}=f(\mathbf{x}^{(n)})$   1.    augment the data vector $\mathcal{D}_n = \left\{\mathcal{D}_{n-1} , (\mathbf{x}^{(n)},y^{(n)})\right\}$   1.    update the statistical model for $f(\mathbf{x})$1. **end for**   Check for the minimum in the data vector that has been collected (note that it doesn't necessarily have to be the last sample). **Hints:*** You have to implement all steps in the above pseudo-code.* You can try with $k=2$ starting points.* For the statistical model you can use `GPyOpt`. Follow the examples from the lectures and the exercise notebook.* Any knowledge about the objective function should be built into the covariance function. Let us assume that we dan't have much information and that we use a standard RBF kernel.* It is recommended to constrain the RBF lengthscale so that it doesn't become unrealistic small. With the GPy model called `model`, such a constraint can be imposed using `model['rbf.lengthscale'].constrain_bounded(.1,10)`.* Implement the so called Lower Confidence Bound (LCB) acquisition function for use in step 5A. Then, the acquisition function is simply: $\mathcal{A}(\boldsymbol{x}; | \mathcal{D}_{n-1}) = \mu(\boldsymbol{x}) - \sigma(\boldsymbol{x})$, where  * $\mu(\boldsymbol{x})$ is the mean of the GP model trained with the data $\mathcal{D}_{n-1})$.  * $\sigma(\boldsymbol{x})$ is the standard deviation of the GP model trained with the data $\mathcal{D}_{n-1})$.* Remember that the statistical model has to be updated (the hyperparameters re-optimized) at step 5D.  Tasks* Implement the BayesOpt minimizer* Assume that you are allowed a total of 100 function evaluations ($k$ of them for the starting points and $k-2$ in the loop). Are you able to find the minimum to within $\pm 0.05$?* Plot the final statistical model together with the true function and mark the minimum.* Plot also the convergence of the minimum value $\min(y_n)$ as a function of the iteration number $n \in \{1, \ldots, 100\}$.<jupyter_code># Insert your code here
#<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10142247510668563, 'fraction_numerical': 0.011948790896159318, 'mean_word_length': 4.953429297205758, 'pattern_counts': {'":': 0, '<': 20, '<?xml version=': 0, '>': 19, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '330573', 'license': 'CC0-1.0', 'n_tokens_mistral': 1907, 'n_tokens_neox': 1804, 'n_words': 960, 'path': 'doc/pub/ProblemSets/Set3/TIF285_ProblemSet3-Problem5-extra.ipynb', 'repo_name': 'quantshah/tif285', 'size': '7063'}