<jupyter_start><jupyter_text>The sum of squared errors is a convex function of $\mathbf{w}$: $$E(\mathbf{w})=(\mathbf{y}-\mathbf{\Phi w})^\top(\mathbf{y}-\mathbf{\Phi w})$$ The weight vector is $$\mathbf{\hat{w}}=(\mathbf{\Phi^\top\Phi})^{-1}\mathbf{\Phi^\top y}$$  Observation noiseThe observations are noisy:$$y_n = f_\mathbf{w}(x_n)+\epsilon_n$$The noise can be characterized with a probability density function, for example, a Gaussian density function, $\epsilon_n\sim\mathcal{N}(0,\sigma^2_{noise})$,$$p(\epsilon_n)=\frac{1}{\sqrt{2\pi\sigma^2_{noise}}}\exp\bigg(-\frac{\epsilon^2_n}{2\sigma^2_{noise}}\bigg)$$ The independent noise terms can be stacked as $\mathbf{\epsilon}=[\epsilon_1,...,\epsilon_N]^\top$, \begin{align}\mathbf{\epsilon}&\sim\mathcal{N}(\mathbf{\epsilon};0,\sigma^2_{noise}\mathbf{I})\\p(\mathbf{\epsilon})&=\prod^N_{i=1}p(\epsilon_n)=\bigg(\frac{1}{\sqrt{2\pi\sigma^2_{noise}}}\bigg)^N\exp(-\frac{\epsilon^\top\epsilon}{2\sigma^2_{noise}})\end{align} Given that $\mathbf{y}=\mathbf{f}+\mathbf{\epsilon}$, \begin{align}p(\epsilon) &= p(\mathbf{y}|\mathbf{f},\sigma^2_{noise}) = \mathcal{N}(\mathbf{y};\mathbf{f},\sigma^2_{noise}) \\&= \bigg(\frac{1}{\sqrt{2\pi\sigma^2_{noise}}}\bigg)^N\exp\bigg(-\frac{\|\mathbf{y}-\mathbf{f}\|^2}{2\sigma^2_{noise}}\bigg) \\&= \bigg(\frac{1}{\sqrt{2\pi\sigma^2_{noise}}}\bigg)^N\exp\bigg(-\frac{E(\mathbf{w})}{2\sigma^2_{noise}}\bigg)\end{align}  LikelihoodThe likelihood of the parameters is the probability of the data given parameters.- $p(\mathbf{y}|\mathbf{w},\sigma^2_{noise})$ is the probability of the observed data given the weights.- $\mathcal{L}(\mathbf{y}|\mathbf{w},\sigma^2_{noise})$ is the likelihood of the weights.  Maximum likelihoodFit the model weights to the data by maximizing the likelihood:$$\mathbf{\hat{w}}=\arg\max\mathcal{L}(\mathbf{w})=\arg\max\exp\bigg(-\frac{E(\mathbf{w})}{2\sigma^2_{noise}}\bigg)=\arg\min E(\mathbf{w})$$With an additive Gaussian independent noise model, the **maximum likelihood** and the **least squares** solutions are the same.<jupyter_code>print(errlistLU)
print(w)<jupyter_output>[2082.9435443058337, 915.5179868728524, 852.0086960680927, 779.9021317868293, 761.0539882151551, 662.4078175155137, 643.8318455004589, 529.4502292781594, 490.29415012657256, 471.5180097213798, 322.2365737351904, 321.91181712781224, 300.01857072519704, 298.67675984945015, 289.20164220557547, 284.06943927914, 267.1003813972985, 266.28824536898287, 294.06906171749114, 296.3996717705036, 194.073186960229]
[ 6.39773752e+00  1.60476718e+01 -1.99027687e+01 -1.24549999e+02
  1.12954352e+02  3.30341907e+02 -1.20378571e+02 -3.84728228e+02
 -8.16607630e+00  2.01465464e+02  5.35819868e+01 -4.21247674e+01
 -2.01745910e+01  1.16151911e+00  1.91175012e+00  2.98135810e-01
  5.14467134e-02  2.69740281e-02  4.53298263e-03 -8.50544471e-05
 -4.92070226e-05]<jupyter_text>Assume $\sigma_{noise}=10$,<jupyter_code>np.exp(-np.array(errlistLU)/100/2)
plt.plot(np.arange(21),np.exp(-np.array(errlistLU)/100/2))
plt.title('Likelihood')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.21168362323596981, 'fraction_numerical': 0.20315063997374466, 'mean_word_length': 6.309352517985611, 'pattern_counts': {'":': 0, '<': 8, '<?xml version=': 0, '>': 8, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '544493', 'license': 'MIT', 'n_tokens_mistral': 1558, 'n_tokens_neox': 1154, 'n_words': 166, 'path': 'Probabilistic Machine Learning/Introduction to Probabilistic Machine Learning/.ipynb_checkpoints/Likelihood and the concept of noise-checkpoint.ipynb', 'repo_name': 'zcemycl/ProbabilisticPerspectiveMachineLearning', 'size': '3047'}