<jupyter_start><jupyter_text>Reducing loss/ optimization as  learningAuthor: <NAME>Some material modified from the [Google ML Crash Course](https://developers.google.com/machine-learning/crash-course/reducing-loss/an-iterative-approach)  What is Loss?In the previous section we looked at ordinary least squares (OLS) linear regression from a machine learning context. OLS linear regression minimizes **error** by minimizing the Sum of the square residuals:    $$ RSS = \sum_{i=1}^{n} (y_{i} - f(x_{i}))^2 $$ The Sum of the Square Residuals (above) is only one way of representing error. Other methods of representing **error** include:* Root mean squared error * MEan Absolute error (L1)* Mean Squared Error (L2)* Cross entropy (binary classification)* Hinge (binary classification) All of these are are **Loss Functions**    Cost functions vs. loss functionsA **loss function** is something that can be calculated on a single observation, for example:$$\mathscr{L}(y\hat,y) = $\frac{1}{2}(\hat{y} - y )^2 $$A **cost function** is the sum of lost functions over a batch of data, For example:$$Cost Function = \frac{1}{m}\sum_{i=1}^{m}\mathscr{L}(\hat{y}^{(i)} - y^{(i)})^2 $$So the loss is how good a single prediction is and cost is how good a set of predictions are.  Why are there different loss functions?Loss functions are chosen to make the process of learning the model parameters (fitting) as easy as possible. If you are a consumer of machine learning algorithms fitting parameters is the hidden middle step in most learning methods.Recall the steps of model training:1. Select a model2. Fit the best parameters to the model3. Evaluate the model ![learning-opt.svg](attachment:learning-opt.svg)  Learning Parameters and fittingThe process of learning model parameter values is at the heart of machine learning. This is the step where learning occurs. As someone who applies machine learning to their data rather than developing their own ML algorithms this process can be somewhat hidden.Common optimization methods for ML today are: * Stochastic gradient Descent* ADAM* AdaGrad* RMSProp* MomentumYou can visually try out the optimizers  by clicking on as any point on the surface below to see how the optimizer finds a minima.The graphic is from [<NAME>](https://bl.ocks.org/EmilienDupont/aaf429be5705b219aaaf8d691e27ca87).<jupyter_code>from IPython.display import IFrame

IFrame(src='../nb-images/sgd.html', width=1000, height=600)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09021035598705501, 'fraction_numerical': 0.015372168284789644, 'mean_word_length': 5.308673469387755, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '403381', 'license': 'MIT', 'n_tokens_mistral': 683, 'n_tokens_neox': 638, 'n_words': 313, 'path': 'assets/nb-lessons/04_Reducing_loss_optimization_as_learning.ipynb', 'repo_name': 'gklarenberg/ml-training-site', 'size': '2485'}