<jupyter_start><jupyter_text>ML Strategy * Collect more data* Collect more diverse trainign set* Train algorithm longer with gradient descetn* Try adam isntead of gradient descent* Try bigger networks* Try smaller networks* Try dropout* Add L2 regularizatión* Network architecture* Network archicteture     - Activvation    - \ hidden units     Orthogonalization  For a supervised learning system to do well, you usually need to tune the knobs of your system to make sure that four things hold true. 1. **Fit training set well on cost function** First, is that you usually have to make sure that you're at least doing well on the training set. So performance on the training set needs to pass some acceptability assessment. For some applications, this might mean doing comparably to human level performance. But this will depend on your application, and we'll talk more about comparing to human level performance next week.2. **Fit dev set well on cost function**3. **Fit test set well on cost function**3. **Performs well in real world**el priemro se soluccion con bigget network, the optiization algorithmel segundo con regularization o con un bigger traingin setet tres con bigger dev sety el cuarto cambiando el dev set o la cost function The exact details of what's precision and recall don't matter too much for this example. But briefly, the definition of precision is, of the examples that your classifier recognizes as cats,Play video starting at 1 minute 23 seconds and follow transcript1:23What percentage actually are cats?Play video starting at 1 minute 32 seconds and follow transcript1:32So if classifier A has 95% precision, this means that when classifier A says something is a cat, there's a 95% chance it really is a cat. And recall is, of all the images that really are cats, what percentage were correctly recognized by your classifier? So what percentage of actual cats, Are correctly recognized? *  I often recommend that you set up a single real number evaluation metric for your problem. Let's look at an example.precision: the examples that your classifier recognizes as cats, What percentage actually are catso if classifier A has 95% precision, this means that when classifier A says something is a cat, there's a 95% chance it really is a cat.recall: of all the images that really are cats, what percentage were correctly recognized by your classifier? So what percentage of actual cats, Are correctly recognized? So if classifier A is 90% recall, this means that of all of the images in, say, your dev sets that really are cats, classifier A accurately pulled out 90% of them. trade-off between precision and recallThe problem with using precision recall as your evaluation metric is that if classifier A does better on recall, which it does here, the classifier B does better on precision, then you're not sure which classifier is better.you just have to find a new evaluation metric that combines precision and recall. In the machine learning literature, the standard way to combine precision and recall is something called an F1 score. Think as average of precision (P) and recall$$F1 = \frac{2}{\frac{1}{P}+\frac{1}{R}}$$ Harmonic mean of precition P and Recall Rwhat I recommend in this example is, in addition to tracking your performance in the four different geographies, to also compute the average. And assuming that average performance is a reasonable single real number evaluation metric, by computing the average, you can quickly tell that it looks like algorithm C has a lowest average error.---**Satisficing and Optimizing metric**To summarize, if there are multiple things you care about by say there's one as the optimizing metric that you want to do as well as possible on and one or more as satisficing metrics were you'll be satisfice. Almost it does better than some threshold you can now have an almost automatic way of quickly looking at multiple core size and picking the, quote, best one. Now these evaluation matrix must be evaluated or calculated on a training set or a development set or maybe on the test set. So one of the things you also need to do is set up training, dev or development, as well as test sets. In the next video, I want to share with you some guidelines for how to set up training, dev, and test sets. So let's go on to the next.cost = accuracy - 0.5 * running timemaximize accuracy but subject that maximizes accuracy but subject to that the running time, that is the time it takes to classify an image, that that has to be less than or equal to 100 milliseconds. that running time is what we call a satisficing metricSo in this case accuracy is the optimizing metric and a number of false positives every 24 hours is the satisficing metric**Train/dev/test distributions**The way you set up your training dev, or development sets and test sets, can have a huge impact on how rapidly you or your team can make progress on building machine learning application.* Dev set So, that dev set is also called the development set, or sometimes called the hold out cross validation set. And, workflow in machine learning is that you try a lot of ideas, train up different models on the training set, and then use the dev set to evaluate the different ideas and pick one. And, keep iterating to improve dev set performance until, finally, you have one clause that you're happy with that you then evaluate on your test set.* choose a dev set and test set to reflect data you expect to get in future and consider important to do well on. And, in particular, the dev set and the test set here, should come from the same distribution. So, whatever type of data you expect to get in the future, and once you do well on, try to get data that looks like that.   Size of Dev set* So if you had a hundred examples in total, these 70/30 or 60/20/20 rule of thumb would be pretty reasonable. If you had thousand examples, maybe if you had ten thousand examples, these heuristics are not unreasonable.* say you have a million training examples. it might be quite reasonable to set up your data so that you have 98% in the training set, 1% dev, and 1% test. Size of test set* Set your test set to be enough to give high confidence in the overall performance of your system.* Maybe all you need is a train and dev set, And I think, not having a test set might be okay* I do find it reassuring to have a separate test set you can use to get an unbiased estimate of how I was doing before you shift it, but if you have a very large dev set so that you think you won't overfit the dev set too badSo to summarize, in the era of big data, I think the old rule of thumb of a 70/30 is that, that no longer applies. And the trend has been to use more data for training and less for dev and test, especially when you have a very large data sets. And the rule of thumb is really to try to set the dev set to big enough for its purpose, which helps you evaluate different ideas and pick this up from AOP better. And the purpose of test set is to help you evaluate your final cost buys. You just have to set your test set big enough for that purpose, and that could be much less than 30% of the data. So, I hope that gives some guidance or some suggestions on how to set up your dev and test sets in the Deep Learning era. Next, it turns out that sometimes, part way through a machine learning problem, you might want to change your evaluation metric, or change your dev and test sets. Let's talk about it when you might want to do that.---  When to change dev/test sets and metricsYou've seen how set to have a dev set and evaluation metric is like placing a target somewhere for your team to aim at. **Orthogonalization for better performance**1. PLace target$$Error = \frac{1}{\sum_i w^{(i)}}\sum_i w^{(i)}L\{Y_{pred}^{(i)}, y^{(i)} \}$$$w^{(i)}$ = 1 if $x^{(i)}$ is non-porn  $w^{(i)}$ = 10 if $x^{(i)}$ is porn*Orthogonalization for cat picture: anti-porn** So far weve only discussed how to define a metric to evaluate classifiers (Place the target)* Worry separately about how to do wel on this metric---Bayes optimal error. Best posible error. That can't never being surpass.*Why compare to human-level performance*Humans are quite good a lot of task. So long as ML is worse than human, you can:* get labeled data from humans* Gain insight from manual error analysis. Why did a person get this right* Better analysis of bias/variances Avoidable Bias* o the fact that there's a huge gap between how well your algorithm does on your training set versus how humans do shows that your algorithm isn't even fitting the training set well. So in terms of tools to reduce bias or variance, in this case I would say focus on reducing bias. So you want to do things like train a bigger neural network or run training set longer, just try to do better on the training set* Avoidable bias: difference between bayes and training error. You don't actually want to get below Bayes error.* Variance: Difference betweem training error and dev error.* Focus on either bias or variance reduction techniques  Understanding Human Level Performance**Proxy for Bayes Error**"So to recap, having an estimate of human-level performance gives you an estimate of Bayes error. And this allows you to more quickly make decisions as to whether you should focus on trying to reduce a bias or trying to reduce the variance of your algorithm.Suppose: Surpassing human-level performance* once you've surpassed this 0.5% threshold, your options, your ways of making progress on the machine learning problem are just less clear. It doesn't mean you can't make progress, you might still be able to make significant progress, but some of the tools you have for pointing you in a clear direction just don't work as well*Problems where ML significantly surpasses human -level performance** Online advertising* Product recommendations* Logistic (predicting transit time)* Loan approvals1. All this examples are actually learning from structured data. Where you might have a databse of waht has users clicked on. This are not actual perception problems. These are not computer vision2. today there are speech recognition systems that can surpass human-level performance. And there are also some computer vision, some image recognition tasks, where computers have surpassed human-level performance3. Medical       ECGs, skin cancer, narrow radiology task        Improving your model performance**supervised learning algorithm to work well**1. You can fit the training set pretty well     - LOW Avoidable bias    - Problem is solved by training a bigger network or training longer2. The Training set performance generalizes pretty well to the dev/test set    - Variance    - Problem is solved regularization or getting more training data that could help you generalize better to dev set dat    **Steps.**1. looking at the difference between your training error and your proxy for Bayes error and just gives you a sense of the avoidable bias. In other words, just how much better do you think you should be trying to do on your training set. 2. And then look at the difference between your dev error and your training error as an estimate of how much of a variance problem you have. In other words, how much harder you should be working to make your performance generalized from the training set to the dev set that it wasn't trained on explicitly.<jupyter_code>[9]*7<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.038852832836344504, 'fraction_numerical': 0.006753201192773198, 'mean_word_length': 4.43258694616484, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '8112', 'license': 'MIT', 'n_tokens_mistral': 2644, 'n_tokens_neox': 2503, 'n_words': 1913, 'path': 'Curso3.ipynb', 'repo_name': 'camilogutierrez/MachineLearning', 'size': '11402'}