<commit_before>---
layout: post
title: Is Optimization a Sufficient Language for Understanding Deep Learning?
date:  2019-3-11 13:00:00
author: SA + ?? 
visible: False
---

In this Deep Learning era, machine learning usually boils down to defining a suitable objective/cost function for the learning task at hand, and then optimizing this function using some variant of gradient descent (implemented via backpropagation).  Little wonder that hundreds of ML papers each year are devoted to various aspects of optimization. Today I will suggest that if our goal is mathematical understanding of deep learning, then  the optimization viewpoint is potentially insufficient ---at least in the conventional view:

> **Conventional View (CV) of Optimization**: Find me a solution of minimum possible value of the objective, as fast as possible. 

Note that *a priori* it is not obvious if all learning should involve optimizing a single objective. Whether or not this is true for the working of the brain is a longstanding open question in neuroscience. Brain components appear to have been repurposed/cobbled together through various accidents of evolution and the whole assemblage may or may not boil down to optimization of an objective. See [this survey by Marblestone et al](https://arxiv.org/pdf/1606.03813.pdf). 

I am suggesting that deep learning algorithms also have important properties that are not always reflected in the objective value. Current deep nets, being vastly overparametrized, have multiple optima. They are trained until the objective is almost zero, and training is said to succeed if the optimum (or near-optimum) model thus found also performs well on unseen/held-out data ---i.e., *generalizes.* The value of the objective may imply nothing about generalization (see [Zhang et al.](https://arxiv.org/abs/1611.03530)). 

Of course experts will now ask: "Wasn't generalization theory invented precisely for this reason as the "second leg" of machine learning,  where optimization is the first leg?" For instance this theory shows how to add regularizers to the training objective to ensure the solution generalizes. Or that *early stopping* (i.e., stopping before reaching the optimum) or even adding noise to the gradient (e.g. by playing with batch sizes and learning rates) can be preferable to perfect optimization, even in simple settings such as regression. 

However, in practice explicit regularizers  and noising tricks can't prevent deep nets from attaining low training objective even on data with random labels; see [Zhang et al.](https://arxiv.org/abs/1611.03530). Current generalization theory is designed to give *post hoc* explanations for why a particular model generalized. It is agnostic about *how* the solution was obtained, and thus makes few prescriptions ---apart from recommending some regularization--- for optimization.   (See my earlier [blog post](http://www.offconvex.org/2017/12/08/generalization1/), which explains the distinction between descriptive and prescriptive methods, and  that generalization theory is primarily descriptive.) The fundamental mystery is:

> Even vanilla gradient descent (GD) is good at finding models with reasonable generalization. Furthermore, methods to speed up gradient descent (e.g., acceleration or adaptive regularization) can sometimes lead to worse generalization. 

In other words, GD has an innate bias towards finding solutions with good generalization. Magic happens along the GD trajectory and is not captured in the objective value per se. We're reminded of the old adage. 

> The journey matters more than the destination. 

Below to illustrate this viewpoint I describe new papers with rigorous analysis of gradient descent in two simple but suggestive settings. I am barely sketching the results and hope that more details will appear in future blog posts.

Acknowledgements: My views on this topic were initially shaped by the excellent papers from TTI Chicago group regarding the implicit bias of gradient descent ([Behnam Neyshabur's thesis](https://arxiv.org/pdf/1709.01953.pdf) is a good starting point), and of course by my various coauthors. 
<commit_msg>Update draft.md<commit_after>---
layout: post
title: Is Optimization a Sufficient Language for Understanding Deep Learning?
date:  2019-3-11 13:00:00
author: SA + ?? 
visible: False
---

In this Deep Learning era, machine learning usually boils down to defining a suitable objective/cost function for the learning task at hand, and then optimizing this function using some variant of gradient descent (implemented via backpropagation).  Little wonder that hundreds of ML papers each year are devoted to various aspects of optimization. Today I will suggest that if our goal is mathematical understanding of deep learning, then  the optimization viewpoint is potentially insufficient ---at least in the conventional view:

> **Conventional View (CV) of Optimization**: Find a solution of minimum possible value of the objective, as fast as possible. 

Note that *a priori* it is not obvious if all learning should involve optimizing a single objective. Whether or not this is true for the working of the brain is a longstanding open question in neuroscience. Brain components appear to have been repurposed/cobbled together through various accidents of evolution and the whole assemblage may or may not boil down to optimization of an objective. See [this survey by Marblestone et al](https://arxiv.org/pdf/1606.03813.pdf). 

I am suggesting that deep learning algorithms also have important properties that are not always reflected in the objective value. Current deep nets, being vastly overparametrized, have multiple optima. They are trained until the objective is almost zero, and training is said to succeed if the optimum (or near-optimum) model thus found also performs well on unseen/held-out data ---i.e., *generalizes.* The value of the objective may imply nothing about generalization (see [Zhang et al.](https://arxiv.org/abs/1611.03530)). 

Of course experts will now ask: "Wasn't generalization theory invented precisely for this reason as the "second leg" of machine learning,  where optimization is the first leg?" For instance this theory shows how to add regularizers to the training objective to ensure the solution generalizes. Or that *early stopping* (i.e., stopping before reaching the optimum) or even adding noise to the gradient (e.g. by playing with batch sizes and learning rates) can be preferable to perfect optimization, even in simple settings such as regression. 

However, in practice explicit regularizers  and noising tricks can't prevent deep nets from attaining low training objective even on data with random labels; see [Zhang et al.](https://arxiv.org/abs/1611.03530). Current generalization theory is designed to give *post hoc* explanations for why a particular model generalized. It is agnostic about *how* the solution was obtained, and thus makes few prescriptions ---apart from recommending some regularization--- for optimization.   (See my earlier [blog post](http://www.offconvex.org/2017/12/08/generalization1/), which explains the distinction between descriptive and prescriptive methods, and  that generalization theory is primarily descriptive.) The fundamental mystery is:

> Even vanilla gradient descent (GD) is good at finding models with reasonable generalization. Furthermore, methods to speed up gradient descent (e.g., acceleration or adaptive regularization) can sometimes lead to worse generalization. 

In other words, GD has an innate bias towards finding solutions with good generalization. Magic happens along the GD trajectory and is not captured in the objective value per se. We're reminded of the old adage. 

> The journey matters more than the destination. 

Below to illustrate this viewpoint I describe new papers with rigorous analysis of gradient descent in two simple but suggestive settings. I am barely sketching the results and hope that more details will appear in future blog posts.

Acknowledgements: My views on this topic were initially shaped by the excellent papers from TTI Chicago group regarding the implicit bias of gradient descent ([<NAME>'s thesis](https://arxiv.org/pdf/1709.01953.pdf) is a good starting point), and of course by my various coauthors. 

{'fraction_non_alphanumeric': 0.049933822644687764, 'fraction_numerical': 0.01395740584767176, 'mean_word_length': 4.784272790535838, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 10, 'https://': 8, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1404308', 'n_tokens_mistral': 2002, 'n_tokens_neox': 1775, 'n_words': 1175}