<commit_before>---
layout:     post
title:      "We were promised flying cars, but all we got was endless video conferencing."
date:       2020-06-29 0:00:00
summary:    "Reflections on the recent progress in combining machine learning and control theory to build safe, agile, and autonomous systems."
author:     <NAME>
visible:    false
blurb: 		  false
---

I’m giving a keynote address at the [virtual IFAC congress this July](https://www.ifac2020.org/), and I submitted an abstract that forces me to reflect on the current state of research at the intersection of machine learning and control. 2020 is particularly appropriate for reflection: For personal reasons, I've been working in this space for about half a decade now and [wrote a blog series on the topic two years ago](https://www.argmin.net/2018/06/25/outsider-rl/) and it seemed like ideal timing. For the broader community, 2020 happens to be the year we were promised fleets of self-driving cars. Of course, for a myriad of reasons, we’re nowhere close to achieving this goal. Full self-driving has been a key motivator of work in learning-enabled autonomous systems, and it’s important to note this example as a marker of how difficult problems this space really are.

The research community has come to terms with this difficulty, and has committed itself to address the many pressing challenges. Over the last year I attended several great meetings on this topic, including an [NSF funded workshop at UW](https://ajwagen.github.io/adsi_learning_and_control/), a plenary session at [ITA](https://ita.ucsd.edu/ws/), a workshop on intersections of [learning, control, and optimization at IPAM](https://www.ipam.ucla.edu/programs/workshops/intersections-between-control-learning-and-optimization/?tab=overview), and the [second annual conference on Learning for Dynamics and Control](https://sites.google.com/berkeley.edu/l4dc/home). There is clearly a ton of enthusiasm from researchers in a many different disciplines, and we’re seeing fascinating results mixing techniques from machine learning, computer science, and control. Obviously, I’m going to be leaving out many incredible papers, but, focusing on the theoretical end of the spectrum, perhaps I could highlight [new work on policy optimization](https://arxiv.org/abs/1912.11899) that demonstrates how simple optimization techniques can efficiently solve classic, nonconvex control-design problems or [work connecting regret minimization and adaptive control](https://arxiv.org/abs/1902.08721) that provides nonasymptotic bounds.

This work has been very useful for establishing language to bridge communication between the diverse research camps interested in the space of learning, dynamics and automation. But, as I’ll discuss shortly, I’d argue that it hasn’t provided many promising approaches to improving large-scale autonomy. That’s ok! These problems are incredibly difficult and aren’t going to be solved by wishing for them to be solved. But I also think it might be worth taking a moment to reflect on which problems we are working on. It’s always a bit too easy for theorists to focus  on improving technical results and lose sight of why someone proved those results in the first place.

As an illustrative example, my research group spent a lot of time studying the [Linear Quadratic Regulator](http://www.argmin.net/2018/02/08/lqr/) (LQR). The point of this work was initially to establish baselines: LQR has a closed form solution when the model is known, so we wanted to understand how different algorithms might perform when the underlying model was unknown. It turns out that if you are willing to collect enough data, the best thing you can do for LQR is [estimate the dynamical model, and then exploit this model as if it were true](https://arxiv.org/abs/1902.07826). This so-called “certainty equivalent control” is what practitioners have been doing since the mid-60s to fly satellites and solve other optimal control problems. Proving this result required a bunch of new mathematical insights that established connections between high dimensional statistics and automatic control theory. But it did not bring us closer to solving new challenges in robotics or autonomous systems. Our work here merely showed that what the controls community had been doing for 50 years was already about as well as we could do for this important baseline problem.

So what are the ways forward? Are there things that theory-minded folks can work on short term that might help us understand paths towards improving learning systems in complex feedback loops? Let me suggest a few challenges that I see as both very pressing, but also ones where we might be able to make near-term progress.

## Machine Learning is still not reliable technology

At the aforementioned [IPAM meeting](https://www.ipam.ucla.edu/programs/workshops/intersections-between-control-learning-and-optimization/?tab=overview), <NAME> gave a [fantastic survey of the sorts of standards of reliability imposed in aerospace engineering](https://www.youtube.com/watch?v=Wi8Y---ce28). Go watch it! I don’t want to spoil it for you, but his discussion of <NAME> is gripping. Richard covers what is needed to get to the sorts of reliability we’d like in autonomous systems. Unfortunately, having [88.5% Top-1 accuracy on ImageNet](https://arxiv.org/abs/2003.08237)---while a stunning achievement---doesn’t tell us how to get to systems with failure rates on the order of 1 in a billion. As Boeing has tragically shown, cutting corners on autonomous system safety standards has horrible, tragic consequences.

How can we make machine learning more robust? How can we approach the failure rates needed for safe, reliable autonomy? And how can we establish testing protocols to assure we have such low failure rates?

## Prediction systems in feedback loops
<commit_msg>new title, but it's boring<commit_after>---
layout:     post
title:      "What We've Learned to Control"
date:       2020-06-29 0:00:00
summary:    "We were promised flying cars, but all we got was endless video conferencing. Reflections on the recent progress in combining machine learning and control theory to build safe, agile, and autonomous systems."
author:     <NAME>
visible:    false
blurb: 		  false
---

I’m giving a keynote address at the [virtual IFAC congress this July](https://www.ifac2020.org/), and I submitted an abstract that forces me to reflect on the current state of research at the intersection of machine learning and control. 2020 is particularly appropriate for reflection: For personal reasons, I've been working in this space for about half a decade now and [wrote a blog series on the topic two years ago](https://www.argmin.net/2018/06/25/outsider-rl/) and it seemed like ideal timing. For the broader community, 2020 happens to be the year we were promised fleets of self-driving cars. Of course, for a myriad of reasons, we’re nowhere close to achieving this goal. Full self-driving has been a key motivator of work in learning-enabled autonomous systems, and it’s important to note this example as a marker of how difficult problems this space really are.

The research community has come to terms with this difficulty, and has committed itself to address the many pressing challenges. Over the last year I attended several great meetings on this topic, including an [NSF funded workshop at UW](https://ajwagen.github.io/adsi_learning_and_control/), a plenary session at [ITA](https://ita.ucsd.edu/ws/), a workshop on intersections of [learning, control, and optimization at IPAM](https://www.ipam.ucla.edu/programs/workshops/intersections-between-control-learning-and-optimization/?tab=overview), and the [second annual conference on Learning for Dynamics and Control](https://sites.google.com/berkeley.edu/l4dc/home). There is clearly a ton of enthusiasm from researchers in a many different disciplines, and we’re seeing fascinating results mixing techniques from machine learning, computer science, and control. Obviously, I’m going to be leaving out many incredible papers, but, focusing on the theoretical end of the spectrum, perhaps I could highlight [new work on policy optimization](https://arxiv.org/abs/1912.11899) that demonstrates how simple optimization techniques can efficiently solve classic, nonconvex control-design problems or [work connecting regret minimization and adaptive control](https://arxiv.org/abs/1902.08721) that provides nonasymptotic bounds.

This work has been very useful for establishing language to bridge communication between the diverse research camps interested in the space of learning, dynamics and automation. But, as I’ll discuss shortly, I’d argue that it hasn’t provided many promising approaches to improving large-scale autonomy. That’s ok! These problems are incredibly difficult and aren’t going to be solved by wishing for them to be solved. But I also think it might be worth taking a moment to reflect on which problems we are working on. It’s always a bit too easy for theorists to focus  on improving technical results and lose sight of why someone proved those results in the first place.

As an illustrative example, my research group spent a lot of time studying the [Linear Quadratic Regulator](http://www.argmin.net/2018/02/08/lqr/) (LQR). The point of this work was initially to establish baselines: LQR has a closed form solution when the model is known, so we wanted to understand how different algorithms might perform when the underlying model was unknown. It turns out that if you are willing to collect enough data, the best thing you can do for LQR is [estimate the dynamical model, and then exploit this model as if it were true](https://arxiv.org/abs/1902.07826). This so-called “certainty equivalent control” is what practitioners have been doing since the mid-60s to fly satellites and solve other optimal control problems. Proving this result required a bunch of new mathematical insights that established connections between high dimensional statistics and automatic control theory. But it did not bring us closer to solving new challenges in robotics or autonomous systems. Our work here merely showed that what the controls community had been doing for 50 years was already about as well as we could do for this important baseline problem.

So what are the ways forward? Are there things that theory-minded folks can work on short term that might help us understand paths towards improving learning systems in complex feedback loops? Let me suggest a few challenges that I see as both very pressing, but also ones where we might be able to make near-term progress.

## Machine Learning is still not reliable technology

At the aforementioned [IPAM meeting](https://www.ipam.ucla.edu/programs/workshops/intersections-between-control-learning-and-optimization/?tab=overview), <NAME> gave a [fantastic survey of the sorts of standards of reliability imposed in aerospace engineering](https://www.youtube.com/watch?v=Wi8Y---ce28). Go watch it! I don’t want to spoil it for you, but his discussion of <NAME> is gripping. Richard covers what is needed to get to the sorts of reliability we’d like in autonomous systems. Unfortunately, having [88.5% Top-1 accuracy on ImageNet](https://arxiv.org/abs/2003.08237)---while a stunning achievement---doesn’t tell us how to get to systems with failure rates on the order of 1 in a billion. As Boeing has tragically shown, cutting corners on autonomous system safety standards has horrible, tragic consequences.

How can we make machine learning more robust? How can we approach the failure rates needed for safe, reliable autonomy? And how can we establish testing protocols to assure we have such low failure rates?

## Prediction systems in feedback loops

{'fraction_non_alphanumeric': 0.054937590218221956, 'fraction_numerical': 0.015284028190540885, 'mean_word_length': 4.728599221789883, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 24, 'lorem ipsum': 0, 'www.': 12, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2629527', 'n_tokens_mistral': 2919, 'n_tokens_neox': 2672, 'n_words': 1655}