<commit_before>---
layout:     post
title:      "Even State Feedback Isn't Naturally Robust"
date:       2020-07-23 0:00:00
summary:    "Simple examples of discrete-time optimal control problems where even state feedback pushes you into fragile policies."
author:     <NAME>
visible:    false
blurb: 		  false
---
Doyle's derived his LQG counterexample in the time before the ubiquity of numerical computing. This meant that numerical examples did not carry the rhetorical weight of algebraic closed form instances. The need for clean, persuasive formulae also meant that controllers were idealized in continuous time. Continuous-time optimal control often produced policies that couldn't be implemented because of the limits of physical reality: no system can act instantaneously with arbitrary power. These issues of infeasibility were [certainly noted in the literature](https://ieeexplore.ieee.org/document/1099822/) during the hey day of optimal control, but continuous time models still often made it difficult to pinpoint these issues.

Discrete-time models don't share many of these issues. In discrete time, we explicitly encode the sequential, computational nature of decision and control. Discrete-time formulae are unfortunately less elegant than their continuous-time counterparts, but, as I hope to show here, they are often more revealing. Indeed, constructing examples where discrete-time optimal control leads to fragile solutions seems to be surprisingly easy.

Here, I'll highlight a few examples where relatively innocuous problem formulations lead to very fragile control policies. The examples are weirdly simple and almost comical to a point. But anyone who has played with discrete-time optimal control may have stumbled into similar control policies and had to step back and think about why.

Let's revisit the discrete-time LQR problem:

$$
	\begin{array}{ll} \text{minimize} & \sum_{t=1}^N \mathbb{E}_{w_t}\left[x_t^\top Q x_t + u_t^\top R u_t\right]\\
	\text{subject to} & x_{t+1} = A x_t + B u_t + w_t
	\end{array}
$$

We again assume $x_t$ is observed perfectly without noise. While such perfect state information is not realistic, even ideal state feedback ends up being fragile in discrete time. $w_t$ is assumed to be stochastic, but I don't think much changes if we move to a more adversarial setting. Here, we need the decision variable $u_t$ to be _causal_. It must be a function of only the values $x_s$ and $u_s$ with $s\leq t$. For stochastic disturbances, the optimal $u$ can always be found by dynamic programming.

Consider the following innocuous dynamics:

$$
	A = \begin{bmatrix} 0 & 1\\ 0 & 0\end{bmatrix} \,,~~~ B = \begin{bmatrix} 0\\1 \end{bmatrix}\,,<commit_msg>remove possessive<commit_after>---
layout:     post
title:      "Even State Feedback Isn't Naturally Robust"
date:       2020-07-23 0:00:00
summary:    "Simple examples of discrete-time optimal control problems where even state feedback pushes you into fragile policies."
author:     <NAME>
visible:    false
blurb: 		  false
---
Doyle derived his LQG counterexample in the time before the ubiquity of numerical computing. This meant that numerical examples did not carry the rhetorical weight of algebraic closed form instances. The need for clean, persuasive formulae also meant that controllers were idealized in continuous time. Continuous-time optimal control often produced policies that couldn't be implemented because of the limits of physical reality: no system can act instantaneously with arbitrary power. These issues of infeasibility were [certainly noted in the literature](https://ieeexplore.ieee.org/document/1099822/) during the hey day of optimal control, but continuous time models still often made it difficult to pinpoint these issues.

Discrete-time models don't share many of these issues. In discrete time, we explicitly encode the sequential, computational nature of decision and control. Discrete-time formulae are unfortunately less elegant than their continuous-time counterparts, but, as I hope to show here, they are often more revealing. Indeed, constructing examples where discrete-time optimal control leads to fragile solutions seems to be surprisingly easy.

Here, I'll highlight a few examples where relatively innocuous problem formulations lead to very fragile control policies. The examples are weirdly simple and almost comical to a point. But anyone who has played with discrete-time optimal control may have stumbled into similar control policies and had to step back and think about why.

Let's revisit the discrete-time LQR problem:

$$
	\begin{array}{ll} \text{minimize} & \sum_{t=1}^N \mathbb{E}_{w_t}\left[x_t^\top Q x_t + u_t^\top R u_t\right]\\
	\text{subject to} & x_{t+1} = A x_t + B u_t + w_t
	\end{array}
$$

We again assume $x_t$ is observed perfectly without noise. While such perfect state information is not realistic, even ideal state feedback ends up being fragile in discrete time. $w_t$ is assumed to be stochastic, but I don't think much changes if we move to a more adversarial setting. Here, we need the decision variable $u_t$ to be _causal_. It must be a function of only the values $x_s$ and $u_s$ with $s\leq t$. For stochastic disturbances, the optimal $u$ can always be found by dynamic programming.

Consider the following innocuous dynamics:

$$
	A = \begin{bmatrix} 0 & 1\\ 0 & 0\end{bmatrix} \,,~~~ B = \begin{bmatrix} 0\\1 \end{bmatrix}\,,
{'fraction_non_alphanumeric': 0.06653151492812384, 'fraction_numerical': 0.010320678215997052, 'mean_word_length': 4.492914979757085, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '808348', 'n_tokens_mistral': 1477, 'n_tokens_neox': 1345, 'n_words': 748}