<commit_before>---
layout:     post
title:      Total Control
date:       2018-01-30 0:00:00
summary:    An outsider tour of reinforcement learning, Part 2. RL as optimal control.
author:     <NAME>
visible:    false
---

In addition to the reasons I’ve discussed so far, I’ve been fascinated with the resurgence in reinforcement learning because it operates at the intersection of two areas I love: machine learning and control. It is amazing how little we understand about this intersection.  And the approaches used by the two disciplines are also frequently at odds.  Controls is the theory of designing complex actions from well-specified models, while machine learning makes intricate, model-free predictions from data alone.

At the core of control theory are dynamical systems with _inputs_ and _outputs_. These systems have internal _state_ which reacts to current conditions and the inputs, and the outputs are some function of the state and the input. If we are to turn off the inputs, the state is all we need to know to predict the future output for all time.

As a simple example, remember Newton’s laws. Say that we want to fly a quadrotor floating at 1 meter off the ground to 2 meters off the ground. In order to raise the position, we’d increase the propeller speed to create an upward force. Propeller speed is an input. The force must be strong enough to counteract gravity which can be considered another input. The dynamics then follow Newton’s laws: the acceleration is proportional to the total force applied minus gravity.  It is inversely proportional to the mass of quadrotor. The velocity is equal to the previous velocity plus some multiple of acceleration. And the position is equal to the previous position plus a multiple of velocity. From these equations we can compute a set of forces that lift the quadrotor up to the desired height. The state of the system here is pair of position and velocity.


{: .center}
![don't try this at home kids](/assets/rl/quad_diagram.png){:width="450px"}

I can write such a dynamical system compactly as a _difference equation_

$$<commit_msg>prepping to go live<commit_after>---
layout:     post
title:      Total Control
date:       2018-02-01 0:00:00
summary:    An outsider tour of reinforcement learning, Part 2. RL as optimal control.
author:     <NAME>
visible:    false
---

In addition to the reasons I’ve discussed so far, I’ve been fascinated with the resurgence in reinforcement learning because it operates at the intersection of two areas I love: machine learning and control. It is amazing how little we understand about this intersection.  And the approaches used by the two disciplines are also frequently at odds.  Controls is the theory of designing complex actions from well-specified models, while machine learning makes intricate, model-free predictions from data alone.

At the core of control theory are dynamical systems with _inputs_ and _outputs_. These systems have internal _state_ which reacts to current conditions and the inputs, and the outputs are some function of the state and the input. If we are to turn off the inputs, the state is all we need to know to predict the future output for all time.

As a simple example, remember Newton’s laws. Say that we want to fly a quadrotor floating at 1 meter off the ground to 2 meters off the ground. In order to raise the position, we’d increase the propeller speed to create an upward force. Propeller speed is an input. The force must be strong enough to counteract gravity which can be considered another input. The dynamics then follow Newton’s laws: the acceleration is proportional to the total force applied minus gravity.  It is inversely proportional to the mass of quadrotor. The velocity is equal to the previous velocity plus some multiple of acceleration. And the position is equal to the previous position plus a multiple of velocity. From these equations we can compute a set of forces that lift the quadrotor up to the desired height. The state of the system here is pair of position and velocity.


{: .center}
![don't try this at home kids](/assets/rl/quad_diagram.png){:width="450px"}

I can write such a dynamical system compactly as a _difference equation_

$$
{'fraction_non_alphanumeric': 0.03784295175023652, 'fraction_numerical': 0.008987701040681173, 'mean_word_length': 4.208128078817734, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4101816', 'n_tokens_mistral': 1025, 'n_tokens_neox': 956, 'n_words': 667}