<commit_before># trfl: Common RL Operations

`trfl` provides commonly used components, especially loss functions, for
training RL models in TensorFlow.

## Background

Common RL algorithms describe a particular update to either a Policy, a Value
function, or an Action-Value (Q) function. In Deep-RL, a policy, value- or Q-
function is typically represented by a neural network (the _model_, not to be
confused with an _environment model_, which is used in _model-based RL_). We
formulate common RL update rules for these neural networks as differentiable
_loss_ functions, as is common in (un-)supervised learning. Under automatic
differentiation, the original update rule is recovered. We find that loss
functions are more modular and composable than traditional RL updates, and more
natural when combining with supervised or unsupervised objectives.

The loss functions and other operations provided here are implemented in pure
TensorFlow. They are not complete algorithms, but implementations of RL-specific
mathematical operations needed when building fully-functional RL agents. In
particular, the updates are only valid if the input data are sampled in the
correct manner. For example, the
[Sequence-Advantage-Actor-Critic loss] (trfl.md#sequence_advantage_actor_critic_loss)
(i.e. A2C) is only valid if the input trajectory is an unbiased sample from the
current policy; i.e. the data are _on-policy_. This library cannot check or
enforce such constraints.

## Example usage

```python
import tensorflow as tf<commit_msg>Small fix to documentation<commit_after># trfl: Common RL Operations

`trfl` provides commonly used components, especially loss functions, for
training RL models in TensorFlow.

## Background

Common RL algorithms describe a particular update to either a Policy, a Value
function, or an Action-Value (Q) function. In Deep-RL, a policy, value- or Q-
function is typically represented by a neural network (the _model_, not to be
confused with an _environment model_, which is used in _model-based RL_). We
formulate common RL update rules for these neural networks as differentiable
_loss_ functions, as is common in (un-)supervised learning. Under automatic
differentiation, the original update rule is recovered. We find that loss
functions are more modular and composable than traditional RL updates, and more
natural when combining with supervised or unsupervised objectives.

The loss functions and other operations provided here are implemented in pure
TensorFlow. They are not complete algorithms, but implementations of RL-specific
mathematical operations needed when building fully-functional RL agents. In
particular, the updates are only valid if the input data are sampled in the
correct manner. For example, the
[Sequence-Advantage-Actor-Critic loss](trfl.md#sequence_advantage_actor_critic_loss)
(i.e. A2C) is only valid if the input trajectory is an unbiased sample from the
current policy; i.e. the data are _on-policy_. This library cannot check or
enforce such constraints.

## Example usage

```python
import tensorflow as tf
{'fraction_non_alphanumeric': 0.045572916666666664, 'fraction_numerical': 0.0006510416666666666, 'mean_word_length': 5.158316633266533, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2521643', 'n_tokens_mistral': 769, 'n_tokens_neox': 724, 'n_words': 429}