<commit_before># RLDS

RLDS stands for Reinforcement Learning Datasets and it is an ecosystem of tools
to store, retrieve and manipulate episodic data in the context of Sequential
Decision Making including Reinforcement Learning (RL), Learning for
Demonstrations, Offline RL or Imitation Learning.

This repository includes a library for manipulating RLDS compliant datasets.
For other parts of the pipeline please refer to:

*   [EnvLogger](http://github.com/deepmind/envlogger) to create synthetic
    datasets
*   [RLDS Creator](http://github.com/google-research/rlds-creator) to create
    datasets where a human interacts with an environment.
*   [TFDS](http://www.tensorflow.org/datasets/catalog/overview) for existing RL
    datasets.

## QuickStart & Colabs

See how to use RLDS in this
[tutorial](https://colab.research.google.com/github/google-research/rlds/blob/main/rlds/examples/rlds_tutorial.ipynb).

You can find more examples in the following colabs:

*   [Performance best practices](https://colab.research.google.com/github/google-research/rlds/blob/main/rlds/examples/rlds_performance.ipynb)
*   [RL examples](https://colab.research.google.com/github/google-research/rlds/blob/main/rlds/examples/rlds_examples.ipynb)

## Dataset Format

The dataset is retrieved as a `tf.data.Dataset` of Episodes.

![drawing](docs/images/rlds.png "RLDS Diagram")

*   **Episode**: dictionary that contains a `tf.data.Dataset` of Steps, and
    metadata.

*   **Step**: dictionary that contains:

    *   `observation`: current observation
    *   `action`: action taken in the current observation
    *   `reward`: return after appyling the action to the current observation
    *   `is_terminal`: if this is a terminal step
    *   `is_first`: if this is the first step of an episode that contains the
        initial state.
    *   `is_last`: if this is the last step of an episode, that contains the
        last observation. When true, `action`, `reward` and `discount`, and
        other cutom fields subsequent to the observation are considered invalid.
    *   `discount`: discount factor at this step.
    *   extra metadata

    When `is_terminal = True`, the `observation` corresponds to a final state,
    so `reward`, `discount` and `action` are meaningless. Depending on the
    environment, the final `observation` may also be meaningless.

    If an episode ends in a step where `is_terminal = False`, it means that this
    episode has been truncated. In this case, depending on the environment, the
    action, reward and discount might be empty as well.

## How to create a dataset

Although you can read datasets with the RLDS format even if they were not
created with our tools (for example, by adding them to [TFDS](#load-with-tfds)),
we recommend the use of [EnvLogger] and [RLDS Creator].

### Synthetic datasets

Envlogger provides a [dm_env] `Environment` class wrapper that records
interactions between a real environment and an agent.

```
env = envloger.EnvironmentLogger(
      environment,
      data_directory=`/tmp/mydataset`)
```

Besides, two callbacks can be passed to the `EnviromentLogger` constructor to
store per-step metadata and per-episode metadata. See the [EnvLogger]
documentation for more details.

Note that per-session metadata can be stored but is currently ignored when
loading the dataset.

Note that the Envlogger follows the [dm_env] convention. So considering:

*   `o_i`: observation at step `i`
*   `a_i`: action applied to `o_i`
*   `r_i`: reward obtained when applying `a_i` in `o_i`
*   `d_i`: discount for reward `r_i`
*   `m_i`: metadata for step `i`

Data is generated and stored as:

```none
    (o_0, _, _, _, m_0) → (o_1, a_0, r_0, d_0, m_1)  → (o_2, a_1, r_1, d_1, m_2) ⇢ ...
```

But loaded with RLDS as:

```none
    (o_0,a_0, r_0, d_0, m_0) → (o_1, a_1, r_1, d_1, m_1)  → (o_2, a_2, r_2, d_2, m_2) ⇢ ...
```

### Human datasets

If you want to collect data generated by a human interacting with an
environment, check the [RLDS Creator].

## How to read your dataset

RL datasets can be loaded with [TFDS](#load-with-tfds)
and they are retrieved with the canonical
[RLDS dataset format](#dataset-format).

### Load with TFDS


#### Datasets in the TFDS catalog

These datasets can be loaded directly with:

```py
tfds.load('dataset_name').as_dataset()['train']
```

See the full documentation and the catalog in the [TFDS] site.

#### Datasets in your own repository

Datasets can be implemented with TFDS both inside and outside of the TFDS
repository. See examples
[here](https://www.tensorflow.org/datasets/external_tfrecord?hl=en#load_dataset_with_tfds).

## How to add your dataset to TFDS

You can add your dataset directly to TFDS
following the instructions at https://www.tensorflow.org/datasets.

* If your data has been generated with Envlogger or the RLDS Creator, you can just use the rlds helpers in TFDS (see [here](https://github.com/tensorflow/datasets/blob/master/tensorflow_datasets/rlds/robosuite_panda_pick_place_can/robosuite_panda_pick_place_can.py) an example).
* Otherwise, make sure your `generate_examples` implementation provides the same structure
  and keys as RLDS loaders if you want your dataset to be compatible with RLDS
  pipelines
  ([example](https://github.com/tensorflow/datasets/blob/master/tensorflow_datasets/d4rl/dataset_utils.py)).


Note that you can follow the same steps to add the data to your own repository
(see more details in the [TFDS documentation](https://www.tensorflow.org/datasets/add_dataset?hl=en)).

## Performance best practices

As RLDS exposes RL datasets in a form of Tensorflow's
[tf.data](https://www.tensorflow.org/api_docs/python/tf/data), many Tensorflow's
[performance hints](https://www.tensorflow.org/guide/data_performance#optimize_performance)
apply to RLDS as well. It is important to note, however, that RLDS datasets are<commit_msg>Small modifications to the README.<commit_after># RLDS

RLDS stands for Reinforcement Learning Datasets and it is an ecosystem of tools
to store, retrieve and manipulate episodic data in the context of Sequential
Decision Making including Reinforcement Learning (RL), Learning for
Demonstrations, Offline RL or Imitation Learning.

This repository includes a library for manipulating RLDS compliant datasets.
For other parts of the pipeline please refer to:

*   [EnvLogger](http://github.com/deepmind/envlogger) to create synthetic
    datasets
*   [RLDS Creator](http://github.com/google-research/rlds-creator) to create
    datasets where a human interacts with an environment.
*   [TFDS](http://www.tensorflow.org/datasets/catalog/overview) for existing RL
    datasets.

## QuickStart & Colabs

See how to use RLDS in this
[tutorial](https://colab.research.google.com/github/google-research/rlds/blob/main/rlds/examples/rlds_tutorial.ipynb).

You can find more examples in the following colabs:

*   [Performance best practices](https://colab.research.google.com/github/google-research/rlds/blob/main/rlds/examples/rlds_performance.ipynb)
*   [RL examples](https://colab.research.google.com/github/google-research/rlds/blob/main/rlds/examples/rlds_examples.ipynb)

## Dataset Format

The dataset is retrieved as a `tf.data.Dataset` of Episodes where each episode
contains a `tf.data.Dataset` of steps.

![drawing](docs/images/rlds.png "RLDS Diagram")

*   **Episode**: dictionary that contains a `tf.data.Dataset` of Steps, and
    metadata.

*   **Step**: dictionary that contains:

    *   `observation`: current observation
    *   `action`: action taken in the current observation
    *   `reward`: return after appyling the action to the current observation
    *   `is_terminal`: if this is a terminal step
    *   `is_first`: if this is the first step of an episode that contains the
        initial state.
    *   `is_last`: if this is the last step of an episode, that contains the
        last observation. When true, `action`, `reward` and `discount`, and
        other cutom fields subsequent to the observation are considered invalid.
    *   `discount`: discount factor at this step.
    *   extra metadata

    When `is_terminal = True`, the `observation` corresponds to a final state,
    so `reward`, `discount` and `action` are meaningless. Depending on the
    environment, the final `observation` may also be meaningless.

    If an episode ends in a step where `is_terminal = False`, it means that this
    episode has been truncated. In this case, depending on the environment, the
    action, reward and discount might be empty as well.

## How to create a dataset

Although you can read datasets with the RLDS format even if they were not
created with our tools (for example, by adding them to [TFDS](#load-with-tfds)),
we recommend the use of [EnvLogger] and [RLDS Creator] as they ensure that the
data is stored in a lossless fashion and compatible with RLDS.

### Synthetic datasets

Envlogger provides a [dm_env] `Environment` class wrapper that records
interactions between a real environment and an agent.

```
env = envloger.EnvironmentLogger(
      environment,
      data_directory=`/tmp/mydataset`)
```

Besides, two callbacks can be passed to the `EnviromentLogger` constructor to
store per-step metadata and per-episode metadata. See the [EnvLogger]
documentation for more details.

Note that per-session metadata can be stored but is currently ignored when
loading the dataset.

Note that the Envlogger follows the [dm_env] convention. So considering:

*   `o_i`: observation at step `i`
*   `a_i`: action applied to `o_i`
*   `r_i`: reward obtained when applying `a_i` in `o_i`
*   `d_i`: discount for reward `r_i`
*   `m_i`: metadata for step `i`

Data is generated and stored as:

```none
    (o_0, _, _, _, m_0) → (o_1, a_0, r_0, d_0, m_1)  → (o_2, a_1, r_1, d_1, m_2) ⇢ ...
```

But loaded with RLDS as:

```none
    (o_0,a_0, r_0, d_0, m_0) → (o_1, a_1, r_1, d_1, m_1)  → (o_2, a_2, r_2, d_2, m_2) ⇢ ...
```

### Human datasets

If you want to collect data generated by a human interacting with an
environment, check the [RLDS Creator].

## How to load a dataset

RL datasets can be loaded with [TFDS](#load-with-tfds)
and they are retrieved
with the canonical [RLDS dataset format](#dataset-format).

See [this section](#how-to-add-your-dataset-to-tfds) for instructions on how to
add an RLDS dataset to TFDS.

### Load with TFDS


#### Datasets in the TFDS catalog

These datasets can be loaded directly with:

```py
tfds.load('dataset_name').as_dataset()['train']
```

This is how we load the datasets in the
[tutorial](https://colab.research.google.com/github/google-research/rlds/blob/main/rlds/examples/rlds_tutorial.ipynb).

See the full documentation and the catalog in the [TFDS] site.

#### Datasets in your own repository

Datasets can be implemented with TFDS both inside and outside of the TFDS
repository. See examples
[here](https://www.tensorflow.org/datasets/external_tfrecord?hl=en#load_dataset_with_tfds).

## How to add your dataset to TFDS

You can add your dataset directly to TFDS
following the instructions at https://www.tensorflow.org/datasets.

* If your data has been generated with Envlogger or the RLDS Creator, you can just use the rlds helpers in TFDS (see [here](https://github.com/tensorflow/datasets/blob/master/tensorflow_datasets/rlds/robosuite_panda_pick_place_can/robosuite_panda_pick_place_can.py) an example).
* Otherwise, make sure your `generate_examples` implementation provides the same structure
  and keys as RLDS loaders if you want your dataset to be compatible with RLDS
  pipelines
  ([example](https://github.com/tensorflow/datasets/blob/master/tensorflow_datasets/d4rl/dataset_utils.py)).


Note that you can follow the same steps to add the data to your own repository
(see more details in the [TFDS documentation](https://www.tensorflow.org/datasets/add_dataset?hl=en)).

## Performance best practices

As RLDS exposes RL datasets in a form of Tensorflow's
[tf.data](https://www.tensorflow.org/api_docs/python/tf/data), many Tensorflow's
[performance hints](https://www.tensorflow.org/guide/data_performance#optimize_performance)
apply to RLDS as well. It is important to note, however, that RLDS datasets are
{'fraction_non_alphanumeric': 0.09366029689165915, 'fraction_numerical': 0.004592799147051587, 'mean_word_length': 4.271941201902291, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 21, 'lorem ipsum': 0, 'www.': 12, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1285147', 'n_tokens_mistral': 3850, 'n_tokens_neox': 3662, 'n_words': 1417}