license:cc-by-4.0 arxiv:1801.00690 arxiv:2107.09645 region:us
---
license: cc-by-4.0
---

# rllte.hub.datasets: Provide test scores and learning cures of various RL algorithms on different benchmarks.


## Support list
| Benchmark | Algorithm | Remark | Reference |
|:-|:-|:-|:-|
|[Atari Games](https://www.jair.org/index.php/jair/article/download/10819/25823)|PPO|**50M**, 💯📊🤖|[Paper]() |
||SAC|**1M**, 💯📊🤖|[Paper](https://proceedings.mlr.press/v164/raffin22a/raffin22a.pdf) |
|[DeepMind Control (Pixel)](https://arxiv.org/pdf/1801.00690)|DrQ-v2|**1M**, 💯📊🤖|[Paper](https://arxiv.org/pdf/2107.09645.pdf?utm_source=morioh.com) |
|[DeepMind Control (State)](https://arxiv.org/pdf/1801.00690)|SAC|**10M**, 💯📊🤖||
||DDPG|**10M**, 💯📊🤖||
|[Procgen Games](http://proceedings.mlr.press/v119/cobbe20a/cobbe20a.pdf)|PPO|**25M**, 💯📊🤖| [Paper](http://proceedings.mlr.press/v139/raileanu21a/raileanu21a.pdf) |
||DAAC|**25M**, 💯📊🤖| [Paper](http://proceedings.mlr.press/v139/raileanu21a/raileanu21a.pdf) |
|[MiniGrid Games](https://github.com/Farama-Foundation/Minigrid)||

- **🐌**: Incoming.
- **(25M)**: 25 million training steps.
- **💯Scores**: Available final scores.
- **📊Curves**: Available training curves.
- **🤖Models**: Available trained models.

## Example

### `.load_scores`
Suppose we want to evaluate algorithm performance on the [Procgen](https://github.com/openai/procgen) benchmark. Here is an example:

``` py title="example.py"
from rllte.hub.datasets import Procgen

procgen = Procgen()
procgen_scores = procgen.load_scores()
print(procgen_scores['ppo'].shape)

# Output:
# (10, 16)
```
For each algorithm, this will return a `NdArray` of size (`10` x `16`) where `scores[n][m]` represent the score on run `n` of task `m`.

### `.load_curves`

Meanwhile, `.load_curves` will return the learning curves by a Python `Dict` like:

``` py
curves = {
    "ppo": {
        "train": {"bigfish": np.ndarray(shape=(Number of seeds, Number of points)), ...}, 
        "eval": {"bigfish": np.ndarray(shape=(Number of seeds, Number of points)), ...}, 
    },
    "daac": {
        "train": {"bigfish": np.ndarray(shape=(Number of seeds, Number of points)), ...}, 
        "eval": {"bigfish": np.ndarray(shape=(Number of seeds, Number of points)), ...}, 
    },
    ...
}
```
A code example for loading curves of the [Procgen](https://github.com/openai/procgen) benchmark:
``` py title="example.py"
from rllte.hub.datasets import Procgen

if __name__ == "__main__":
    # load data
    procgen = Procgen()
    curves = procgen.load_curves()

    print(curves['ppo']['train']['bigfish'].shape)
    print(curves['ppo']['eval']['bigfish'].shape)

# Output:
# (10, 1525)
# (10, 153)
```