We evaluate algorithms on the continuous control tasks in the D4RL benchmark [1]}.
Due to limited computational resources, we select therein a representative set of datasets.
Specifically,
(1) we select the “medium-expert,” “medium-replay,” and “medium” datasets for the Hopper, HalfCheetah, and Walker2d tasks in the Gym-MuJoCo domain, which are commonly used benchmarks in prior work [2]}, [3]}, [4]}, [5]}.
As in recent literature [6]}, [7]}, [8]}, we do not test on the “random” and “expert” datasets, as they are considered as less practical [9]} and can be simply solved by directly applying standard off-policy RL algorithms [10]} and behavior cloning on the offline dataset.
We note that a data-quality agnostic setting may not align with practical offline RL applications, i.e., one typically should know the quality of the offline datasets, e.g., whether it is collected by random policy or by experts.
(2) Apart from the Gym-MuJoCo domain, we further consider the Maze2D domain of tasksWe use the tasks “maze2d-umaze,” “maze2d-medium,” and “maze2d-large.” for the non-Markovian data-collecting policy and the Adroit tasksWe use the tasks “pen-human,” “pen-cloned,” “pen-expert,” and “door-expert.” [11]} for their high dimensionality and sparse rewards.
