
Disagreement [1]}: an ensemble of models is trained to match the environment dynamics. The variance of the ensemble predictions is used as the curiosity signal.

Intrinsic Curiosity Model (ICM; [2]}): intrinsic rewards are computed as the mean-squared error (MSE) between a dynamics model's predictions in feature space and the true features. States are processed into features using a feature network, trained jointly with the model to optimize an inverse-dynamics objective.

Random Network Distillation (RND; [3]}): features are obtained with a fixed randomly initialized neural network. Intrinsic rewards for each transition are the prediction errors between next-state features and the output of a distillation network, trained to match the outputs of the random feature network.

Variational Information Maximizing Exploration (VIME; [4]}): the dynamics is modeled as a Bayesian neural network (BNN; [5]}). Intrinsic rewards for single transitions are shaped as the information gain computed with respect to the BNN's parameters before and after updating the network, using the new transition's data.

Random: an agent that explores by performing a series of random actions. Note that employing random actions is equivalent to having a policy with maximum entropy of actions, for each state. Thus, despite its simplicity, the random baseline provides a metric of how in-depth do maximum entropy RL methods explore, when receiving no external rewards [6]}.

