Adversarial Counterfactual Environment Model Learning
A good model for action-effect prediction, named environment model, is
important to achieve sample-efficient decision-making policy learning in many
domains like robot control, recommender systems, and patients' treatment
selection. We can take unlimited trials with such a model to identify the
appropriate actions so that the costs of queries in the real world can be
saved. It requires the model to handle unseen data correctly, also called
counterfactual data. However, standard data fitting techniques do not
automatically achieve such generalization ability and commonly result in
unreliable models. In this work, we introduce counterfactual-query risk
minimization (CQRM) in model learning for generalizing to a counterfactual
dataset queried by a specific target policy. Since the target policies can be
various and unknown in policy learning, we propose an adversarial CQRM
objective in which the model learns on counterfactual data queried by
adversarial policies, and finally derive a tractable solution GALILEO. We also
discover that adversarial CQRM is closely related to the adversarial model
learning, explaining the effectiveness of the latter. We apply GALILEO in
synthetic tasks and a real-world application. The results show that GALILEO
makes accurate predictions on counterfactual data and thus significantly
improves policies in real-world testing.