ResAct: Reinforcing Long-term Engagement in Sequential Recommendation with Residual Actor
Long-term engagement is preferred over immediate engagement in sequential
recommendation as it directly affects product operational metrics such as daily
active users (DAUs) and dwell time. Meanwhile, reinforcement learning (RL) is
widely regarded as a promising framework for optimizing long-term engagement in
sequential recommendation. However, due to expensive online interactions, it is
very difficult for RL algorithms to perform state-action value estimation,
exploration and feature extraction when optimizing long-term engagement. In
this paper, we propose ResAct which seeks a policy that is close to, but better
than, the online-serving policy. In this way, we can collect sufficient data
near the learned policy so that state-action values can be properly estimated,
and there is no need to perform online exploration. Directly optimizing this
policy is difficult due to the huge policy space. ResAct instead solves it by
first reconstructing the online behaviors and then improving it. Our main
contributions are fourfold. First, we design a generative model which
reconstructs behaviors of the online-serving policy by sampling multiple action
estimators. Second, we design an effective learning paradigm to train the
residual actor which can output the residual for action improvement. Third, we
facilitate the extraction of features with two information theoretical
regularizers to confirm the expressiveness and conciseness of features. Fourth,
we conduct extensive experiments on a real world dataset consisting of millions
of sessions, and our method significantly outperforms the state-of-the-art
baselines in various of long term engagement optimization tasks.