Inspired by the Dyna-Q framework [1]}, [2]} propose Deep Dyna-Q (DDQ) to deal with large-scale RL problems with deep learning models. As shown in Figure REF  (Right), DDQ allows a bot to optimize the policy by interacting with both human users and user simulators. Training of DDQ consists of three parts:
