Most existing deep reinforcement learning (DRL) frameworks consider action spaces that are either
 discrete or continuous space. Motivated by the project of design Game AI for King of Glory
 (KOG), one the world’s most popular mobile game, we consider the scenario with the discrete-continuous
 hybrid action space. To directly apply existing DLR frameworks, existing approaches
 either approximate the hybrid space by a discrete set or relaxing it into a continuous set, which is
 usually less efficient and robust. In this paper, we propose a parametrized deep Q-network (P-DQN)
 for the hybrid action space without approximation or relaxation. Our algorithm combines DQN and
 DDPG and can be viewed as an extension of the DQN to hybrid actions. The empirical study on the
 game KOG validates the efficiency and effectiveness of our method. In recent years, the exciting field of deep reinforcement learning (DRL) have witnessed striking empirical achievements in complicated sequential decision making problems that are once believed unsolvable. One active area of the application of DRL methods is to design artificial intelligence (AI) for games. The success of DRL in the game of Go provides a promising methodology for game AI. In addition to the game of Go, DRL has been widely used in other games such as Atari BID19 , Robot Soccer BID8 BID17 , and Torcs ) to achieve super-human performances.However, most existing DRL methods only handle the environments with actions chosen from a set which is either finite and discrete (e.g., Go and Atari) or continuous (e.g. MuJoCo and Torcs) For example, the algorithms for discrete action space include deep Q-network (DQN) BID18 , Double DQN (Hasselt et al., 2016) , A3C BID20 ; the algorithms for continuous action space include deterministic policy gradients (DPG) BID29 and its deep version DDPG .Motivated by the applications in Real Time Strategic (RTS) games, we consider the reinforcement learning problem with a discrete-continuous hybrid action space. Different from completely discrete or continuous actions that are widely studied in the existing literature, in our setting, the action is defined by the following hierarchical structure. We first choose a high level action k from a discrete set {1, 2, · · · , K}; upon choosing k, we further choose a low level parameter x k ∈ X k which is associated with the k-th high level action. Here X k is a continuous set for all k ∈ {1, . . . , K}.1 Therefore , we focus on a discrete-continuous hybrid action space A = (k, x k ) x k ∈ X k for all 1 ≤ k ≤ K .To apply existing DRL approaches on this hybrid action space, two straightforward ideas include:• Approximate A by an finite discrete set. We could approximate each X k by a discrete subset, which, however, might lose the natural structure of X k . Moreover , when X k is a region in the Euclidean space, establishing a good approximation usually requires a huge number discrete actions.• Relax A into a continuous set. To apply existing DRL framework with continuous action spaces, BID8 define the following approximate space DISPLAYFORM0 where F k ⊆ R. Here f 1 , f 2 , . . . , f K is used to select the discrete action either deterministically (by picking arg max i f i ) or randomly (with probability softmax(f )). Compared with the original action space A, A might significantly increases the complexity of the action space. Furthermore , continuous relaxation can also lead to unnecessary confusion by over-parametrization. For example , (1, 0, · · · , 0, x 1 , x 2 , x 3 , · · · , x K ) ∈ A and (1, 0, · · · , 0, x 1 , x 2 , x 3 , · · · , x K ) ∈ A indeed represent the same action (1, x 1 ) in the original space A.In this paper, we propose a novel DRL framework, namely parametrized deep Q-network learning (P-DQN), which directly work on the discrete-continuous hybrid action space without approximation or relaxation. Our method can be viewed as an extension of the famous DQN algorithm to hybrid action spaces. Similar to deterministic policy gradient methods, to handle the continuous parameters within actions, we first define a deterministic function which maps the state and each discrete action to its corresponding continuous parameter. Then we define a action-value function which maps the state and finite hybrid actions to real values, where the continuous parameters are obtained from the deterministic function in the first step. With the merits of both DQN and DDPG, we expect our algorithm to find the optimal discrete action as well as avoid exhaustive search over continuous action parameters. To evaluate the empirical performances, we apply our algorithm to King of Glory (KOG), which is one of the most popular online games worldwide, with over 200 million active users per month. KOG is a multi-agent online battle arena (MOBA) game on mobile devices, which requires players to take hybrid actions to interact with other players in real-time. Empirical study indicates that P-DQN is more efficient and robust than BID8 's method that relaxes A into a continuous set and applies DDPG. Previous deep reinforcement learning algorithms mostly can work with either discrete or continuous action space. In this work, we consider the scenario with discrete-continuous hybrid action space. In contrast of existing approaches of approximating the hybrid space by a discrete set or relaxing it into a continuous set, we propose the parameterized deep Q-network (P-DQN), which extends the classical DQN with deterministic policy for the continuous part of actions. Empirical experiments of training AI for King of Glory, one of the most popular games, demonstrate the efficiency and effectiveness of P-DQN.
A DQN and DDPG hybrid algorithm is proposed to deal with the discrete-continuous hybrid action space.
{"src": "scitldr-AIC_train", "paper_id": "Sy_MK3lAZ"}