Inventory Management with Attention-Based MetaActions
Keisuke Izumiya∗ and Edgar Simo-Serra∗
∗Department of Computer Science and Communications EngineeringWaseda University, Tokyo, JapanEmail: k-izumiya@ruri.waseda.jp, ess@waseda.jp
Abstract—Roguelike games are a challenging environmentfor Reinforcement Learning (RL) algorithms due to having torestart the game from the beginning when losing, randomizedprocedural generation, and proper use of in-game items beingessential to success. While recent research has proposed roguelikeenvironments for RL algorithms and proposed models to handlethis challenging task, to the best of our knowledge, none havedealt with the elephant in the room, i.e., handling of items. Itemsplay a fundamental role in roguelikes and are acquired duringgameplay. However, being an unordered set with a non-ﬁxedamount of elements which form part of the action space, it is notstraightforward to incorporate them into an RL framework. Inthis work, we tackle the issue of having unordered sets be part ofthe action space and propose an attention-based mechanism thatcan select and deal with item-based actions. We also propose amodel that can handle complex actions and items through a metaaction framework and evaluate them on the challenging game ofNetHack. Experimental results show that our approach is ableto signiﬁcantly outperform existing approaches.Index Terms—reinforcement learning, attention, NetHack
I. INTRODUCTION
Designing autonomous agents to play video games can playan important role in game balancing, testing, and design.Furthermore, video games play an important role in devel-oping robust Reinforcement Learning (RL) algorithms thatcan then be applied to other real-world situations. In thiswork, we tackle the challenging game of NetHack, a roguelikegame based on procedurally generated content, and developa general attention-based approach to handle unordered setsof actions, such as inventory management, with autonomousagents.RL, a branch of machine learning, is a method that has thegreat advantage of being able to learn without directly usingthe label data. In particular, Deep Reinforcement Learning(DRL), which incorporates deep learning techniques, has beenactively researched in recent years and is often applied togames such as Atari 2600 [1], Go [2], and StarCraft II [3].Among games, roguelikes are suitable as a challenging targetproblem for DRL because they contain elements that currentDRL methods have not entirely overcome, such as a hugestate space, sparseness and delay of rewards, and the need forstrategies. According to [4], roguelikes are turn-based games,and their grid-based environments are randomly procedurallygenerated each time the game begins. The variety of enemiesand items makes the game complex enough that there aremultiple ways to complete the game. The required tasks
include resource management, combat with large numbers ofenemies, and exploration.NetHack, which is the target problem of this research, isone of the most popular open-source roguelikes, and it is stillbeing updated even though it is one of the earliest roguelikes.The game’s objective is to search through over 50 levelsof procedurally generated dungeons using various items toﬁnd the Amulet of Yendor and bring it back. It is incrediblychallenging even for humans because there is a wide variety ofenemies, items, and actions the player can take. Furthermore,the dungeons are not straight paths but rather branching pathsthat must be traversed back and forth. Examples of NetHackscreens are shown in Figure 1. The player can use fourmain types of information: messages, dungeon, status, andinventory. Messages show events and conﬁrmation messages.The dungeon shows the ﬂoor where the player @ is. Mostﬂoors consist of square rooms and passages # (top ﬁgure), butsome ﬂoors do not (middle ﬁgure). An empty area indicatesthat nothing exists or the area has not been explored. The statusindicates the player’s attributes such as strength, experience,intelligence, and hunger. The inventory shows the items theplayer possesses (bottom ﬁgure).As described in Section II, there are several studies onusing DRL for roguelikes. However, most of these studiesfocus on constructing the environment itself or on learningin non-general situations. It is important that, to the best ofour knowledge, there is no study that deals with items inthe general situation that exist in most role-playing games,including roguelikes, and whose use is essential. Therefore,we propose a mechanism that handles items appropriately andshow its effectiveness in experimental results1.Our main contributions are as follows:
• We propose an attention-based mechanism for handlingan unordered set of items in a deep reinforcement learningframework.
• We develop a new agent model that signiﬁcantly outper-forms existing approaches in terms of in-game score onthe challenging game of NetHack.
• We perform an in-depth evaluation of the proposed ap-proach and compare with existing approaches.
1The source code is available at https://github.com/izumiya-keisuke/inventory-management
Message
Player
Passage
Stairs
Status
Dungeon
Fig. 1. Examples of NetHack gameplay screens. The top two images showexamples of the main in-game screen, while the bottom images shows anexample of the inventory screen. Different parts of the screens are annotatedfor ease of understanding.
II. RELATED WORK
A. Reinforcement Learning Algorithms
Mnih et al. [5] proposed a method that does not use experi-ence replay but instead uses data collected by running multipleagents in parallel, represented by asynchronous advantageactor-critic (A3C). In addition to A3C, there are several othermethods belonging to the actor-critic method, such as Actor-Critic with Experience Replay (ACER) [6], UNsupervisedREinforcement and Auxiliary Learning (UNREAL) [7], whichintroduces auxiliary tasks in addition to experience replay, andimportance weighted actor-learner architecture (IMPALA) [8],which can be extended to thousands of machines by elaborat-ing on the data communication method and off-policy correc-tion. Our approach is based on IMPALA with modiﬁcations tebe able to do inventory management. The details of IMPALAare shown in Section III-A.
B. Reinforcement Learning for Games
Most of the modern RL algorithms are evaluated on Atari2600 games provided by ALE [9] or the environments pro-vided by OpenAI Gym [10] as a benchmark. For more populargames, some studies tackled perfect information games suchas Go and Chess [2], [11]. There have also been studies onmultiplayer imperfect information games, such as StarCraftII [3] and Dota 2 [12]. These methods combine the DRLmethods with MCTS [13]. There is also the research [14] thatused an environment based on Minecraft [15], [16], a gamethat uses items similar to NetHack. In this research we focuson roguelike game of Nethack.
C. Reinforcement Learning for Roguelikes
Some research has focused on tackling roguelikes withRL. To facilitate RL’s application, highly customizable en-vironments based on Rogue, the game that originated theroguelike genre and a simpler game than NetHack, havebeen created [17], [18]. With these environments, DRL tech-niques aimed at the exploration of the dungeon [17]–[21](the exploration of the current ﬂoor and descending to thedeeper ﬂoor) have been proposed. For NetHack, the subjectof this study, some recent research has created environmentsbased on NetHack and tackled various tasks. Campbell andVerbrugge [22] aimed at learning enemy combat with anabstracted state and action space. They also focused on ﬁndinghidden doors and passages, which is essential for dungeonexploration, and proposed a method using occupancy maps toachieve the goal efﬁciently. Küttler et al. [23] tackled severaltasks such as scoring a game and collecting gold in a generalsituation by using random network distillation, which is one ofthe exploration facilitation techniques. Most of these methodsdo not take into account items, which are almost essential forcompleting the game, and this is one of the issues to be solved.In this research, we propose a technique which is able performinventory management based on attention allowing for signif-icant performance improvements over existing approaches.
D. Attention
Attention is a mechanism widely used in deep learningwhere the model can learn which parts of the data andfeatures to pay attention to has been actively studied in naturallanguage processing. In the past, attention had often used incombination with an RNN, as in [24], but many methodsusing attention alone have been proposed and showed betterperformance than existing methods since the appearance ofTransformer models [25], which used self-attention withoutRNN. Now models that use self-attention are commonlyemployed in the ﬁeld of image recognition, where ConvolutionNeural Network (CNN) have been commonly used [26]. In theﬁeld of RL, Berner et al. [12] used an attention mechanism tochoose the target unit, and Zambaldi et al. [27] used self-attention to describe the relationship between entities. Wefocus on the position-independent property of attention andpropose a mechanism to handle inventory using this property.
III. METHOD
A. Reinforcement Learning Background
RL is often modeled by Markov Decision Processes (MDP).The state space and the action space are denoted as S and A,respectively. The state at the discrete time t = 0, 1, 2, . . . isdenoted as St ∈ S. The action At ∈ A the agent takes attime t brings a reward Rt and a new state St+1. The goal ofRL is to learn the policy π(a | s) = Pr{At = a | St = s}that maximizes the expected return Eπ��
t γtRt�, where γ isa discount factor.We base our work on IMPALA [8], which is an asyn-chronous off-policy actor-critic method. As in IMPALA, weprepare multiple independent agents, actors, and environmentsand update the parameters of the learning agent, learner, usingthe data collected by these agents. We use an off-policycorrection method called V-trace because each actor has itsown local policy µ, which lags behind the learner’s policy π.At the state s, the agent outputs the state value V (s), whichestimates the return from the state s, and a policy π(a | s).The gradient of the loss function for the policy parameter is
ρt∇ log π(At | St)�Rt + γvt+1 − V (St)�
����policy gradient term
+ β ∇�
a∈A−π(a | St) log π(a | St)
����entropy term
vt = V (St) +
t+n−1�
s=tγs−t�s−1�
δs,(2)
δt = ρt�Rt + γV (St+1) − V (St)�,(3)
ρt = min�
ρ, π(At | St)
µ(At | St)
ct = min�
c, π(At | St)
µ(At | St)
β, n, ρ, and c are hyperparameters. The ﬁrst term in (1) is apolicy gradient, and the second one is entropy. The gradientof the loss function for the state value parameter is�vt −V (St)�∇V (St).
B. Baseline Model
As a baseline, we use a slightly modiﬁed version of themodel used in [28], which is an unsurpassed model to ourbest knowledge. The diagram of the model is shown abovethe dashed line in Figure 2. The model can handle three ofthe four types of main information as described in Section I,all excluding the inventory. Since a message is a string of256 characters in length, each character is embedded with itsASCII code, and then the features of the message are extractedby a 1D CNN. Although this is an outdated method in the ﬁeldof natural language processing, we use it because it is one ofthe simplest models and message processing is not the focusof our experiments. Status is a vector consisting of numerical
values such as HP and strength, and the feature is extractedby a multilayer perceptron (MLP). The dungeon’s shape is21 × 79, and each grid is a vector embedded with attributessuch as its color and its kind. We extract the feature of thedungeon with a 2D CNN. In addition, the feature of 9 × 9grids centered on the agent is extracted in the same way asthe entire dungeon. This has been validated by [29] and [30]to help train the agent. These features are then concatenatedand input to the MLP and Gated Recurrent Unit (GRU) in thatorder. h and h′ represent the hidden state at the previous timeand at the current time, respectively. Finally, the GRU output,which can be regarded as the feature of the current state, isinput to two MLPs to obtain the state value and the policy.
C. Action Recursion
We add the embedding layer to the model to incorporateknowledge of the previous action.In this layer, the agent’sprevious action is embedded and concatenated with the maininformation’s features. It is intuitively that a player mayplan a sequence of actions. In particular, in NetHack, someactions are often repeated multiple times, and others consistof multiple actions. An example of the former is the searchfor hidden passages or doors. If there are hidden passages ordoors in grids adjacent to the agent, it can ﬁnd them witha certain probability by using the search action. Therefore,a common strategy is to repeat the search action a certainnumber of times to ﬁnd them. An example of the latter iskicking and actions which use items. Kicking is composedof two actions: the kick action itself and a kicking direction,while the item-using action is composed of two actions: thecontext of the action such as quaff and read and the item tobe used. Therefore, we need to input the actions that the agenthas taken before into the model. Although it is theoreticallypossible for the model to learn to propagate this informationthrough the hidden state of the GRU layer, we found that inpractice, this effect did not occur and that it was beneﬁcial todirectly use the previous action as an input.It is possible to treat all pairs of actions and their targets asseparate actions to handle actions that require multiple inputs,but we do not do so because this increases the size of the actionspace and makes learning difﬁcult. In NetHack, for example,there are more than ten actions using items and more thanﬁfty types of items that can be held, so treating each potentialcommand pair as an action makes the size of the action spacebeyond 500. Also, there is room for the number of actionsinput to the model, but we decide to use only the previousaction because it should possible to consider any number ofactions thanks to GRU. Note that we embed the action withthe action space A′, which is deﬁned in Section III-F.
D. Meta Actions
In NetHack, specifying an item is limited to a few situations,such as selecting a potion to quaff immediately after selectinga quaff action and selecting an armor to wear immediatelyafter selecting a wear action. In these situations, it is ofprimary importance to specify an item. Therefore, we add the
Dungeon
Dungeon (crop) 
Status
Message
ConcatGRU
State value V(s)
Policy π(a | s)
ActionEmb
Item 1‧ ‧ ‧ 
Item N
Attention
‧ ‧ ‧ 
EmbEmbedding layer
MLPMLP
CNNCNN
ConcatConcatenating layer
GRUGRU
SumSummation layer
Attention Attention layer
Fig. 2. Overview of our agent model architecture. The baseline model is shown above the dashed line. The whole model represents our proposed modelwhere we use attention-based inventory management framework. We note that there are still some other differences in low-level between our full model andthe baseline, such as the usage of meta actions.
meta action use item to the action space. When the agentchooses this action, it speciﬁes the item according to theprobability exp(yi)/ �
i exp(yi) where yi is the ith item’sscore. The method of calculating the score is described inSection III-E.The introduction of the meta action also has the advantagethat items can be handled independently from normal actions.It is not easy for policy to incorporate items directly becausethe number of items is not constant. Using meta actionsmakes the size of the action space ﬁxed and makes theimplementation easier. Furthermore, we can handle the normalaction space and the action space of inventory separately. Wepropose the computation of the entropy term in the loss func-tion described in Section III-F as one example of separating.Besides, for the same reason as described in Section III-C, itcan prevent the action space from becoming too large.
E. Attention-Based Inventory Feature Extractor
It is not easy to handle the inventory appropriately withneural networks because it is an unordered set of items. Toexpress the unordered nature, we propose the attention-basedmechanism to extract the feature of the inventory. First, eachitem in the inventory is transformed into a vector xi using theembedding and the MLP. This embedding is done in a similarway to the embedding of each grid in the dungeon. Then, thesum of vectors of all items, x = �
i xi, is input to anotherMLP, and the output is deﬁned as the feature of the inventory.Note that the summation does not depend on the order ofthe items. The inventory feature is concatenated with othermain information features and the embedded previous action,and the feature representation is processed in the same way
as the baseline model. Then, to specify the items to be used,each item’s score is calculated using the attention operation.Speciﬁcally, we denote the feature of the current state as fand prepare the matrices WQ and WK and the vector wV,and calculate each item’s score yi by the following equation:
yi = q⊤ki√dkvi,(6)
where q = WQf, ki = WKxi, vi = w⊤Vxi, and dk is thedimension of q and ki.The feature vector is then used with the item meta actionsto determine what item to use while being invariant to theorder of the items.
F. Loss Function
The use of the meta action and the attention-based itemselection policy necessitates modifying the loss function. First,we deﬁne modiﬁed action spaces and policies. We denote theuse item action as b0 and the action using the ith item as bi.The virtual action space is deﬁned as Av = A ∪ {b0}, theaction space for the inventory is deﬁned as Ai = {b1, b2, . . .},and the actual action space is deﬁned as A′ = A ∪ Ai.Similarly, we deﬁne the virtual policy as πv : S ×Av → [0, 1],the policy for the inventory as πi : S × Ai → [0, 1], and theactual policy as π′ : S × A′ → [0, 1]. Note that these policiessatisfy the following relations for all s ∈ S:
π′(a | s) =
�πv(a | s)for a ∈ A,πv(b0 | s)πi(a | s)for a ∈ Ai,(7)
a∈Avπv(a | s) =�
a∈Aiπi(a | s) =�
a∈A′π′(a | s) = 1.(8)
The RL algorithm used in this study is IMPALA [8]. Amongthe gradient of the loss function for the policy parameter, ascan be seen in (1), two modiﬁcations are applied: all π arereplaced with π′ in the policy gradient term, and the entropyterm is replaced with
a∈Av−πv(a | St) log πv(a | St)
+ λπv(b0 | St)∇�
b∈Ai−πi(b | St) log πi(b | St),(9)
where λ is a hyperparameter. When λ = 1, it means thatall actions belonging to A′ are treated equally in the entropycalculation, and (9) is equal to ∇ �
a∈A′ −π′(a | St) log π′(a |St).
IV. EXPERIMENTS AND RESULTS
A. Experiment Settings
In this study, we used the in-game score as a reward,which shown in the status bar at the bottom as seen inFigure 1. In NetHack, the in-game score can be earned byvarious events, such as descending to a new level, defeatingenemies, or getting gold. We used this setting because itcontains many elements necessary to complete the game andconsiders combat with enemies, which is often done with theuse of items, for which our model is explicitly designed tohandle. Furthermore, previous research described in Section IIrestrict the actions that agents can take to movement only ormovement and a few other actions. In this study, however,we used an action space covering most of the actions directlyrelated to the game.We compared against the approach of Küttler et al. [28],which is based on IMPALA [8] and TorchBeast [31] and usesa limited amount of actions. In particular, only movementin 8 directions, climbing up/down, reading messages, eating,searching, and kicking are allowed for a total 14 differentactions. Our baseline extended the amount of actions byconsidering an additional 11 actions (apply, drop, pickup,puton, quaff, read, takeoff, throw, wear, wield, and zap) inaddition to meta actions. Our proposed approach extendedthe baseline with action recursion, meta action, and attention-based inventory feature extraction. We trained agents for allapproaches for one billion steps. In addition, we conductedthe tests over ten episodes with the same seed after training.Most evaluation on NetHack was done using the characteras mon-hum-neu-mal (indicating that the role is monk, therace is human, the alignment is neutral, and the gender ismale). The game content in NetHack varies greatly depend-ing on the characters, especially on the roles. Therefore, inaddition to mon-hum-neu-mal, two other characters withdifferent characteristics were used in the experiment. Theadditional characters were val-dwa-law-fem (valkyrie,dwarf, lawful, and female) and tou-hum-neu-fem (tourist,human, neutral, and female), and the characteristics of the usedcharacters are as follows:
TABLE IAVERAGE SCORE OVER TEN EPISODES OF TESTING. WE SHOW RESULTSFOR DIFFERENT APPROACHES AND THREE DIFFERENT CHARACTERSCENARIOS. BEST RESULTS ARE SHOWN IN BOLD.
MonkValkyrieTourist
[28]807.1645.442.2Baseline1431.2686.256.8Ours w/o Inventory1348.4840.9191.1Ours w/o Action Recursion1500.1890.5153.0Ours2345.0906.7283.7
Monk2000
000.51 (x109)
Valkyrie
000.51 (x109)
Tourist
00.5(x109)1
Baseline
w/o Inventory
w/o Recursion
Fig. 3.Average return during training. The horizontal axis shows thetraining step, and the vertical one shows the average return of all agents.
• monk: strong at the beginning of the game, with variousitems and high combat performance, but more challeng-ing to conquer from the middle of the game.
• valkyrie: has very few items at the beginning of thegame, but the character has high combat performance.
• tourist: has various items but is very weak at the begin-ning of the game, so tactics speciﬁc to this character isrequired.We evaluated all approaches independently in each of the threecharacter settings.
B. Comparison against Existing Approaches
We compared against existing approaches and testing resultsare summarized in Table I and the training evolution is shownin Figure 3. We can see that the proposed approach is able tosigniﬁcantly outperform existing approaches. In particular, wecan see the largest relative increase in the most challengingclass Tourist, which does not start out with any combat skillsand must rely on inventory usage to stay alive.
C. Ablation Study
We compared four models’ performance: the baselinemodel, the model without action recursion, the model withoutthe attention-based mechanism for the inventory, and the fullproposed model. Figure 3 shows the average return duringtraining, and Table I shows the average return in the test. Wecan see that all the different components of our approach playan important role in obtaining good results.
TABLE IIAVERAGE NUMBER OF TIMES THE AGENT TOOK THE ACTION OF USINGITEMS WITHIN AN EPISODE. RESULTS ARE SHOWN FOR THE POLICYLEARNED WHEN PLAYING WITH THE CLASS OF Monk. ACTIONS THATHAVE NOT BEEN TAKEN EVEN ONCE ARE OMITTED.
dropeatquaffread
Baseline0.04.01.00.2Ours w/o Inventory0.06.82.60.0Ours w/o Action Recursion0.04.02.70.0Ours0.66.22.71.7
D. Effect of Difﬁculty
We can see that the difﬁculty of the game plays a fundamen-tal role in the total score obtained. As Monk does not rely onitems and obtains strong intrinsic abilities when leveling up,it shows the highest performance overall. Valkyrie and Touristrely heavily on items to stay alive and show lower performanceoverall.
E. Analysis of the Policy
We show characteristic examples of the learned policy ofour model in Figure 4. The ﬁgures with black backgroundshow the state of the game, and the probabilities on the rightside show the actions the agent will take in this state and theirprobabilities. In the top three ﬁgures, the three actions with thehighest probability in the action space Av are shown. In thebottom ﬁgure, the transitions of probabilities of the two actionswith the highest probability, search and North, are shown untilthe agent takes search action ﬁve times in a row and ﬁnds thehidden door. In addition, the inventory screens are shown inthe blue boxes and the probability of using each item, or πi, isshown to the left of each item in the top two examples becausethe action with the highest probability is use item.The ﬁrst example shows the screen immediately after theagent takes the eat action, and the message at the top of thescreen asks the agent which item to eat. In the second example,the agent is in the state immediately after taking the quaffaction and is asked the item to quaff. The third example showsthe state immediately after taking the kick action, and the agentis asked in which direction to kick. The fourth example showsthe screen immediately after the agent moves to the left by theWest action.We can see that in many cases, the agent is able to learn touse items depending on the game state to stay alive. We cansee that the attention is able to understand the different usageof the types of items and is invariant to the order of the items.
V. DISCUSSION AND CONCLUSION
We have presented an attention-based approach that isable to handle unordered sets, such as item inventories, inreinforcement learning models. We evaluate on the challengingbenchmark that is the roguelike game NetHack and haveshown that our proposed inventory management framework,in combination with our handling of meta actions and actionrecursion, our approach is able to signiﬁcantly outperformexisting approaches.
0.0400.0050.000
use item 0.956  West 0.028  NorthEast 0.003 
South 0.983  kick 0.011  East 0.004 
search  0.957  0.717  0.717  0.679  0.616 
North  0.030  0.214  0.219  0.250  0.304 
0.0000.0000.0000.0000.0000.9810.0090.0000.0000.0000.0090.000
use item 0.986  eat 0.007  quaff 0.002 
Player
Player
Player
0.0000.0000.0200.0000.0000.000
0.3780.0000.0000.0000.0000.0000.000
Fig. 4. Examples of the policy output of the proposed model. The rightside shows the probabilities of taking actions that belong to Av with highprobabilities. In the top two ﬁgures, the blue boxes represent the inventoryscreens, and the probabilities to the left of the box represent πi. In the bottomﬁgure, the transitions of the probabilities are shown.
TABLE IIITHE ITEM SPECIFIED BY THE ACTION THAT USES ITEMS. UP TO THREEITEMS ARE SHOWN IN ORDER OF THE NUMBER OF TIMES THEY HAVE BEENUSED. THE NUMBER IN PARENTHESES INDICATES THE AVERAGE NUMBEROF TIMES THE ITEM WERE USED. RESULTS ARE FOR THE CLASS OF Monk.
ActionUsed Items
dropunlabeled scroll (0.5), unknown potion (0.1)eatfood ration (3.0), apple (1.5), orange (1.0)quaffpotion of healing (2.7)readspellbook (1.3), unknown scroll (0.2), unlabeled scroll (0.2)
As shown in Figure 3 and Table I, the proposed modeloutperforms the existing model. The ﬁrst example in Figure 4shows that the agent was able to use items appropriatelybecause it selected edible items as targets to eat with a veryhigh probability at the right time. The bottom two examplesshow the effect of action recursion. In the second example, theagent was ﬁghting two enemies and had very little HP left.Therefore, the possible effective tactics are to ﬂee or quaffa potion of healing to recover HP, and this example showsthat the agent chose the latter. In the third example, the agentwas able to take kick action appropriately that requires twoconsecutive actions: kick and the direction to kick. The door
+ on the south of the agent @ can be opened only by kickingat the stage because the door is locked and the agent doesnot have a key, so kicking is one of the appropriate actionsto take here. In the last example, some rooms likely exist onthe dungeon’s left side because it is unexplored there, but it isdifﬁcult to imagine a path to the left side based on the currentlyobservable dungeon. Therefore, it is natural to assume thatthere is a hidden door somewhere in the leftmost part of theexplored area where the agent @ is now, and it is appropriateto search for it. In fact, the agent found the hidden door on theleft side after ﬁve search actions. Also, since the search actionsearches around the agent, the agent cannot ﬁnd the door ifit exists in the room’s upper left area. Therefore, one of thenatural strategies is to search upward again if the door is notfound after a certain amount of search actions. The probabilitytransitions of the search and North actions indicate that thisstrategy is being taken.The ablation study shown in Table II indicates that it isnecessary to incorporate both of mechanisms to use itemsappropriately. The most frequently performed actions were eatand quaff; both are directly related to increasing the agent’ssurvival time. The character we used, Monk, always has somefood and three potions of healing at the beginning of thegame. In NetHack, a hunger level progresses with actions, andhigh hunger level makes the agent’s combat power weak andeventually causes death by starvation. Therefore, eating foodis essential for the agent to survive for a long time and to beable to get a higher reward. A potion of healing also helpsthe agent’s survival since quafﬁng it restores HP. Besides, theproposed model also took read actions. Monk starts the gamewith a random scroll and is trained to read it if it is a scroll ofenchant armor. This is because reading it enhance the agent’s
AC, i.e. defense and evasion, which is advantageous in combat,where most agent deaths occur. The list of items used is givenin Table III.On the other hand, other actions of using items were notlearned. Many of them have a temporary disadvantage whentaken or use items that the Monk does not have at the beginningof the game, i.e., items that need to be picked up to use. Inaddition, the identity of items that the agent does not have atthe beginning of the game is unknown. It can only be revealedby limited actions such as using the item or using a speciﬁcitem. However, some items have disadvantageous effects, andthe Monk does not have identifying items initially, so learningthese actions is difﬁcult.The model without the action recursion or inventory han-dling mechanism successfully took actions of using items acertain number of times. As mentioned earlier, an action touse an item usually consists of multiple inputs, so it seemsimpossible for a model that does not use action recursion tolearn this kind of action. However, a message often indicateswhat input is required for the second action, as shown inFigure 4. Therefore, although it is not easy to learn messages,which are natural language, it is possible to learn actions thatrequire multiple inputs without action recursion. However, weobserved that the model often failed to take appropriate theseactions in the test. Also, the use of items is learned even thoughthe model does not use the inventory feature. In NetHack,items are speciﬁed by the alphabetic characters shown on theleft of the inventory screen in the bottom ﬁgure in Figure 1.Therefore, it is possible to learn using items that the agent hasat the beginning of the game without the inventory feature iftheir alphabets are used to specify standard actions.Figure 3 and Table I show that the proposed method’seffectiveness is large for the Monk and the Tourist and smallfor Valkyrie. The Monk and Tourist have many items at thebeginning of the game, but the Valkyrie has few items. Asmentioned earlier, it is not easy to learn to use items thatit does not possess at the beginning of the game, which isthe reason why the proposed method is not very effective forValkyrie.Although our model allows for inventory usage, the agentstill has issues with long-term planning and is unable to learnto equip new equipment. This is also partially due to the factthat in NetHack, equipment is not always a strict upgrade,it can be cursed and give large penalties in many situations.Detecting if equipment is cursed or uncursing it is a hardendeavour which discourages the agent from learning suchcomplex behaviours. We hope that with further developmentsin reinforcement learning that it may be possible to overcomesuch issues.In conclusion, we have shown that our proposed modelis able to properly use items, which are essential elementsin role-playing games, including roguelikes. As a result, thein-game score increased signiﬁcantly. On the other hand, theproposed model could not sufﬁciently learn the use of itemsthat were not possessed at the beginning of the game becausetheir identities were unknown at ﬁrst. Also, our proposed
model did not consider spells, which is a common elementin many of role-playing games. Spells often have the samefeature: players can cast multiple spells, and casting themconsumes a shared resource. The proper handling of theseremains as future work.
REFERENCES
[1] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wier-stra, and M. Riedmiller, “Playing atari with deep reinforcement learn-ing,” in NIPS Deep Learning Workshop, 2013.[2] D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. VanDen Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam,M. Lanctot et al., “Mastering the game of go with deep neural networksand tree search,” nature, vol. 529, no. 7587, pp. 484–489, 2016.[3] O. Vinyals, I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik,J. Chung, D. H. Choi, R. Powell, T. Ewalds, P. Georgiev et al., “Grand-master level in starcraft ii using multi-agent reinforcement learning,”Nature, vol. 575, no. 7782, pp. 350–354, 2019.[4] International Roguelike Development Conference, “Berlin interpre-tation,”2008,accessed:March14th,2021.[Online].Available:http://www.roguebasin.com/index.php?title=Berlin_Interpretation[5] V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley,D. Silver, and K. Kavukcuoglu, “Asynchronous methods for deepreinforcement learning,” in Proceedings of The 33rd International Con-ference on Machine Learning, ser. Proceedings of Machine LearningResearch, M. F. Balcan and K. Q. Weinberger, Eds., vol. 48.NewYork, New York, USA: PMLR, Jun 2016, pp. 1928–1937.[6] Z. Wang, V. Bapst, N. Heess, V. Mnih, R. Munos, K. Kavukcuoglu, andN. de Freitas, “Sample efﬁcient actor-critic with experience replay,” inInternational Conference on Learning Representations, 2017.[7] M. Jaderberg, V. Mnih, W. M. Czarnecki, T. Schaul, J. Z. Leibo,D. Silver, and K. Kavukcuoglu, “Reinforcement learning with un-supervised auxiliary tasks,” in International Conference on LearningRepresentations, 2017.[8] L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V. Mnih, T. Ward,B. Yotam, F. Vlad, H. Tim, I. Dunning, S. Legg, and K. Kavukcuoglu,“IMPALA: Scalable Distributed Deep-RL with Importance WeightedActor-Learner Architectures,” 35th International Conference on Ma-chine Learning, ICML 2018, vol. 4, pp. 2263–2284, 2018.[9] M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling, “The ArcadeLearning Environment: An evaluation platform for general agents,”Journal of Artiﬁcial Intelligence Research, 2013.[10] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schul-man, J. Tang, and W. Zaremba, “Openai gym,” arXiv preprintarXiv:1606.01540, 2016.[11] J. Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre,S. Schmitt, A. Guez, E. Lockhart, D. Hassabis, T. Graepel et al.,“Mastering atari, go, chess and shogi by planning with a learned model,”Nature, vol. 588, no. 7839, pp. 604–609, 2020.[12] C. Berner, G. Brockman, B. Chan, V. Cheung, P. D˛ebiak, C. Dennison,D. Farhi, Q. Fischer, S. Hashme, C. Hesse et al., “Dota 2 with largescale deep reinforcement learning,” arXiv preprint arXiv:1912.06680,2019.[13] R. Coulom, “Efﬁcient selectivity and backup operators in monte-carlotree search,” in International conference on computers and games.Springer, 2006, pp. 72–83.[14] S. Frazier and M. Riedl, “Improving deep reinforcement learning inminecraft with action advice,” in Proceedings of the AAAI Conferenceon Artiﬁcial Intelligence and Interactive Digital Entertainment, vol. 15,no. 1, 2019, pp. 146–152.[15] W. H. Guss, C. Codel, K. Hofmann, B. Houghton, N. Kuno, S. Milani,S. Mohanty, D. P. Liebana, R. Salakhutdinov, N. Topin et al., “TheMineRL competition on sample efﬁcient reinforcement learning usinghuman priors,” NeurIPS Competition Track, 2019.[16] W. H. Guss, M. Y. Castro, S. Devlin, B. Houghton, N. S. Kuno,C. Loomis, S. Milani, S. Mohanty, K. Nakata, R. Salakhutdinov,J. Schulman, S. Shiroshita, N. Topin, A. Ummadisingu, and O. Vinyals,“Neurips 2020 competition: The MineRL competition on sample efﬁ-cient reinforcement learning using human priors,” NeurIPS CompetitionTrack, 2020.[17] A. Asperti, C. De Pieri, and G. Pedrini, “Rogueinabox: an environmentfor roguelike learning,” International Journal of Computers, vol. 2, 2017.
[18] Y. Kanagawa and T. Kaneko, “Rogue-gym: A new challenge for general-ization in reinforcement learning,” in 2019 IEEE Conference on Games(CoG), 2019, pp. 1–8.[19] A. Asperti, C. De Pieri, M. Maldini, G. Pedrini, and F. Sovrano, “Amodular deep-learning environment for rogue,” WSEAS Trans. Syst.Control, vol. 12, pp. 362–373, 2017.[20] A. Asperti, D. Cortesi, and F. Sovrano, “Crawling in rogue’s dungeonswith (partitioned) a3c,” in Machine Learning, Optimization, and DataScience, G. Nicosia, P. Pardalos, G. Giuffrida, R. Umeton, and V. Sci-acca, Eds. Cham: Springer International Publishing, 2019, pp. 264–275.[21] A. Asperti, D. Cortesi, C. De Pieri, G. Pedrini, and F. Sovrano,“Crawling in rogue’s dungeons with deep reinforcement techniques,”IEEE Transactions on Games, vol. 12, no. 2, pp. 177–186, 2020.[22] J. Campbell and C. Verbrugge, “Learning combat in NetHack,” in Thir-teenth Annual AAAI Conference on Artiﬁcial Intelligence and InteractiveDigital Entertainment (AIIDE 2017), October 2017, pp. 16–22.[23] Y. Burda, H. Edwards, A. Storkey, and O. Klimov, “Exploration byrandom network distillation,” in International Conference on LearningRepresentations, 2019.[24] Y. Kim, C. Denton, L. Hoang, and A. M. Rush, “Structured attentionnetworks,” in International Conference on Learning Representations,2017.[25] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez,L. u. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advancesin Neural Information Processing Systems, I. Guyon, U. V. Luxburg,S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett,Eds., vol. 30.Curran Associates, Inc., 2017.[26] H. Zhao, J. Jia, and V. Koltun, “Exploring self-attention for imagerecognition,” in Proceedings of the IEEE/CVF Conference on ComputerVision and Pattern Recognition (CVPR), June 2020.[27] V. Zambaldi, D. Raposo, A. Santoro, V. Bapst, Y. Li, I. Babuschkin,K. Tuyls, D. Reichert, T. Lillicrap, E. Lockhart, M. Shanahan,V. Langston, R. Pascanu, M. Botvinick, O. Vinyals, and P. Battaglia,“Deep reinforcement learning with relational inductive biases,” in Inter-national Conference on Learning Representations, 2019.[28] H. Küttler, N. Nardelli, A. H. Miller, R. Raileanu, M. Selvatici,E. Grefenstette, and T. Rocktäschel, “The NetHack Learning Envi-ronment,” in Proceedings of the Conference on Neural InformationProcessing Systems (NeurIPS), 2020.[29] F. Hill, A. Lampinen, R. Schneider, S. Clark, M. Botvinick, J. L.McClelland, and A. Santoro, “Environmental drivers of systematicityand generalization in a situated agent,” in International Conference onLearning Representations, 2020.[30] C. Ye, A. Khalifa, P. Bontrager, and J. Togelius, “Rotation, translation,and cropping for zero-shot generalization,” in 2020 IEEE Conference onGames (CoG).IEEE, 2020, pp. 57–64.[31] H. Küttler, N. Nardelli, T. Lavril, M. Selvatici, V. Sivakumar, T. Rock-täschel, and E. Grefenstette, “TorchBeast: A PyTorch Platform forDistributed RL,” arXiv preprint arXiv:1910.03552, 2019.