Computer Science
Computer Science-2021.gz:143184
Computer Science
841ceab1aa0d4e4c34ad4ba69b59369617f26e75
Goal-Aware Cross-Entropy for Multi-Target Reinforcement Learning

Learning in a multi-target environment without prior knowledge about the targets requires a large amount of samples and makes generalization difficult. To solve this problem, it is important to be able to discriminate targets through semantic understanding. In this paper, we propose goal-aware cross-entropy (GACE) loss, that can be utilized in a self-supervised way using auto-labeled goal states alongside reinforcement learning. Based on the loss, we then devise goal-discriminative attention networks (GDAN) which utilize the goal-relevant information to focus on the given instruction. We evaluate the proposed methods on visual navigation and robot arm manipulation tasks with multi-target environments and show that GDAN outperforms the state-of-the-art methods in terms of task success ratio, sample efficiency, and generalization. Additionally, qualitative analyses demonstrate that our proposed method can help the agent become aware of and focus on the given instruction clearly, promoting goal-directed behavior.


Introduction
Reinforcement learning (RL) has been expanding to various fields including robotics, to solve increasingly complex problems. For instance, RL has been gradually mastering skills such as robot arm/hand manipulation on an object [2,48,36,22] and navigation to a target destination [18,39]. However, to benefit humans like the R2-D2 robot in the Star Wars, RL must extend to realistic settings that require interaction with multiple objects or destinations, which is still challenging for RL.
For this reason, it is important for multi-target tasks to be considered. We use the term multi-target tasks to refer to tasks that require the agent to interact with variable goals. In a multi-target task, targets are possible goal candidates, which may be objects or key entities that play a decisive role in determining the success or failure of the task execution. The goals may be selected among the targets by the current multi-target task, specified with a cue or an instruction such as "Bring me a {spoon, cup or specific object}" or "Go to the {kitchen, livingroom or specific destination}". The states in which the agent reaches the goal are called goal states.
Reinforcement learning allows learning multi-target or instruction-based tasks in an end-to-end manner. Latest reinforcement learning studies on these tasks mainly focus on prior knowledge about targets [27,30,8]. Other studies focus on learning representation on the environment [46] or about the targets only implicitly [44]. These methods lead to insufficient understanding of the goal, and sample-inefficiency and generalization problems arise.
For this matter, we propose a Goal-Aware Cross-Entropy (GACE) loss and Goal-Discriminative Attention Networks (GDAN) 2 for multi-target tasks in reinforcement learning. These methods, unlike previous studies without prior knowledge, allow semantic understanding of goals including their appearances and other characteristics. The agent automatically labels and collects goal states data through trial-and-error in a self-supervised manner. Based on these self-collected data, we use GACE loss as an auxiliary loss to train a goal-discriminator that learns goal representations. Lastly, the GDAN extracts the information from the goal-discriminator as a goal-relevant query, with which an attention is performed to infer goal-directed actions.
We additionally propose visual navigation and robot arm manipulation tasks as benchmarks for multi-target task experiments. These tasks involve targets which are the multiple types of objects randomly placed within the environments. In the benchmarks, we make these tasks visually complex using randomized background textures, allowing us to evaluate generalization.
In summary, the contributions of this paper are as follows: • We propose a Goal-Aware Cross-Entropy loss for learning auto-labeled goal states in a selfsupervised manner, solving instruction-based multi-target tasks in reinforcement learning.
• We additionally propose Goal-Discriminative Attention Networks that use the goal-relevant query from the goal-discriminator to focus exclusively on important goal-related information.
• Our method achieves significantly better performances in terms of the success ratio, sampleefficiency, and generalization on visual navigation and robot arm manipulation multi-target tasks. In particular, compared with the baseline methods, our method excels in Sample Efficiency Improvement metric by 17 times in visual navigation task and by 4.8 times in manipulation task.
• We present two instruction-based benchmarks for goal-based multi-target environments, to interact with multiple targets for realistic settings. These benchmarks are made publicly available, along with the implementation of our proposed method.

Related work
Multi-target tasks in reinforcement learning There has been a sparsity of studies that aim to solve multi-target tasks with reinforcement learning. Wu et al. [44] proposes instruction-based indoor environments, as well as attention networks for multi-target learning to respond to the given instruction. Deisenroth and Fox [11] uses a model-based policy search method for multiple targets, instead of learning goal-relevant information. The aforementioned methods are for learning dynamics model or learning targets indirectly for multiple target tasks. In our method, on the other hand, the policy directly learns to distinguish goals without the dynamics model.
The terms targets and goals have been used in diverse manners in the existing literature. There are subgoal generation methods that generate intermediate goals such as imagined goal [31,14] or random goals [34] to help agents solve the tasks. Multi-goal RL [47,35,13,10] aims to deal with multiple tasks, learning to reach different goal states for each task. Scene-driven visual navigation tasks [49,12,30] specify the goal by an image. However, these tasks require the agent only to search for visually similar locations or objects, rather than gaining semantic understanding of goals and promoting generalization. Instruction-based tasks [1,40] focus on learning to follow detailed instructions. These tasks have a different purpose from our tasks, where we focus on having appropriate interactions with multiple targets depending on the instruction.
Representation learning in reinforcement learning To improve performance in RL, several recent works have used a variety of representation learning methods. Among these methods, many make use of auxiliary tasks taught via unsupervised learning techniques such as VAE [24,19] for learning dynamics [16,45,38] and contrastive learning method [7] for learning representations [26] of a given environment. Nair et al. [32] uses goal states, specified as the last states of the trajectories, to learn the difference between future state and goal state for redistributing model error in model-based reinforcement learning. The aforementioned methods are for learning representations of the environment rather than for learning those of key states for solving the task. In contrast, our method directly learns goal-relevant state representations.
Jaderberg et al. [21] proposes various auxiliary tasks for representation learning in RL. One such method is reward prediction, which learns the environment by predicting the reward of the future state from three classes: {positive, 0, negative}. This method focuses on myopic reward prediction, and not on ultimately learning the goal state to solve the task. Consequently, it is not a suitable method to apply to multi-target environments, where the goal can be selected among a diverse range of objects.
Attention methods in reinforcement learning The attention method was initially introduced for natural language processing [3] but is now being actively studied in various fields such as computer vision [43]. There are also attention-based methods for reinforcement learning which use inputs from different parts of the state [4,9]. Other methods include the use of encoding information of sequential frames up to the last step [29], as well as the application of self-attention [6]. Unlike these methods, our approach explicitly extracts and focuses solely on goal-related information for solving a task.

Preliminary
Reinforcement learning (RL) from Sutton and Barto [41] aims to maximize cumulative rewards by trial-and-error in a Markov Decision Process (MDP). An MDP is defined by a tuple (S, A, R, P, γ), where S is the set of states, A is the set of actions, R : S × A → R is the reward function, P : S × A × S → R is the transition probability distribution, and γ ∈ (0,1] is the discount factor. At each time step t, the agent observes state s t ∈ S, selects an action a t ∈ A according to its policy π : S → A, and receives a reward r t and next state s t+1 . In finite-horizon MDPs, return R t = Σ T −t k=0 γ k r t+k is accumulated discounted rewards, where T is the maximum episode length. State value function V (s) = E[R t |s t = s] is the expected return from state s.
Instruction-based multi-target reinforcement learning Universal value function approximators [37] estimate state value function as V (s, x), for jointly learning the task from embedded state s and goal information x. This is relevant to the notions of multi-target RL that we define as below.
Multi-target environments contain N targets, or goal candidates T = {z 1 , z 2 , ..., z N }. These environments provide an instruction I z that specifies which target the agent must interact with, or the goal z ∈ T . The instruction is given randomly every episode, in the form such as "Get the Bonus" or "Reach the Green Box" as in our benchmark tasks, in which cases the goals are Bonus and Green Box respectively. Hence, the state value function in such task is V (s, I z ) = E[R t |s t = (s, z)], wheres t is state conditioned on the goal z. The policy π is also conditioned on the instruction I z as π(a t |s t , I z ) where a t is action at time t given state s t and I z .

Auto-labeled goal states for self-supervised learning
Prior to describing the main method in Sec. 3.3 and 3.4, this section explains the automatic collection of goal data for self-supervised learning. Suppose that the instruction I z specifies the target z as the goal and the episode ends when the goal is reached at time step t . We refer to the state s t as a goal state, which we assume is highly correlated with the reward or the rewarding state. Throughout the training, the reached goal label z and the corresponding goal state s t are automatically collected as a tuple (s t , one_hot(z)), called storage data. Rather than being manually provided the goal information, the agent actively gathers the data needed to learn the goals, relying only on the instruction I z given by the environment. This allows the agent to learn in an end-to-end, self-supervised manner. The states for the failed episodes are also stored as a negative case with low probability N . We clarify that the storage data does not serve as the prior during the learning and is instead used for training alongside the multi-target reinforcement learning task.

Goal-discriminator by goal-aware cross-entropy
Our proposed method adds an auxiliary task to the main reinforcement learning task as shown in Figure 1a. A general reinforcement learning model consists of a feature extractor σ(·), which converts state s t to an encoding e t , and an ActorCritic f (·) that outputs a policy π and value V : where I is the instruction. The details of the ActorCritic f (·) and the resulting loss L RL depend on the specific reinforcement learning algorithm that is chosen. In our visual navigation experiments, we use asynchronous advantage actor-critic (A3C) [28] as the main algorithm, where the loss L RL is defined as the following where L p and L v respectively denote policy and value loss, R t denotes the sum of decayed rewards from time steps t to T , and H and β denote the entropy term and its coefficient respectively. See Appendix D for algorithm details.
The vanilla base algorithm is inefficient and indirect at gaining an understanding of goals. Suppose that we have a set of goal states S z ⊂ S corresponding to each goal z. Intuitively, given an instruction I z that specifies z, the agent must learn the value function V (s, I) that discriminates the goals, such that V (s z goal , I z ) > V (s j goal , I z ) for s z goal ∈ S z and s j goal ∈ S j , ∀z = j. To achieve this effect, we propose Goal-Aware Cross-Entropy (GACE) loss as our contribution, which trains the goal-discriminator that facilitates semantic understanding of goals alongside the policy in Figure 1a. The inference process is as follows. First, the storage data collected in Sec. 3.2 are sampled as a batch of s goal,i 's (with i as data index within the batch) and fed into the feature extractor σ(·) for state encoding in Eq. 6. Next, the encoded input data e s goal,i is fed into the goal-discriminator d(·), a multi-layer perceptron (MLP), in Eq. 7. This yields a prediction g goal,i , a vector that contains probability distribution over possible goals that s goal,i belongs to.
From the output g goal,i , we calculate the Goal-Aware Cross-Entropy loss L GACE as Eq. 8, where M is the batch size, and z i is the automatic label corresponding to state s goal,i .
We complete the training procedure by optimizing the overall loss L total as the weighted sum of the two losses in Eq. 9. We focus on improving the policy for performing the main task and assign weight η to L GACE for performing goal-aware representation learning for the feature extractor σ(·).
We note that the goal-discriminator d(·) is updated according to L GACE only during training, not during inference, excluding the ActorCritic.
The explained procedure forms a visual representation learning method, where the GACE loss makes the goal-discriminator become goal-aware without external supervision. Such goal-awareness is advantageous for sample-efficiency in multi-target environments, as well as generalization, as it makes the agent robust in a noisy environment. Such effects are demonstrated in our quantitative results and qualitative analyses.

Goal-discriminative attention networks
The goal-discriminator discussed so far can influence the policy inference. However, in order to effectively utilize the discriminator to enhance the performance and efficiency of the agent, we propose Goal-Discriminative Attention Networks (GDAN). Overall, the goal-aware attention described in Figure 1b involves a goal-relevant query q z t from within the goal-discriminator d(·), and the key k t and value v t from encoded state in the ActorCritic f (·).
During the inference, the state encoding vector e t is passed through the first linear layer of the goaldiscriminator, which yields the query vector q z t that implicitly represents the goal-relevant information. Meanwhile, the vector e t is inferenced through a gated-attention [44] with instruction I for grounding. The gated-attention vector is passed through LSTM (Long Short-Term Memory) [20] and splits in half into key k t and value v t . Suppose the vectors q z t , k t , and v t are d q -, d k -, and d v -dimensional, respectively. The query and key are linearly projected to d v -dimensional space using learnable parameters W q of dimensions d v × d q and W k of dimensions d v × d k , respectively. The activation function φ, which is tanh in our case, is applied to the resulting vectors to yield a goal-aware attention vector u z t in Eq. 10. The attention vector u z t contains the goal-relevant information in s t . Lastly, Hadamard product is performed between u z t and v t in Eq. 11, yielding the attention-dependent state representation vector h t used for calculating the policy and value. The vector h t is concatenated with gated-attention vector from a state encoding vector e t in ActorCritic f (·).
We underscore two points about the proposed networks. First, while the goal-discriminative feature extractor σ(·) trained by GACE loss may be sufficient, by constructing attention networks, the implicit goal-relevant information from the goal-discriminator directly affects the ActorCritic f (·) that determines the policy and value. Second, the query vectors are from a part of the goal-discriminator, which allows the ActorCritic to actively query the input for goal-relevant information rather than having to filter out large amounts of unnecessary information. These two design choices enable the agent to selectively allocate attention for goal-directed actions (Figure 5c), making full use of the GACE loss method. Details of our architecture are covered in Appendix B.

Experiments
Experimental setup The experiments are conducted to evaluate the success ratio, sample-efficiency, and generalization of our method, as well as compare our work to competitive baseline methods in multi-target environments. We develop and conduct experiments on (1) visual navigation tasks based on ViZDoom [23,18], and (2) robot arm manipulation tasks based on MuJoCo [42]. These multitarget tasks involve N targets which are placed at positions p 1 , p 2 , · · · , p N , and provide the index of one target as the goal z ∈ {1, 2, · · · , N }. At time step t, the agent receives state s t normalized to range [0,1]. The agent receives a positive reward r success if it gets sufficiently close to the goal (i.e. x t − p z ≤ ) where x t is the agent position at time t. On the other hand, the agent is penalized by r nongoal if it reaches a non-goal object, or r timeout if it times out at t ≥ T . Lastly, the agent receives a time step penalty r step , which is empirically found to accelerate training. Reaching either a goal or non-goal object terminates the episode. All experiments are repeated five times.
Details of sample efficiency metrics We introduce Sample Requirement Ratio (SRR) and Sample Efficiency Improvement (SEI) metrics to measure the sample efficiency of each algorithm. For the given task, we select a reference algorithm A and its reference success ratio X%. Suppose this algorithm reaches the success ratio X% within n A updates. We measure the SRR metric of another algorithm B as SRR B = n B /n A , where B reaches X% within n B updates. Also, SEI metric of B is measured as SEI B = (n A − n B )/n B . Lower SRR, higher SEI indicate higher sample efficiency.

Visual navigation task with discrete action
We conduct experiments on egocentric 3D navigation tasks based on ViZDoom (Figure 2a). The RGB-D state is provided as an input to the agent, and three different actions (MoveForward, TurnLeft, TurnRight) are allowed. One item from each of the four different classes of objects -each class (e.g. Bonus) containing two different items (e.g. HealthBonus and ArmorBonus) -is placed within the map, with one class randomly selected as the goal by the instruction for each episode like "Get the Bonus". In every episode, the agent is initially positioned at the center of the map. The rewards are set as r success = 10, r nongoal = −1, r timeout = −0.1, r step = −0.01. Full details of the environment are provided in the Appendix C.
Details of each task To evaluate the performance of our method on tasks of varying difficulties, we set up four configurations. The V1 setting consists of a closed fixed rectangular room with walls at the map boundaries, and object positions are randomized across the room. V2 is identical to V1, except that the textures for the background, such as ceiling, walls, and floor, are randomly sampled from a pool of textures every episode. 40 textures are used for the seen environment, while 10 are used for the unseen environment which is not used for training to evaluate generalization. Hence the tasks allow 40 3 and 10 3 different state variations in seen and unseen environments respectively. V3 is more complicated, with larger map size, shuffled object positions, and additional walls within the map boundaries to form a maze-like environment. "Shuffled positions" indicates that the object positions are chosen as a random permutation of a predetermined set of positions. Lastly, V4 is equivalent to V3 with the addition of randomized background textures for the seen and unseen environments.V2 and V4 allow us to evaluate the agent's generalization to visually complex, unseen environments. V1 and V3 can be regarded as the upper bound of performance on V2 and V4 respectively.
Baselines for comparison As the base algorithm, we use A3C, an on-policy RL algorithm, with the model architecture that uses gated-attention [5,33] with LSTM. This baseline is proposed in [44] 3 for multi-target learning on navigation. Other competitive general RL methods are added onto the base multi-target learning algorithm, due to the rarity of appropriate multi-target RL algorithms for comparison. A3C+VAE [16,38] learns VAE features for all time steps, unlike our methods which learn only for goal states. A3C+RAD applies augmentation -the random crop method, known to be the most effective in [25] -to input states in order to improve sample efficiency and generalization. A3C+GACE and A3C+GACE&GDAN are our methods applied to the A3C as covered in Sec. 3.3 and 3.4.
Results Performance is measured as the success ratio of the agent across 500 episodes in all tasks. As shown in Figure 4, most baselines successfully learn the V1 task, albeit at different ratios. In V4, A3C+VAE (green) initially shows the steepest curve, probably because it reaches the object located near the initial agent position before the learning commences properly. A3C+RAD (blue) shows the steepest learning curve in V1, but fails to learn in visually complex environments such as V2 and V4. In V3, A3C+GACE (orange) and A3C+GACE&GDAN (red) achieve 52.6% and 78.2% respectively. We attribute such discrepancy to A3C+GACE&GDAN's more direct and efficient usage of goal-related information from the goal-discriminator. Especially, A3C+GACE&GDAN achieves as high as 86.6% and 54.9% in V2 seen and V4 seen respectively. Furthermore, in V2 unseen and V4 unseen tasks, the agents trained with our methods generalize well to unseen environments. Thus, our methods achieve the state-of-the-art sample-efficiency and performance in all tasks. The full details are covered in the Appendix B.
The SRR and SEI measurements of the models are shown in Table 1. For measuring sample efficiency, we select A3C as the reference algorithm and 56.55% as the reference success ratio in V1 task. This is the highest performance of A3C within 2M updates. Our methods show the highest sample efficiency compared to the baselines.

Robot manipulation task with continuous action
To evaluate our method in the continuous-action domain, we conduct experiments on the UR5 robot arm manipulation tasks with 6 degrees of freedom, based on MuJoCo (Figure 2b). The environment comprises a robot arm and three or five objects of different colors (red, green, blue, etc). The state is an RGB image provided from the fixed third-person perspective camera view that captures both the robot and the objects in Figure 2b. The rewards are r success = 1, r nongoal = −0.3, r timeout = −0.1, r step = −0.01.

Details of each task
Our method is evaluated on three different vision-based tasks. In the R1 task, the object positions are shuffled among three preset positions on grey background, and one of the three objects is randomly specified as the goal by the instruction like "Reach the Green Box". The robot arm must reach the goal object within the time limit T steps while avoiding other objects. R2 task complicates the R1 task by replacing the grey background with random checkered patterns. The agent's performance is measured on the seen environment, in which the agent is trained, as well as the unseen environment. The seen and unseen environments each uses a different set of 5 colors of the checkered background. R3 task is a more complex variant of R1 setting, where five target classes are available, three of which are randomly sampled by the environment and randomly positioned among the three preset locations as in R1. The full details of these tasks are outlined in the Appendix C.  Baselines for comparison As a baseline method, we use a deterministic variant of Soft Actor-Critic (Pixel-SAC [17]), an off-policy algorithm commonly used in robotic control tasks. As a baseline, we evaluate SAC+AE [45], which uses β-VAE [19] for encoding and decoding states. Another baseline is SAC+CURL which performs data augmentation (cropping) and contrastive learning between the same images. These baseline methods are all competitive methods in the continuous-action domain. We use SAC as a base algorithm to apply our methods SAC+GACE and SAC+GACE&GDAN. Note that the ActorCritic is as depicted in Figure 1b, without the LSTM.

Results
The results for robot manipulation tasks are shown in Table 2 and Figure 7 (in Appendix B). Performance is measured as the success ratio of the agent for 100 episodes. SAC shows higher difficulty in learning R2 tasks, which are more visually complicated than R1 task. Unexpectedly, SAC+AE and SAC+CURL has improved performance in R2 seen. We speculate that the two algorithms that perform representation learning are more suitable for the task with diversity. Especially, SAC+CURL consistently achieves competitive performances in all environments except for R2 unseen task. However, it does not seem capable of learning with complex unseen backgrounds, showing huge deterioration in performance for R2 unseen. SAC+GACE consistently shows strong performance in all environments, as well as high generalization capability for R2 unseen task. Finally, SAC+GACE&GDAN attains state-of-the-art performance in all environments including the unseen task for generalization. In particular, we observe the smallest performance discrepancy between R2 seen and unseen tasks. This shows that it can learn robustly in a complex background acting as a noise. Furthermore, our methods show the highest performance in R3 task, adapting well to a more diverse set of possible target objects. This supports that the GACE mechanism addresses the scalability issue well.
The SRR and SEI measurements of these methods are shown in Table 3. We select SAC as the reference algorithm and 63.1% as the reference success ratio in R1 task. SAC+GACE shows the state-of-the-art sample efficiency compared to all baseline models, meaning that it learns faster than SAC+GACE&GDAN for relatively easy tasks.
Comparison with single-target task We conduct an ablation study, where effectively single-target policies are trained on a variant of R1 task with the instruction fixed to a single target as a goal. We train these policies for 120K updates, such that the total number of updates (360K) is roughly equal to the number of updates for training the multi-target baseline models. The average performance of these policies is 47±11%, which is noticeably lower than the performance of multi-target SAC agent (63.1±6.9%). Continuing the training, the single-target policies converge to 87.5±2.4% success ratio in 333,688 updates, serving as a competitive baseline. However, training individual single-target  policies for multi-target tasks poses a scalability problem. It not only requires memory for weights that is directly proportional to the number of targets, but also uses a large amount of samples to learn the targets separately. In contrast, the multi-target agent can efficiently learn a joint representation, improving memory and sample efficiency as well as allowing generalizable learning of many targets.

Analyses
Effectiveness of goal-aware cross-entropy loss Additional experiments are conducted in the V1 task to investigate how our method influences the agent's learning. To vary the extent to which the GACE loss may affect learning, the goal-discriminator weights are frozen after 10K and 20K updates such that the GACE loss L GACE does not contribute to learning afterwards. We note that in Figure 4a, although the GACE loss (frozen weights) does not further contribute to learning, the discriminator accuracy improves only by updating the policy. This indicates that throughout the training, the agent gradually develops a feature extractor σ(·) that can discriminate targets. Our method makes it possible for the feature extractor to directly learn to discriminate.
In addition, the learning curves in Figure 4b corroborate that such development of the feature extractor is accelerated by the GACE loss. Even when the agent is trained with the GACE only temporarily (as with GACE 10K and 20K), the learning curve is steeper than that with vanilla A3C. Furthermore, the unfrozen GACE (red line) shows a steeper learning curve than GACE 20K, which is again steeper than GACE 10K. Consequently, it can be seen that representation learning of feature extractor updated by GACE loss has positive influence on policy performance than learning solely with policy updates.
Simple ActorCritic f (·) We perform an ablation study about attention model that simply concatenates e t from feature extractor σ(·) with q z t from ActorCritic f (·) in Figure 1b to verify the effectiveness of the attention method. We conduct the experiment in R1 task. As a result, a success ratio of 81.5 ± 10.0 % is obtained, which is lower than that of GACE loss and GACE&GDAN. We speculate that since not every state corresponds to the goal -such goal states are quite sparse -the additional information of the goal-discriminator acts as a noise, hindering the efficient learning of the policy. This supports that, in contrast to such naive use of goal-discriminator features, our proposed attention method can effectively process goal-related information and is robust against noise. Analysis using saliency maps To ascertain that an agent trained with GACE and GACE&GDAN indeed becomes goal-aware, we use saliency maps [15] to visualize the operation of three agents within the V2 unseen task, as shown in Figure 5. The blue shades indicate the regions that the agent significantly attends to for policy inference, and the red shades indicate those regions for the value estimation. The overlapping blue and red regions appear purple. The three agents are trained with A3C, GACE, and GACE&GDAN, respectively, for 4M updates. Figure 5a shows the operation of the A3C agent. It shows overly high sensitivity to edges in the background, approaching the walls rather than searching for the goal. Also, in the rightmost image in (a), although it does attend to the goal object located on the right side, a TurnLeft action is performed. This suggests that the agent cannot discern between objects because it lacks understanding of targets. Figure 5b corresponds to the GACE agent, showing high sensitivity to all targets and intermittent edges. As its attention to targets demonstrates, the ability to distinguish the goal is far superior to that of the A3C agent. Finally, Figure 5c shows that the GACE&GDAN agent exhibits high sensitivity to all goals while hardly responding to irrelevant edges. In addition, upon noticing the goal, the agent allocates attention only to the goal, rather than unnecessarily focusing on the non-goals. These results support that our method indeed promotes goal-directed behavior that is visually explainable.

Conclusion
We propose GACE loss and GDAN for learning goal states in a self-supervised manner using a reward signal and instruction, promoting a goal-focused behavior. Our methods achieve state-of-the-art sample-efficiency and generalization in two multi-target environments compared to previous methods that learn all states. We believe that the disparity between the performance improvements in the two benchmark suites is due to the degree of similarity between the goal states and the others.
The limitation of our method is that it may not be well-applied in tasks where instructions do not specify a target (e.g. "Walk forward as far as possible") or tasks with little correlation between states and rewards based on success signal. Also, our methods assume that the total number of target object classes is known beforehand. Our method can be abused depending on the goal specified by a user. Nonetheless, this paper brings to light the possible methods of prioritizing data for efficient training, especially in multi-target environment.

APPENDIX
This appendix provides additional information not described in the main text due to the page limit. It contains additional analysis results in Section A, experiment details in Section B, environment details in Section C and a pseudocode of and SAC+GACE algorithms in Section D.

A Analysis
Our proposed methods show an outstanding performance in terms of sample efficiency, generalization, and task success ratio. This is consistent with the previous studies [38,21], where auxiliary tasks for reinforcement learning encourage the agent to learn extra representations of the environment, as well as provide additional gradients for updates. To clarify more on the roles and characteristics of the proposed methods, we conduct an additional experiment.
A.1 Generalization to unseen environment in visual navigation task Figure 6 shows the accuracy curves of the goal-discriminator, or more precisely, how consistently the highest probability is assigned to the ground-truth label z i for the goal-discriminator output g goal.i . The goal state prediction accuracies of the goal-discriminator on V2 seen and V2 unseen tasks after 200K training updates are 99.32 ± 0.62% and 89.15 ± 3.44%, respectively. The accuracy curves are shown in Figure 6. As the goal-discriminator learns various goal states, the agent becomes capable of discerning goals accurately even in an unseen environment. We believe this figure shows that GACE and GDAN can show high performance even in unseen tasks thanks to the excellent generalization ability of GACE. Prior to initiating the learning of goal-aware representation, the minimum amount of storage data is collected through a random policy during warmup phase. In the case of robot manipulation tasks, where a random agent achieves low success ratio, the states for the failed episodes are stored as a negative case z N +1 , where N is the number of target objects in the environment. After the warmup phase, as the agent learns to succeed in the given task through trial and error, the successful goal states are stored. At the same time, due to their commonness, unsuccessful states -where the agent fails to reach any target object at all within time limit -are stored as negative case z N +1 with a low probability N < 1. Empirically, setting N value close to the success ratio of the random policy leads to high performance.

B.2 Model for visual navigation task
The ViZDoom environment is a first-person perspective navigation task, requiring memory for reasonable inference. For this, we use a model that contains Long Short-Term Memory (LSTM), and all compared methods use the same model architecture.
Four consecutive frames of the environment are stacked into one state. The stacked state s t is fed as input to the image feature extractor σ(·), which consists of 4 convolutional layers, each with a kernel size of 3, a stride of 2, and zero-padding of 1. The first two layers each contains 32 channels and the rest of the layers each has 64 channels. The output of σ(·) is flattened and fed into a fully-connected layer of 256 units to be converted into image features e t . Batch normalization is applied after every convolutional layer. ReLU is used as the activation function for all of the layers.
We first describe the model corresponding to the GACE method without attention networks, and then elaborate the GDAN method afterwards. ActorCritic f (·) consists of a word embedding, an LSTM, an MLP for policy, and an MLP for value. The goal index from the instruction is converted to a word embedding I of dimension 25. The embedding I is fed into a linear layer and converted to I as a 256-dimensional vector. The gated attention vector M is calculated as the Hadamard product between the image feature output e t of the feature extractor and the instruction embedding I .
The input to the LSTM is the concatenation between M and sigmoid(I ). The hidden layer of the LSTM and the gated attention output M are concatenated to form the input for both the MLP for policy and that for value. The MLP for policy consists of two layers with 128 and 64 units respectively and the MLP for value consists of two layers with 64 and 32 units respectively.
The goal-discriminator d(·) consists of MLP of two layers that respectively contains 256 and 4 units. Each of the four output units belongs to each target category. This goal-discriminator is only used during training.
GDAN adds an attention method to the architecture. At the end of the model, the attention-dependent state representation vector h t , mentioned in Sec. 3.4 of the main paper, is concatenated with a state encoding vector e 1 t := LST M (e t ). Lastly, the concatenated vector is fed as input to 2-layered perceptron (yet within f (·)) to infer policy π and value V .

B.4 Model for robot arm manipulation task
Robot arm manipulation tasks are conducted in the MuJoCo environment. For these tasks, we construct a model shown in Figure 1b without LSTM and train the agent with Algorithm 1. Unlike the model for visual navigation tasks, we do not use LSTM within the model for robot arm manipulation tasks.
The main model architecture for the GACE method without attention in robot arm manipulation tasks is similar to the architecture used in visual navigation tasks, except that in place of LSTM, a single linear layer is used. At the end of the model, the attention-dependent state representation vector h t is concatenated with a state encoding vector e 1 t := Linear(e t ). The concatenated vector is used within the ActorCritic f (·) to infer policy π and value V .
B.5 UR5 manipulation task learning curves

B.6 Hyperparameters
The hyperparameters for visual navigation tasks and UR5 manipulation tasks are shown in Table 6 and Table 7, respectively.
Most of the parameters commonly used in the base algorithm are used as in the correspondingly cited papers, and additional fine-tuning is conducted to suit the learning environment. In addition to the parameters suggested by our method, a value of [0.3, 0.5] is recommended for η, and it is recommended to set N close to the success ratio of the random policy.

C Environment details
The additional details regarding the visual navigation tasks in ViZDoom and the robot arm manipulation tasks in MuJoCo that could not be covered in the main paper are outlined in this section.

C.1 Visual navigation in ViZDoom
Each visual navigation task provides RGB-D images with dimensions 42×42. The images of four consecutive frames are stacked into one input state s t . In V1 and V2 tasks, the maximum number of time steps in an episode is T = 25, and the size of maps for these tasks is 7×7. All objects are randomly located. The success ratio of random policy is 6.6%. In V3 and V4 tasks, the maximum number of time steps in an episode is T = 50, and the size of maps for these tasks is 10×10. All object positions are shuffled, such that the object positions are chosen as a random permutation of a predetermined set of positions which are chosen evenly across the map. The success ratio of random policy is 8%. An action repeat of 4 frames is used.

C.2 Robot manipulation task in MuJoCo
Input state s t is an RGB image with dimensions of 84×84. No frame stack is used. In each time step, an action is repeated 16 times. The episode time limit is T = 50 steps in all tasks. The joint angles of the robot are constrained within specific boundaries, which are [-3.14, 3.14] or [-5, 5]. For the R2 tasks, the background in each task is randomly sampled from a set of five checkered patterns, some of which are shown in Figure 8. The generalization ability of the agent is measured by making the set of backgrounds in the R2 seen task and the set of backgrounds in the R2 unseen task mutually exclusive. The generalization ability of the agent is measured by assigning five background checkered colors to each of the seen and unseen tasks.