Maintaining a Reliable World Model using Action-aware Perceptual Anchoring

Reliable perception is essential for robots that interact with the world. But sensors alone are often insufficient to provide this capability, and they are prone to errors due to various conditions in the environment. Furthermore, there is a need for robots to maintain a model of its surroundings even when objects go out of view and are no longer visible. This requires anchoring perceptual information onto symbols that represent the objects in the environment. In this paper, we present a model for action-aware perceptual anchoring that enables robots to track objects in a persistent manner. Our rule-based approach considers inductive biases to perform high-level reasoning over the results from low-level object detection, and it improves the robot's perceptual capability for complex tasks. We evaluate our model against existing baseline models for object permanence and show that it outperforms these on a snitch localisation task using a dataset of 1,371 videos. We also integrate our action-aware perceptual anchoring in the context of a cognitive architecture and demonstrate its benefits in a realistic gearbox assembly task on a Universal Robot.


I. INTRODUCTION
Robots today often interact with objects and other agents in their surroundings. To perform complex tasks, robots should reason about the world before they can plan and execute actions. But the interaction among objects in the environment sometimes limits a robot's visibility, causing it to lose sight of certain objects. This is especially true when using the popular eye-in-hand setup like in our case shown in Fig. 1. To remedy this issue, we therefore need the robot to maintain a reliable world model, with which it can estimate object states when they are no longer visible. This capability, often denoted as object permanence, is a frequently studied topic in developmental psychology that describes the ability to understand that objects continue to exist even when they become unobservable. Psychologists believe that human infants develop full object permanence at around two years of age, when they start to understand that objects still exist even if they have been completely removed from their field of view [1].
In robotics, object permanence is an important concept with practical benefits that augments robots' searching and tracking abilities [2,3], facilitates self-recognition [4,5], and enables belief management and perspective taking when engaging with humans [6,7]. Common approaches consider object permanence as a low-level reasoning problem that heavily relies on the visual perception capabilities [2,4,7]. There are also higher-level approaches that use perceptual anchoring to create and maintain the correspondence between symbols and perceived sensor data referring to the same physical objects [8]. These typically use some form of high-level reasoning to track objects that disappear. For instance, there are heuristic approaches that determine objects being grasped by a hand [6] or track the nearest visible object instead of an unobserved one [9,10]. Others use probabilistic reasoning to infer occluded object states [11,12].
But the performance of these models drops with noisy data or more complex scenarios, like when there is flickering detection of false positives or when objects are no longer detected upon insertion into another object and are subsequently 'carried' (e.g., invisible displacements [13]). For example, when a plug is inserted into a case and is no longer visible, it becomes difficult to infer its position from vision alone, especially when the case is moved. Since we know that the plug has been inserted or attached to the case, we can track the plug based on the position of the case.
In this paper, we define an attachment between two objects to be the state resulting from an action that sticks one object to the other, e.g., insert or screw-on, such that the former becomes a part or an extension of the latter and is physically bound to move together. To reason about attachments, the perceptual system should have knowledge about the agent actions that result in such object states. We propose a perceptual anchoring model that uses agent actions to maintain a reliable world model. By considering actions and their effects (i.e., attachments), the robot can predict imperceivable state changes and estimate the positions of invisible objects.
To start our discussion, we first review some related work (Sec. II). After that, we discuss the fundamental hypotheses of our work and describe our system in detail (Sec. III). We show that this rule-based high-level reasoning approach augments the performance of low-level approaches, by first comparing our system to baseline models on the LA-CATER dataset (Sec. IV) and show that our model outperforms them on a snitch localisation task. We further demonstrate the system on a realistic robotic application with a Universal Robot (UR5) where it performs a collaborative gearbox assembly task (Sec. V). We conclude with a discussion on future work to further improve our model (Sec. VI).

II. RELATED WORK
Object permanence has been addressed previously within the computer vision community, especially for multi-object tracking. Papadourakis and Argyros [2] handle temporally long object occlusions by assigning spatially close objects as occluders to objects. Lang et al. [5] trained a CNN to learn a mapping between the current sensor state and the robot's intended motor actions to predict the next sensor state. It considers joint configuration changes of its own arms that might cause object occlusions and uses the predicted image to replace the occluded part of the object in the current image. Shamsian et al. [9] proposed the OPNet architecture that uses two LSTMs to track the target object's visibility, its occluder that carries it, and the target localisation.
High-level approaches in developmental robotics have previously addressed the perceptual anchoring problem. In Roy et al. [7], the robot maintains a 'mental model' of the environment in order to generate different viewpoints of objects. In Milliez et al. [6], the robot makes position hypotheses on occluded objects such as the human grasping it or placing it inside another object. Persson et al. [11] described a probabilistic reasoning system that infers the state of occluded objects and continue tracking through occluded movements. Blodow et al. [12] uses probabilistic anchoring to identify objects that come back into view. Elfring et al. [14] uses a Multiple Hypothesis Tracker algorithm to improve tracking reliability during occlusion and detect clutter (false positives). Heyer and Gräser [15] define relative anchors to represent relations between two reference anchors and use the robot's (camera) movements to re-anchor them. Nitti et al. [16] implemented a relational particle filter that draws on commonsense world knowledge during tracking, such as when one object is inside another. Mininger and Laird [17] address anchoring errors that occur when associating perceptual data with symbolic representations and how to resolve them. While the above approaches address the same problem of tracking occluded objects, only few address invisible displacements and none make use agent actions to predict attachments. We believe that attachments can significantly improve the high-level reasoning process to complement existing approaches.

III. ACTION-AWARE PERCEPTUAL ANCHORING
We propose a perceptual anchoring model that uses agent actions together with perceptual information to maintain a world model of object anchors, described as a set of predicate symbols  = { 1 , 2 , … }. We believe that highlevel reasoning is necessary to maintain object states that are no longer perceived by low-level object detection and can significantly improve object tracking for complex tasks such as invisible displacements. For this, we consider the following hypotheses that object state changes are: H1. Observable: objects do not move on their own and state changes are only caused by agent actions. Thus, we assume that object positions stay the same as last time they were seen, until they are redetected. H2. Gradual: objects do not 'teleport', so detected objects in consecutive frames are likely to have similar attributes e.g., position and size. H3. Consistent: objects are detected consistently over time, so they are only considered to exist/not exist in the world model if they are present/absent for multiple consecutive cycles. These assumptions allow us to make reliable inferences for the positions of objects that are no longer detected by the perceptual system but are physically still present in the scene. The end-to-end complete pipeline consists of three components ( Fig. 2): A) object detection and segmentation, B) action-aware perceptual anchoring, and C) world state inference and action execution. In the following, we describe each component in more detail.

A. Object detection and segmentation
This component performs the initial object detection and feature extraction. It can be replaced by any object detection module as long as it produces a list of object positions and dimensions. The output is a list of percepts Π = { 1 , 2 , … } each associated with a unique identification label and a set of attributes Φ = { , , } corresponding to the object category, its position and dimension of the bounding box.
can also include other attributes such as colour ( ) that can be used in the anchoring component. Since object detection is an on-going research problem, the assigned labels may not always correspond to the same object in the environment, especially if there are multiple objects of the same category. This will be handled by the perceptual anchoring model in the following Sec. III-B. In our experiments on the LA-CATER dataset (Sec. IV) and demonstration with the UR5 (Sec. V) we used a Faster R-CNN (fine-tuned to the respective domains) to output Φ = { , , , } for each detected object .

B. Object permanence with action-awareness (AAPA)
Our model takes the list of detected objects Π and anchors them to the maintained symbol system . The anchoring process consists of two main sub-components: object alignment and hypothesis reasoning.
1) Object alignment: The first part of our model addresses the alignment problem, i.e., finding the correspondence between objects in two consecutive frames. In our case, we find a mapping between the symbol system  at time and the new object percepts Π +1 in two steps: a) Camera movement alignment: Whenever the robot's camera moves, the previously maintained object positions will not be correct, especially when dealing with eye-inhand robotic systems. Thus, we update all object positions in the current world model  to compensate for the camera movement. Given the respective camera positions at time and +1 as and +1 in a frame of reference, we adjust the position of an object ∈  tô = +|̂ +1 −̂ |, wherê are the camera positions transformed onto the camera frame where is written. This also allows us to track objects that go out of view due to camera movements, based on our observability assumption (H1).
b) Consecutive frame alignment: Using the adjusted positions, we now want to find the optimal alignment between the estimated object states = {̂ ∶ 1 … } and the newly perceived objects Π +1 = { ∶ 1 … }. We define as a cost matrix to measure the dissimilarity between̂ and , where each entry ( ,̂ ) is determined by where denotes a type mismatch constant and (⋅, ⋅) a distance measure such as the 2 -distance. In our implementation we used the Munkres assignment algorithm [18] to find the optimal alignment, but this can be replaced by any other alignment algorithm. We create a new symbol system  +1 = { ∶ = 1 … } with ≤ and elements =̂ for which ( ,̂ ) < , where is a maximum threshold that depends on the reliability of the object detection output. We update the position and size of with Φ , the attributes of and assign a confidence value ∈ [0, 1] which will be used in the hypothesis reasoning component.
2) Hypothesis reasoning: The second part of our model considers objects, that have not been assigned successfully in the alignment process. This includes previously maintained anchorŝ ∈ witĥ ∉  +1 , as well as newly detected objects ∈ Π +1 . Based on our hypotheses on object state changes, we reason about occlusions, potential detection errors and other state changes to decide if their anchors should be maintained.
a) Occlusions and out of view: Objects are considered occluded if their maintained anchor̂ ∈ intersects with a detected object. More formally, if there exists ∈ Π +1 such that the bounding boxes overlap, i.e.,̂ ∩ ≠ ∅. Objects are considered out of view if they have not been assigned a percept and their predicted position̂ is outside of the agent's field of view. In both cases, the anchor is maintained in the world model and̂ is added to  +1 . If an object is removed from the view without an agent action, it is marked as a lost object. b) Object attachments: We define the semantic relation where an object is attached to another object and refer to them as child and parent respectively: if there exists a symbol ∈  +1 and a relation attached( , ), then the child is considered to be attached to the parent and updated to follow . This way, we can track , even when it is not detected, as long as its parent is visible. This relation is inferred from effects of executed agent actions, e.g., for pick-up(hand,obj), the world state changes from is-empty(hand) to holding(hand,obj), from which we can infer attached(obj,hand). Similarly for insert(obj,case) leading to inserted(obj,case) and to attached(obj,case). We assume that an object can only have one parent but one parent can have multiple children. However, more attachment relations can hold at the same time, allowing for an attachment hierarchy e.g., attached(plug,case) and attached(case,hand). In this case, an object anchor (e.g., for plug) will be maintained if it has a higher-order parent that has already been anchored (e.g., hand). In our implementation, we recursively check for attachment relations to maintain anchors for all children.

c) Confidence: Each anchor
∈  is assigned a confidence value ∈ [0, 1]. If the object is first detected, we assign = 0 and for each subsequent cycle where it is successfully aligned, is increased by a fixed increment conf + ∈ (0, 1), where we cap ≤ 1. If the object disappears, but has previously reached an anchoring threshold ≥ ℎ , it will be considered by the hypothesis reasoning component. Otherwise its confidence value will be decreased by a decrement conf − ∈ (0, 1) and when it reaches < 0 it will no longer be maintained in the world model. This allows us to handle false positives or negatives from noisy data, where objects appear or disappear intermittently (H3).

3) World model maintenance:
At each cycle we maintain a complete world model, where those that have not been consistently perceived ( < ℎ ) are not considered for hypothesis reasoning, while the rest ( ≥ ℎ ) are anchored. We can optionally define another confidence threshold ≥ ℎ to filter out objects, that we are confident about, to be considered for the world state inference (discussed in Sec. III-C). These thresholds allow the model to be adjusted to the application scenario that depends on the reliability of the object detection output (e.g., higher ℎ to handle flickering more detection errors).

C. World state inference and action execution
The final component of the pipeline consists of the world state inference. It uses the maintained symbol system  +1 to infer world states  +1 (⊇  +1 ) that additionally include relations between the maintained objects such as spatial relations (e.g., behind( ℎ, )) or object attachments (e.g., attached( ℎ, )) inferred from executed actions (insert( ℎ, )). The world state  +1 is then used for task planning to infer the next executable action. It relies on having correctly anchored objects, as an incorrect world model would yield wrong world states and may not find any executable actions. Finally, the complete world model and executed action are passed as input to the perceptual anchoring model in the subsequent cycle.

IV. EXPERIMENTAL EVALUATION
We perform a quantitative evaluation of our model and benchmark against Shamsian et al. [9] on their LA-CATER (Localisation Annotations) dataset on a snitch localisation task. We compare model performances when using a Faster R-CNN model as the object detector to using perfect perception (ground-truth object locations).

A. Dataset
The LA-CATER dataset [9] was generated from the CATER dataset [19] which contains synthetically rendered videos (10-second, 300-frame) of standard 3D objects (cube, sphere, cylinder, inverted cone) moving in space. The objects can perform four actions (slide, rotate, pick and place, contain), where 'contain' describes a cone moving on top of an object and containing it. The dataset contains 14K videos split into train (9.3K), validate (3.3K), and test (1.3K) datasets. Every video contains a 'snitch', a small golden sphere that is used as the target object for the localisation task. The LA-CATER dataset augments CATER with framelevel annotations and ground-truth locations for the entire video sequence as well as labels for four object permanence subtasks (visible, occluded, contained, carried).

B. Baselines
We compare our model to the neural model OPNet and a programmed heuristic model both proposed in [9] and reran them using the code provided by the authors. The heuristic model switches from tracking the target object to tracking the object located closest to the last known location of the target. OPNet is a deep network architecture that uses two LSTMs as reasoning modules to infer the object to track (if the target is carried or contained) and its exact location. OPNet has been trained and validated on the LA-CATER dataset and outperforms other baseline models, including four 'learned' models and two 'programmed' models, including the heuristic model. As the heuristic model was reported to perform similarly on carried tasks (IoU: OPNet 56.04% vs Heuristic 55.87%), we decided to include it in our evaluation.
Since we work with synthetically generated data, the detected objects can be considered more reliable and contain little to no noise. Therefore, we evaluate our AAPA model with = 6500 for the object alignment threshold, ℎ = 0.1 and confidence values conf + = conf − = 0.1. This means that objects that appear in two consecutive frames are considered for hypothesis reasoning. For object attachments, we use the action annotations from the LA-CATER dataset and trigger an attachment relation when a 'contain' action has been executed.
All models receive bounding box information from the object detector (a Faster R-CNN fine-tuned on the LA-CATER dataset). As misclassified objects can affect the overall performance of the models, we also perform comparisons using Perfect Perception, i.e., we use the ground-truth locations of the object bounding boxes. This allows us to compare only the failures that occur during reasoning.

C. Evaluation Metrics
Similar to [9], we use the mean intersection over union (IoU) of predicted and ground truth bounding boxes as an evaluation metric. However, in order to exclude biases from false predictions, we only compute the metric from the moment, when the snitch is first observed by the object detector, resulting in differing IoU values than reported in [9]. As we are mainly interested in the object location, less in the detected bounding box, performances might differ if predicted bounding box dimensions are wrong. Thus, we also use mean 2 -distances (given in pixels) of the centre points of the predicted bounding boxes as a second performance metric. As a reference, the video size is 360x240 pixel, the ground-truth size of the snitch ranges from 15x16 to 41x41 and of any object in the scene from 8x8 to 117x117.

D. Results
In the following we present results using object detection (OD) and perfect perception (PP) and both mean IoU and L 2 -distances. We compare performances on different task categories (visible, occluded, contained, carried) and discuss common failure cases and their causes.
1) Using object detection (OD): Table I.I shows a summary of the model performances comparing AAPA with OPNet and Heuristic by mean IoU and 2 -distance. AAPA outperforms the baseline models in all categories, except on visible tasks, where it performs similarly (AAPA: 88.67% vs OPNet: 88.98% vs Heuristic: 88.00%). AAPA's performance drops for occluded tasks, mainly due to the snitch moving in occlusion as we assume that objects do not move on their own (H1). Other errors included incorrect prediction of bounding box size for partial occlusions. Similarly, for contained tasks, most of the failures occurred when containment happened after the snitch was moving in occlusion. For carried tasks, AAPA performs a lot better than the other models (AAPA 68.25% vs OPNet's 56.04% vs Heuristic 51.92%). Main failures include moving in occlusion (while being carried) and anchoring errors caused by the threshold parameter in the object alignment. The mean IoU performance is also impacted by the fact that AAPA retains the last detected bounding box size before it became contained, while the ground truth size becomes smaller or larger, when 2) Using perfect perception (PP): When using ground truth bounding boxes instead of the object detector, no detection errors are propagated to the models, allowing us to evaluate their reasoning capabilities. All model performances improved, with AAPA improving the most on carried tasks, outperforming the baseline models in all tasks for both mean IoU and 2 -distance (Table I.II). For visible tasks, AAPA and Heuristic benefit the most from ground truth information, as there is no bounding box alignment error. The main failure cases are caused by moving in occlusion and anchoring errors. The impact of AAPA not adjusting the bounding box size while being carried in 3D space can be seen when comparing mean IoU to 2 -distances: the mean IoU for AAPA is better for occluded tasks (90.53%) compared to carried tasks (82.54%), but the mean 2 -distance shows the opposite: 2.51 for occluded vs 1.94 for carried (lower is better).
3) Discussions: While many deep learning approaches exist that address object permanence, they require large amounts of data and remain a black-box solution. This means that success and failure cases are not necessarily explainable and there is little room to investigate wrong reasoning processes. One of the main benefits of our approach is the transparency of our rule-based model, which makes it human readable. This allows us to investigate failure cases and to fine-tune our model parameters e.g., that anchoring failures occurred when objects moved in occlusion. Other failures were due to objects taking a bigger leap between two consecutive frames: Our model considered them as two different object instances, due to our assumption that object positions stay the same as last time they were seen (H1) and they do not 'teleport' (H2). If we increase , the threshold for consecutive frame alignment, these cases are anchored correctly, but would give a bigger leeway to detection errors, resulting in objects anchored to ghost objects. We experimented with these values and chose a threshold that yielded the maximum overall performance for this task.
The figures next to Table I show the distribution of mean 2 -distances for all 1,371 videos for Heuristic, OPNet, and AAPA using OD. AAPA produces more accurate predictions and with only a few outliers, representing the failures mentioned above. The biggest outlier for AAPA comes from a video, where the snitch was occluded in all frames but one. The baseline models (OPNet and Heuristic) will always produce a prediction for each video frame, even when the snitch is not visible at all. On the other hand, our AAPA model assumes that object state changes are consistent, and only maintains anchors for objects that are detected in multiple consecutive frames. Therefore, in this case, AAPA fails to maintain the anchor for the snitch and returns no prediction of its position, which the evaluation considers as a [0, 0] vector and returns an 2 -distance of | | given by the ground-truth position.

V. REAL-WORLD APPLICATION
We evaluate our model qualitatively and demonstrate its use in real-world applications on a collaborative gearbox assembly task with the Universal Robot UR5, an industrial robotic arm with 6DoF and 5kg payload. The robot is mounted with an Intel RealSense D435 camera, acting as an eye-in-hand system (Fig. 1). AAPA has been implemented as part of ICARUS, a cognitive architecture for modelling human cognition [20], which uses the world model for world state inference and task execution (Fig. 2C). For the object detection ( Fig. 2A) we use a Faster R-CNN that has been fine-tuned to object categories in the gearbox assembly domain (e.g., casing top/base, input subassembly). We found the optimal confidence parameters for this scenario to be A. Gearbox assembly scenario For autonomous navigation and intelligent collaboration, robots must track objects, even if they are no longer perceived by the detection system. For eye-in-hand robotic systems, this is particularly important as the grasping action generally results in losing sight of most surrounding objects. We demonstrate the use of AAPA on an assembly task, where the robot has to manipulate gearbox assembly parts in collaboration with a human operator. The cognitive architecture uses the world model maintained by AAPA to reason about the necessary actions to execute. The collaborative gearbox assembly consists of the following steps: 1) Robot attaches small hub cover onto casing top (case0).
2) Human screws in hub cover plugs.
3) Robot inserts input subassembly to casing base (case1). 4) Human moves casing base to the robot. 5) Robot mounts casing top to casing base.

B. Observations and results
The robot successfully executed the gearbox assembly task in real-time (see Fig. 3). We compare the maintained world state with and without our AAPA model. 1) Unique label assignments: The Faster R-CNN detects objects and assigns unique labels in the format typeN, where is the instance number assigned in order of appearance (e.g., case0, case1). When there are multiple objects of the same type, it can produce incorrect labels as the order of appearance changes or an object disappears from the view. AAPA maintains the unique labels of the anchors with the consecutive frame alignment (Fig. 3b,d).
2) Occlusion and out of view: Once objects are occluded or go out of view, the Faster R-CNN loses track of them, which can lead to failures in the world state inference and action execution. With AAPA, the anchors of these objects are maintained in the world model (Fig. 3b,e).
3) Handling detection errors: When objects are cluttered, stacked, or partially occluded, it sometimes detects the wrong object type or fails to detect them. State-of-the-art object detection models can be very error-prone due to environmental changes such as lighting conditions, or simply when they are viewed from different angles or in close-up when grasping. These models commonly produce flickering ghost objects that appear and disappear within a couple of frames. The robot needs to handle these detection errors correctly, otherwise it can result in an incorrect world state inference and task execution. Our model handles these errors using the confidence value that is updated at each cycle (Fig. 3c).
4) Attachment relations: The assembly task triggers object attachments (i.e., attached(small hub cover0, case0), attached(input subassembly0, case1)). When the robot picks up the casing top (Step 5), it fails to detect the surrounding objects, including the hub cover. However, since it is attached to the casing top, its anchor is maintained and updated to follow the casing top as it moves (Fig. 3e). Similarly, when the robot grasps objects during task execution, it sometimes fails to detect them. Since the pick-up action creates an attachment between the gripper and grasped object, the anchor is maintained and updated consistently.

VI. CONCLUSIONS
We presented an action-aware perceptual anchoring model (AAPA) that can handle object detection errors, out of view, and more complex scenarios such as invisible displacements by considering agent actions. We showed that AAPA outperforms baseline models on the LA-CATER dataset for a snitch localisation task. We demonstrated its use for robotic applications on a real-life gearbox assembly task between a human operator and a UR5. AAPA successfully maintains a reliable world model for the robot to complete the task. Our approach can be used for hierarchical attachments as well as other agent actions that cause attachments (e.g., screwing a plug onto a case). Given a list of agent actions that cause attachment relations, our model recursively checks for these relations and automatically infers the attachment hierarchy irrespective of the number of relations defined. In the future we will focus on using human action recognition in combination with (visual) commonsense reasoning to infer attachment relations automatically from agent actions and object types (e.g., screw action, container objects).