Grounding Predicates through Actions

Symbols representing abstract states such as"dish in dishwasher"or"cup on table"allow robots to reason over long horizons by hiding details unnecessary for high-level planning. Current methods for learning to identify symbolic states in visual data require large amounts of labeled training data, but manually annotating such datasets is prohibitively expensive due to the combinatorial number of predicates in images. We propose a novel method for automatically labeling symbolic states in large-scale video activity datasets by exploiting known pre- and post-conditions of actions. This automatic labeling scheme only requires weak supervision in the form of an action label that describes which action is demonstrated in each video. We use our framework to train predicate classifiers to identify symbolic relationships between objects when prompted with object bounding boxes, and demonstrate that such predicate classifiers can match the performance of those trained with full supervision at a fraction of the labeling cost. We also apply our framework to an existing large-scale human activity dataset, and demonstrate the ability of these predicate classifiers trained on human data to enable closed-loop task planning in the real world.


I. INTRODUCTION
Enabling robots to perform long horizon tasks such as preparing meals or assembling furniture is a widely studied problem. Long horizon planning is rooted in early AI work that studied how to give robots the ability to reason through symbols [1]. Symbols allow robots to abstract away low-level details of the environment and perform logical reasoning at a higher level [2][3][4][5]. However, giving robots the ability to perceive symbols in real-world environments is still an unsolved problem. Without some form of sensory grounding, propositions such as "drawer is open" are simply a set of symbols that lack any actionable meaning for the robot. Thus, robots often execute symbolic plans without closed-loop visual feedback-if a robot fails to open a drawer, it has no way of knowing, because it does not know what "drawer is open" looks like. State-of-the-art methods for learning visual groundings of symbols require large amounts of annotated data [3,[6][7][8]. However, obtaining annotations of symbolic states is prohibitively expensive due to the sheer number of propositions in a single image. Furthermore, densely-labeled datasets cannot easily be transferred between domains, since different planning problems often require different symbols. The   More examples can be found on our project website: https://sites. google.com/stanford.edu/groundingpredicates. from table" changes the propositions "hand is empty" and "cup is on table" to "hand is holding cup" and "cup is not on table"). Labeling a dataset with actions is easier than with symbolic states: it requires first defining the preand post-conditions once for each action class, and then annotating each visual example with only its action. Then, partial labels of symbolic state come for free with the action pre-and post-conditions. For task planning applications, the action pre-and post-conditions will already be defined in the task planning domain.
With this novel partial labeling scheme, we train networks to infer symbolic states in images in 20BN Something Something v2 (20BN) [9], a large-scale human activity dataset. The result is a system that can identify symbolic states in real-world environments for robot manipulation (Fig. 1). These classifiers open up many opportunities for long horizon planning in the real world, such as closed-loop task planning or learning from demonstrations of sequential actions.
The main contributions of this paper are three-fold. 1) We provide a framework for automatically labeling symbolic arXiv:2109.14718v2 [cs.RO] 4 Mar 2022 states in real-world image frames through actions, using the logical formalism of Planning Domain Description Language (PDDL) [10]. 2) We evaluate this framework on two domains, 20BN and Gridworld, and analyze its advantages and disadvantages. 3) We demonstrate the ability of predicate classifiers that are only trained on human demonstrations in 20BN to be used for closed-loop task planning in a real-world robot environment.

II. RELATED WORK A. Visual Relationship Detection
The computer vision community has a recently growing body of work on symbolic state detection, under the names visual relationship detection and scene graph generation [3,[6][7][8][11][12][13][14][15]. A scene graph represents image scenes as graphs where nodes are objects or attributes like grass and green, and edges are relationships between nodes, e.g., grass − green [16]. The aim of visual relationship detection is to enable semantic scene understanding and to connect visual concepts with natural language. The three most common datasets for visual relationship detection are CLEVR (100k synthetic images with 3 objects and 16 predicates) [16], VRD (5k real world images with 100 objects and 70 predicates) [6], and Visual Genome (100k real world images with 34k objects and 110k predicates) [17]. A key challenge of visual relationship detection (i.e., symbolic state detection) is the immense number of possible states. Even for the relatively small VRD dataset, 70 binary predicates with 100 objects results in 70 * 100 * 99 = 693k possible propositions. VRD and Visual Genome both rely on manual annotation, and thus obtaining fully labeled symbolic states is infeasible. Our method for labeling symbolic states requires defining the pre-and post-conditions once for each action class, and then partial symbolic state labels are automatically generated for all examples in the dataset.
It is important to note that our method only works when before and after images of the actions being performed are provided; VRD and Visual Genome do not meet this requirement. Furthermore, learning from pre-and post-conditions of actions only works in domains where the symbolic state can be manipulated by actions. Propositions such as "sky is blue" may be difficult to learn through actions, for example. The Action Genome dataset [18] introduces spatio-temporal scene graphs with the purpose of using symbolic state changes across actions to improve action recognition in videos. However, Action Genome, like VRD and Visual Genome, relies on manual annotations, and furthermore does not include negative propositions. Action Genome is based on the Charades dataset for video activity recognition [19]. Charades is one of numerous activity recognition datasets, like ActivityNet [20], Epic-Kitchens [21], and Atomic Visual Actions (AVA) [22], where labeling symbolic states with our method could be possible. However, we choose to evaluate our method on the 20BN Something Something v2 dataset [9], a video activity recognition dataset that focuses on the manipulation of everyday objects and has been shown to be useful for learning robotic manipulation skills [23,24].
The focus of this work is providing an alternative framework for labeling symbolic states, not proposing a novel model architecture for visual relationship detection. To evaluate our labeling method, we therefore train a model whose architecture is based on a state-of-the-art model for visual relationship detection [25]. However, our approach is agnostic to the specific model architecture.

B. Visual Grounding for Planning
A long-term goal for this work is to help bridge the gap between perception and long horizon planning in robotics. A majority of task planning methods operate solely at the level of symbolic abstractions, assuming that in a real world application, there would be some way to perceive symbols in the environment. Some works attempt to learn the symbols themselves, so that perception and planning can work together in an end-to-end fashion [2,26]. In these works, the learned symbols are not easily interpretable, and guaranteeing correct behavior over a long horizon for complex domains is therefore difficult. Our labeling method benefits frameworks that rely on direct supervision of symbols to learn to plan over long horizons, such as [4,[27][28][29][30]. These systems contain submodules that require annotations of predicates in images. This dependency restricts these systems to simulated environments where symbolic states are easily obtainable or to real-world domains small enough that manually annotating symbolic states is feasible.
The work most directly related to ours is perhaps an imitation learning system that learns to ground predicates associated with the pre-and post-conditions of demonstrated actions [31]. With grounded predicates, their system can then generalize to new tasks using classical planning. Specifically, their Visuospatial Skill Learning module learns to classify the predicate far(a) for a cuboid block a using demonstrations from two actions: push(a) : ¬far(a) → far(a) and pull(a) : far(a) → ¬far(a). While they learn to ground predicates from actions, the complexity of their problem is much simpler: one predicate. Our method enables learning to ground any number of predicates simultaneously from actions with arbitrarily complex pre-and post-conditions.

C. Weak Supervision
Our framework might be considered a form of weak supervision, which deals with classification problems where humans define annotation functions (e.g. pre-and post-conditions of actions) that provide noisy labels for unlabeled datasets to alleviate the effort of full labeling. Although our dataset is not strictly unlabeled (we assume action labels are available), methods from this relatively new field could be applied to our framework in future work [32,33]. In Appx. A, we provide back-of-the-envelope calculations to demonstrate how much annotation time weak supervision can save. Our weak supervision framework makes it easier to train predicate classifiers for custom task planning domains, which often contain symbols not transferable from other domains.

III. PLANNING DOMAIN DESCRIPTION LANGUAGE
This paper uses the Planning Domain Description Language (PDDL) [10] to describe symbolic domains. A PDDL domain can be specified with a tuple (Φ, A), where Φ is the set of predicates and A is the set of actions. A PDDL problem is a tuple (O, s init , g), where O is the set of environment objects, s init is the initial state, and g is the goal-specified as a first-order logic formula-to be satisfied.

A. Predicates and Propositions
A predicate has a fixed number of parameters, each of which can be instantiated with an object in O to form a proposition. For example, the predicate in(a, b) has two parameters a and b, and instantiating the parameters with arguments box and hand results in the proposition in(box, hand). In this paper, we use "positive proposition" to refer to atomic formulas (e.g., in(box, hand)) and "negative proposition" to refer to negations of atomic formulas (e.g., ¬in(box, hand)). Let P be the set of all possible positive propositions and N be the number of these propositions (N = |P|).

B. States
Symbolic states are conjunctions (∧) of propositions, e.g., in(box, hand) ∧ above(box, table) ∧ ¬on(box , table). PDDL follows the closed-world assumption, which means that propositions that are not explicitly specified are assumed to be false. Thus, we can also represent states as the set of all true propositions, where propositions not in the set are false by default. Let s denote closed-world states, and let S be the set of all possible states. Note that |S| = 2 N .
Under the open-world assumption, propositions that are not explicitly specified in the state are unknown-neither true nor false. Here, symbolic states can be represented as a pair of signed states (s + , s − ), where s + is the set of positive propositions, s − is the set of negative propositions, and propositions that are excluded from s + and s − are unknown. Letŝ denote open-world states represented in this manner.
In practice, we represent closed-world states as boolean vectors s ∈ {0, 1} N and open-world states as boolean matricesŝ ∈ {0, 1} 2×N , where the first and second row correspond to positive and negative states, respectively.

C. Actions
Actions are defined by their pre-conditions-a firstorder logic formula that must be true before performing the action-and post-conditions-a formula that is applied to the symbolic state after the action. These formulas can include universal (forall), existential (exists), and conditional (when) quantifiers in PDDL. A symbolic planner aims to find a sequence of actions that starts at the initial state s init and ends up in a symbolic state that satisfies the goal g.

IV. PARTIAL STATE LABELS FROM ACTIONS
Our goal is to use a PDDL specification to automatically label symbolic states in an entire dataset of videos only annotated with actions. In this section, we formalize our method for obtaining partial state labels from pre-and post-conditions of actions and discuss how to use these partial labels to train symbolic state classifiers. An overview of the pipeline is provided in Fig. 2.

A. Predicate Classification Network
Let M be the maximum number of parameters of any predicate in P. Given I as an RGB image of any dimension, and M bounding boxes (b 1 , . . . , b M ) of M objects representing ordered predicate arguments, the network outputs a vectorỹ of P probabilities, one for each predicate in P: The benefit of using object bounding boxes is that we can query the predicate network for propositions with specific argument ordering, such as in(spoon, cup) or in(cup, spoon). The predicate network would then output a prediction based on its understanding of what it looks like for argument a to be in argument b. We assume that most robot manipulation applications will already rely on object tracking for planning and control, and therefore obtaining object bounding boxes for predicate detection would not incur any additional cost. However, using bounding boxes is an implementation choice that is orthogonal to our proposed method of labeling symbolic states with pre-and post-conditions. A classifier that predicts propositions without bounding boxes could also be used in place of the predicate classifier for environments where bounding boxes are difficult to obtain.
Details of the network architecture that we use in the experiments can be found in Appx. C.

B. Obtaining Partial State Labels from Actions
The logic formulas that describe pre-and post-conditions can be arbitrarily complex with deeply nested compound formulas. To make these nested formulas usable for computing a neural network loss for symbolic state predictions, we first convert them via first-order logic algebra into a flattened form called disjunctive normal form (DNF). DNFs are written as disjunctions (∨) of conjunctions (∧) of positive and negative propositions. Each conjunction in a DNF can be interpreted as a partially-specified state under the open-world assumption.
With this interpretation, a pre-condition DNF represents a set of possible partial states before performing the action, and a post-condition DNF represents a set of possible partial states after. These candidate states are partial because actions only specify conditions relevant to the action itself. For  pred(a, b)). The predicate classifier outputs C predictions each forỹpre,ỹpost ∈ [0, 1] P , where P is the number of predicates in the symbolic domain. These predicate vectors are transformed into symbolic state vectors ypre, ypost ∈ [0, 1] N , where N is the number of propositions. The ground truth labels used to evaluate these predictions come from partial state labelsŝpre,ŝpost ∈ {0, 1} 2×N , obtained from the PDDL definition of the action pick(a) on the right. DNF cross entropy is used to compare the predictions to the partial state labels. At test time, a single image is fed into the network along with bounding boxes for the predicate arguments, and the predicate classifier predicts the symbolic state of the image. example, the action pick(box) may require ¬in(box, hand) as a pre-condition, but does not care whether open(door) is true and is not even aware of door's existence. The candidate states in a pre-or post-condition DNF can encode varying amounts of information about the open world (i.e., specifying anywhere from 1 to N propositions), and some candidates may be supersets of others. A DNF is satisfied if and only if at least one of the candidate states in the DNF is true (Eq. 5). In other words, when an action is performed on a concrete symbolic state, at least one candidate state in the pre-condition DNF will be true before performing the action, and at least one in the post-condition DNF will be true after. Although it is impossible to know which candidate is true without more information, we do know the truth value of the propositions that appear in all the candidate states.
To label symbolic states using pre-and post-conditions, we therefore collapse each DNF into the largest single partial state that satisfies all its conjunctions. This is equivalent to the intersection of all the positive and negative states.
If collapsing a pre-condition DNF results in the empty set, this means that the pre-conditions are too general to provide labels for symbolic state. Post-conditions with empty collapsed DNFs means that there is a conditional effect (when) that may result in the action producing no changes. Both cases can be avoided by making the pre-conditions more descriptive. For example, if the action close(a) contains the conditional post-condition is-closable(a) =⇒ closed(a), the action can be redefined so that is-closable(a) is a pre-condition and the post-condition is simply closed(a).
Letŝ pre (a) : A → S 2 andŝ post (a) : A → S 2 be functions that return collapsed pre-and post-condition DNFs, respectively, for action a. These functions provide a way to label propositions whose values are guaranteed to be known before or after an action is performed. Any other proposition could be either true or false without violating a pre-or post-condition, so nothing can be said about their ground truth values. A formal proof for this statement can be found in Theorem 1.1.

C. DNF Cross Entropy Loss
As summarized in Fig. 2, a data point in the training set consists of an action a along with its pre-and postconditions, a pair of images (I pre , I post ) corresponding to the action's before and after state, and bounding boxes of the objects whose predicates one wishes to identify. Let y pre ∈ [0, 1] N and y post ∈ [0, 1] N be the predicted outputs of the network. The ground truth labels derived from the pre-and post-conditions of a can be represented as a pair of matricesŝ pre (a),ŝ post (a) ∈ {0, 1} 2×N .
To measure how much a network prediction output agrees with an action's pre-and post-conditions, we define a modified cross entropy loss to handle DNFs. Here, σ(y) is the sigmoid function.
The loss function used to train the network on a pre-and post-condition pair is: loss(y pre , y post ;ŝ pre ,ŝ post ) = CE DN F (y pre ,ŝ pre )

V. EXPERIMENTS
In the following experiments, we apply our framework to a large-scale real-world dataset, where obtaining complete ground truth labels is impractical. We then apply these trained classifiers to perform closed-loop task planning in a real robot environment. Finally, we evaluate the effectiveness of using partial labels of symbolic state from action preand post-conditions compared to complete ground truth labels in a toy Gridworld environment.

A. Learning Predicates from Large-Scale Datasets
The 20BN dataset [9] contains 220, 847 video examples of 174 manipulation actions. Although 20BN does not label bounding boxes for the action arguments, we use the bounding boxes from Something-Else [34]. We have defined the pre-and post-conditions for all but two of the actions-"putting # of sth onto sth" and "stacking # of sth"-since the # variable cannot be easily described as a PDDL object. We have also defined 35 predicates relevant to the 172 actions. The full PDDL description can be found on the project website.
The purpose of this experiment is to show that our framework makes it possible to train symbolic state classifiers on large-scale real world datasets without relying on expensive manual annotations. Back-of-the-envelope calculations in Appx. A indicate that using pre-and post-conditions to partially label the symbolic states in all 132, 853 videos of our subset of 20BN would take an expert roughly four 8-hour work days. By contrast, obtaining the equivalent partial state labels with manual annotation would take 690 8-hour work days.
1) Setup: We train the predicate classifiers using DNF cross entropy (DNF CE), as well as a weighted version of cross entropy (DNF WCE) using Class-Balanced Loss [35] to overcoming imbalances in the predicate distribution. There is no oracle because we do not have ground truth labels of the full symbolic state in this real-world dataset.
2) Results: Both DNF CE and DNF WCE are able to learn the predicates specified in the 20BN PDDL with 0.96 train F1 and 0.92 test F1. Although the two achieve the same test F1 scores (within 0.001), the predicate F1 scores in Table I reveal that overall, DNF WCE performs better on the less frequent predicates, increasing the average predicate F1 score from 0.60 to 0.65.
Given that each proposition in 20BN is positive 36% of the time on average, a random classifier that outputs positive propositions with 0.36 chance would be expected to get 0.36 F1. Thus, an F1 score of 0.65 is significantly better than random. Both DNF CE and DNF WCE achieve 0.93 test accuracy. Fig. 1 shows qualitative examples of DNF WCE.
3) Transfer to Real Robot Domains: To demonstrate the zero-shot transferability of predicate classifiers trained with our framework to robot manipulation domains, we directly apply the DNF WCE classifier trained on 20BN to a robot pick-and-place setting for closed-loop task planning (Fig. 3).
For our real robot experiment, we create a pick-and-place environment where the robot needs to clear fruit off the table and store them in appropriate containers. We use the YOLOv5 object detector [36] for predicate bounding boxes along with Mediapipe object tracking [37] for smoother tracking between frames. The predicate classifier then sends a symbolic state prediction to a task planner that searches for a sequence of actions that satisfies the PDDL goal. The whole pipeline (from perception to task planning) operates at 10Hz and therefore allows closed-loop task planning that can respond to failures and disturbances. A video of this demonstration is provided in the supplementary material.  In this closed-loop task planning example, a user first gives the task planner a high-level goal in the form of a logical formula such as in(banana, drawer). The predicate classifier then predicts the symbolic state of the environment from RGB image data and sends it to a task planner. The task planner performs symbolic tree search to produce a sequence of actions to accomplish the goal, using 20BN actions such as "open sth" or "put sth into sth". As the actions are executed by low-level controllers, the task planner continues to re-plan based on updated symbolic state predictions given by the predicate classifier.

B. Partial vs. Full State Labels in Gridworld
Full symbolic state labels are ideal for training predicate classifiers. However, obtaining such labels is only practical for small domains or simulated environments with direct access to the symbolic state. To evaluate the effectiveness of training on partial vs. complete state labels, we use the Gridworld environment (Fig. 4), where we have full control of the symbolic state.
1) Gridworld Environment: In this environment, an agent needs to obtain a trophy from inside a chest. There are 8 objects: agent, trophy, chest, chest key, door, door key, room a, and room b. There are 6 predicates: The actions are defined with complex pre-and postconditions, making use of universal (forall), existential (exists), and conditional (when) quantifiers.
To generate training data, we first sample a random symbolic state s 0 , where each proposition in s 0 has a 5% chance of being true. There are 172 propositions in Gridworld, so on average, a randomly sampled state will have 8 propositions. Then, given an action with its pre-and postconditionsŝ pre andŝ post , we generate two symbolic states: We then render s pre and s post to obtain images I pre , I post .
2) Setup: We compare an Oracle trained on full state labels to two models. The first, DNF CE, is trained with DNF cross entropy (Eq. 7). Because partial labels contain less information than full labels, we expect DNF CE to take longer to train than Oracle and/or perform slightly worse.
The third model is an ablation study to test whether the model benefits from seeing the visual change induced by an action in order to visually ground symbols. This ablation, Half DNF, trains DNF CE on only the pre-or post-conditions of any given action instance, but not both. With normal DNF training, a single data point comes with a pair of images: one before and one after the action. For the ablation, we feed the model with one image from the pair: either before or after. Across all the data, the model will see examples of both the pre-and post-conditions for each action, but never both for a single action instance. Because this ablation only sees half the data, we double the size of its dataset for a fair comparison. We expect Half DNF to perform worse than DNF CE.
Oracle and DNF CE are trained on 10, 000 examples of actions, and Half DNF is trained on 20, 000. All of them are evaluated against the full ground truth state (as opposed to the partial DNF state) on the same test set of 10, 000 examples. Using the full state allows us to get a clearer picture of the generalization abilities of these models.
3) Results: As shown in Fig. 4, all the models achieve a training F1 score of nearly 1, indicating that they finish learning by 20  DNF's worse performance indicates that seeing the visual changes induced by each action is more beneficial than simply receiving more data (twice the amount).
Increasing the dataset size from 10, 000 to 100, 000 results in perfect test scores for all the DNF models for all the predicates (results not shown). Because DNF training uses partial labels, it requires more data to match the effectiveness of training on complete labels. However, in practice, acquiring a large dataset of actions should be easier than acquiring a small dataset annotated with complete symbolic states (Appx. A). Furthermore, in robotics applications where symbolic state classifiers might be used to determine whether the pre-or post-conditions of an action are satisfied, being able to classify propositions outside of the contexts provided by DNF labels may not be necessary.

VI. CONCLUSION
In this work, we have presented a framework for extracting partial symbolic states from action pre-and post-conditions, which can be used to label large datasets with less effort. This new method drastically reduces the cost of labeling large-scale real-world datasets (690 vs. 4 work days for 20BN). Yet, predicate classifiers trained with our method are still able to nearly match the performance of models trained with full ground truth labels, as shown with our Gridworld experiment. Our closed-loop task planning example demonstrates that predicate classifiers trained on large-scale real-world datasets can be applied to real robot domains. In many cases, predicates learned from a general large-scale dataset may not be applicable to custom task planning domains, where accuracy is critical. However, our labeling framework would perhaps be most useful for these very applications where collecting new datasets is necessary. For these situations, our framework makes obtaining real-world symbolic state classifiers much more feasible.
This work opens up many opportunities for long horizon planning in the real world. Other interesting avenues of work include using natural language to learn the pre-and post-conditions of actions [38,39], or combining visual groundings of symbols with natural language [3,40].