Graph Neural Networks (GNNs), which expressively represent graph structured data consisting of nodes and edges by iteratively aggregating features from target node's neighbors, have gained substantial attention in recent years, showing successes in various downstream tasks working on graphs [1]}, [2]}, [3]}, [4]}, [5]}, [6]}. As such model architectures explicitly leverage the relationships between connected instances when representing them, there have been recent attempts to use GNNs on the object detection task to capture interactions among objects [7]}, [8]}, [9]}. For instance, GAR [8]} constructs a context graph, which considers interactions between objects and scenes but also between the objects themselves by forming objects and scenes as nodes and their connections as edges, and then represents the graph with GNNs. Similarly, Relation R-CNN [9]} generates semantic and spatial relation networks with objects as nodes, using pre-built co-occurrence matrix and distances between objects, respectively, for forming edges. However, both GAR [8]} and Relation R-CNN [9]} have an obvious limitation that the edge generation procedure is not end-to-end trainable, but based on simple heuristics using pre-calculated statistics of co-occurrences between instances. On the other hand, SGRN [7]} learns a spatial-aware relation network between objects based on visual features and spatial distances of initial proposals, which is then forwarded to GNNs for obtaining representations for objects. Note that our work has key differences against such a relevant work: instead of working on all object proposals, we first largely reduce them with non-maximum suppression (NMS) [15]}, [16]}, and then aim at improving only the uncertain objects that the model is largely confused about by generating semantic and spatial yet directed edges from certain objects to uncertain ones, which greatly reduces computational costs especially when dealing with lots of objects in a single image (e.g., DOTA [17]}).
