Object shape provides important information for robotic manipulation; for instance, selecting an effective grasp depends on both the global and local shape of the object of interest, while reaching into clutter requires accurate surface geometry to avoid unintended contact with the environment. Model-based 3D object manipulation is a widely studied problem; however, obtaining the accurate 3D object models for multiple objects often requires tedious work. In this letter, we exploit Gaussian process implicit surfaces (GPIS) extracted from RGB-D sensor data to grasp an unknown object. We propose a reconstruction-aware trajectory optimization that makes use of the extracted GPIS model plan a motion to improve the ability to estimate the object's 3D geometry, while performing a pick-and-place action. We present a probabilistic approach for a robot to autonomously learn and track the object, while achieve the manipulation task. We use a sampling-based trajectory generation method to explore the unseen parts of the object using the estimated conditional entropy of the GPIS model. We validate our method with physical robot experiments across eleven different objects of varying shape from the YCB object dataset. Our experiments show that our reconstruction-aware trajectory optimization provides higher-quality 3D object reconstruction when compared with directly solving the manipulation task or using a heuristic to view unseen portions of the object.
In this paper we examine the correlation between the information content and the spatial realization of range measurements taken by a mapping robot. To do so, we consider the task of constructing an occupancy grid map with a binary Bayesian filter. Using a beam-based sensor model (versus an additive white Gaussian noise model), we prove that any controller tasked to maximize a mutual information reward function is eventually attracted to unexplored space. This intuitive behavior is derived solely from the geometric dependencies of the occupancy grid mapping algorithm and the monotonic properties of mutual information. Since it is dependent on both the robot's position and the uncertainty of the surrounding cells, mutual information encodes geometric relationships that are fundamental to robot control, thus yielding geometrically relevant reward surfaces on which the robot can navigate. We also provide an algorithmic implementation for computing mutual information and show that its worst-case time and space complexities are quadratic and linear, respectively, with respect to the map's spatial resolution. Lastly, we present the results of experiments employing an omnidirectional ground robot equipped with a laser range finder. Our experimental results support our theoretical and computational findings. We build on recent fast and accurate 3-D reconstruction techniques to segment objects during scene reconstruction. We take object outline information from change detection to build 3-D models of rigid objects and represent the scene as static and dynamic components. Object models are updated online during mapping, and can integrate segmentation information from sources other than change detection. Active scene exploration incorporates object recognition methods for analyzing a scene of partially known objects and exploration approaches for autonomous modeling of unknown parts. In this work, recognition, exploration, and planning methods are extended and combined in a single scene exploration system, enabling advanced techniques such as multi-view recognition from planned view positions and iterative recognition by integration of new objects from a scene. Here, a geometry based approach is used for recognition, i.e. matching objects from a database. Unknown objects are autonomously modeled and added to the recognition database. Next-Best-View planning is performed both for recognition and modeling. Moreover, 3D measurements are merged in a Probabilistic Voxel Space, which is utilized for planning collision free paths, minimal occlusion views, and verifying the poses of the recognized objects against all previous information. Experiments on an industrial robot with attached 3D sensors are shown for scenes with household and industrial objects. We develop a computationally efficient control policy for active perception that incorporates explicit models of sensing and mobility to build 3D maps with ground and aerial robots. Like previous work, our policy maximizes an information-theoretic objective function between the discrete occupancy belief distribution (e.g., voxel grid) and future measurements that can be made by mobile sensors. However, our work is unique in three ways. First, we show that by using Cauchy-Schwarz Quadratic Mutual Information (CSQMI), we get significant gains in efficiency. Second, while most previous methods adopt a myopic, gradient-following approach that yields poor convergence properties, our algorithm searches over a set of paths and is less susceptible to local minima. In doing so, we explicitly incorporate models of sensors, and model the dependence (and independence) of measurements over multiple time steps in a path. Third, because we consider models of sensing and mobility, our method naturally applies to both ground and aerial vehicles. The paper describes the basic models, the problem formulation and the algorithm, and demonstrates applications via simulation and experimentation. We present an active detection model for localizing objects in scenes. The model is class-specific and allows an agent to focus attention on candidate regions for identifying the correct location of a target object. This agent learns to deform a bounding box using simple transformation actions, with the goal of determining the most specific location of target objects following top-down reasoning. The proposed localization agent is trained using deep reinforcement learning, and evaluated on the Pascal VOC 2007 dataset. We show that agents guided by the proposed model are able to localize a single instance of an object after analyzing only between 11 and 25 regions in an image, and obtain the best detection results among systems that do not use object proposals for object localization. It is important to enable a robot to manipulate a target object that has no 3-D model information and is situated in an environment with other unknown objects nearby. This poses an open problem of how to combine perception and manipulation to enable the robot to build an appearance-based model of the target object on the spot to facilitate further manipulation of the object while avoiding the other unknown obstacles in the way. In this paper, we introduce an approach to enable a continuum manipulator, which is apt to maneuver through a crowded environment, to gradually build a 3-D surface model of the target object by moving an RGB-D sensor around the object while also detecting and avoiding surrounding unknown obstacles. Our approach interleaves perception and manipulation such that perception guides the manipulator movement, which in turn allows more perception of the target object for object model building and further manipulator motion. Our approach is characterized by a progressive strategy to register RGB-D images of the target object to build and extend a partial model of the object and the corresponding motion planning strategy for the continuum robot to carry out model building and avoid obstacles at the same time. To demonstrate the effectiveness of our approach, experiments on progressive model building of real objects from real RGB-D images are conducted, where a simulated continuum robot plans and executes its motion to carry the RGB-D camera around a target object for taking those images in an augmented reality setting.