We focus on the problem of estimating the orientation of the ground plane with respect to a mobile monocular camera platform (e.g., ground robot, wearable camera, assistive robotic platform). To address this problem, we formulate the ground plane estimation problem as an inter-mingled multi-task prediction problem by jointly optimizing for point-wise surface normal direction, 2D ground segmentation, and depth estimates. Our proposed model -- GroundNet -- estimates the ground normal in two streams separately and then a consistency loss is applied on top of the two streams to enforce geometric consistency. A semantic segmentation stream is used to isolate the ground regions and are used to selectively back-propagate parameter updates only through the ground regions in the image. Our experiments on KITTI and ApolloScape datasets verify that the GroundNet is able to predict consistent depth and normal within the ground region. It also achieves top performance on ground plane normal estimation and horizon line detection.
Augmented reality combines real footage taken of a scene with virtual elements. However, most current methods rely on camera localisation and 3D reconstruction or point cloud generation in order to integrate augmented reality to the footage. In contrast, in this work we present a novel method to augment virtual reality to the scene based on the recognition of dominant planes in interior scenes. Our method uses a rule system to select predefined decisions on a set of variables in order to infer dominant planes in the scene. For this, we propose to combine information from texture features, a measure of blurring in the dominant planes, and a segmentation of regions in a scene based on superpixels. The rule system infers the regions corresponding to the dominant planes, whilst light intensity in the scene is inferred from segmented regions. We also propose an approach to remove regions misclassified as dominant planes. Finally, the floor in the scene is recognised as the most dominant plane and then replaced with an augmented texture. We demonstrate our approach in a video sequence where our method is applied in a frame-to-frame basis, thus, for each single image in the video sequence, the floor is automatically recognised as the most dominant plane and then replaced with a virtual texture and furthermore, whose appearance is modified according to our coarse light model inferred from our approach. We present a novel method to recognise planar structures in a single image and estimate their 3D orientation. This is done by exploiting the relationship between image appearance and 3D structure, using machine learning methods with supervised training data. As such, the method does not require specific features or use geometric cues, such as vanishing points. We employ general feature representations based on spatiograms of gradients and colour, coupled with relevance vector machines for classification and regression. We first show that using hand-labelled training data, we are able to classify pre-segmented regions as being planar or not, and estimate their 3D orientation. We then incorporate the method into a segmentation algorithm to detect multiple planar structures from a previously unseen image. We propose an algorithm to detect planes in a single image of an outdoor urban scene, capable of identifying multiple distinct planes, and estimating their orientation. Using machine learning techniques, we learn the relationship between appearance and structure from a large set of labelled examples. Plane detection is achieved by classifying multiple overlapping image regions, in order to obtain an initial estimate of planarity for a set of points, which are segmented into planar and non-planar regions using a sequence of Markov random fields. This differs from previous methods in that it does not rely on line detection, and is able to predict an actual orientation for planes. We show that the method is able to reliably extract planes in a variety of scenes, and compares favourably with existing methods.