We propose a layered street view model to encode both depth and semantic information on street view images for autonomous driving. Recently, stixels, stix-mantics, and tiered scene labeling methods have been proposed to model street view images. We propose a 4-layer street view model, a compact representation over the recently proposed stix-mantics model. Our layers encode semantic classes like ground, pedestrians, vehicles, buildings, and sky in addition to the depths. The only input to our algorithm is a pair of stereo images. We use a deep neural network to extract the appearance features for semantic classes. We use a simple and an efficient inference algorithm to jointly estimate both semantic classes and layered depth values. Our method outperforms other competing approaches in Daimler urban scene segmentation dataset. Our algorithm is massively parallelizable, allowing a GPU implementation with a processing speed about 9 fps.
One popular model for road scene is the stixel world that simplifies the world using a ground plane and a set of vertical sticks on the ground representing obstacles @cite_13 . Stixels are compact and efficient representation for upright objects on the ground. The stixel representation can simply be seen as the computation of two curves. The first curve runs on the ground plane enclosing the free space that can be immediately reached without collision, and the second curve encodes the vertical objects boundary. In order to compute the stixel world, either depth map from semi-global stereo matching algorithm (SGM) @cite_25 or cost volume @cite_24 can be used. As with SGM, dynamic programming (DP) enables fast implementation for the computation of the stixels. Recently, @cite_22 demonstrated a monocular free-space estimation using appearance cues.
@cite_24 @cite_22 @cite_13 @cite_25
2161337244 2953373759 2509033610 1978260476
Ambitious driver assistance for complex urban scenarios demands a complete awareness of the situation, including all moving and stationary objects that limit the free space. Recent progress in real-time dense stereo vision provides precise depth information for nearly every pixel of an image. This rises new questions: How can one efficiently analyze half a million disparity values of next generation imagers? And how can one find all relevant obstacles in this huge amount of data in real-time? In this paper we build a medium-level representation named "stixel-world". It takes into account that the free space in front of vehicles is limited by objects with almost vertical surfaces. These surfaces are approximated by adjacent rectangular sticks of a certain width and height. The stixel-world turns out to be a compact but flexible representation of the three-dimensional traffic situation that can be used as the common basis for the scene understanding tasks of driver assistance and autonomous systems. This paper presents a compact and accurate representation of 3D scenes that are observed by a LiDAR sensor and a monocular camera. The proposed method is based on the well-established Stixel model originally developed for stereo vision applications. We extend this Stixel concept to incorporate data from multiple sensor modalities. The resulting mid-level fusion scheme takes full advantage of the geometric accuracy of LiDAR measurements as well as the high resolution and semantic detail of RGB images. The obtained environment model provides a geometrically and semantically consistent representation of the 3D scene at a significantly reduced amount of data while minimizing information loss at the same time. Since the different sensor modalities are considered as input to a joint optimization problem, the solution is obtained with only minor computational overhead. We demonstrate the effectiveness of the proposed multimodal Stixel algorithm on a manually annotated ground truth dataset. Our results indicate that the proposed mid-level fusion of LiDAR and camera data improves both the geometric and semantic accuracy of the Stixel model significantly while reducing the computational overhead as well as the amount of generated data in comparison to using a single modality on its own. In this paper we present Semantic Stixels, a novel vision-based scene model geared towards automated driving. Our model jointly infers the geometric and semantic layout of a scene and provides a compact yet rich abstraction of both cues using Stixels as primitive elements. Geometric information is incorporated into our model in terms of pixel-level disparity maps derived from stereo vision. For semantics, we leverage a modern deep learning-based scene labeling approach that provides an object class label for each pixel. Our experiments involve an in-depth analysis and a comprehensive assessment of the constituent parts of our approach using three public benchmark datasets. We evaluate the geometric and semantic accuracy of our model and analyze the underlying run-times and the complexity of the obtained representation. Our results indicate that the joint treatment of both cues on the Semantic Stixel level yields a highly compact environment representation while maintaining an accuracy comparable to the two individual pixel-level input data sources. Moreover, our framework compares favorably to related approaches in terms of computational costs and operates in real-time. In summer 2013, a Mercedes S-Class drove completely autonomously for about 100 km from Mannheim to Pforzheim, Germany, using only close-to-production sensors. In this project, called Mercedes Benz Intelligent Drive, stereo vision was one of the main sensing components. For the representation of free space and obstacles we relied on the so called Stixel World, a generic 3D intermediate representation which is computed from dense disparity images. In spite of the high performance of the Stixel World in most common traffic scenes, the availability of this technique is limited. For instance under adverse weather, rain or even spray water on the windshield results in erroneous disparity images which generate false Stixel results. This can lead to undesired behavior of autonomous vehicles. Our goal is to use the Stixel World for a robust free space estimation and a reliable obstacle detection even during difficult weather conditions. In this paper, we meet this challenge and fuse the Stixels incrementally into a reference grid map. Our new approach is formulated in a Bayesian manner and is based on existence estimation methods. We evaluate our new technique on a manually labeled database with emphasis on bad weather scenarios. The number of structures which are detected mistakenly within free space areas is reduced by a factor of two whereas the detection rate of obstacles increases at the same time.