Traffic congestion is a widespread problem. Dynamic traffic routing systems and congestion pricing are getting importance in recent research. Lane prediction and vehicle density estimation is an important component of such systems. We introduce a novel problem of vehicle self-positioning which involves predicting the number of lanes on the road and vehicle's position in those lanes using videos captured by a dashboard camera. We propose an integrated closed-loop approach where we use the presence of vehicles to aid the task of self-positioning and vice-versa. To incorporate multiple factors and high-level semantic knowledge into the solution, we formulate this problem as a Bayesian framework. In the framework, the number of lanes, the vehicle's position in those lanes and the presence of other vehicles are considered as parameters. We also propose a bounding box selection scheme to reduce the number of false detections and increase the computational efficiency. We show that the number of box proposals decreases by a factor of 6 using the selection approach. It also results in large reduction in the number of false detections. The entire approach is tested on real-world videos and is found to give acceptable results.
A similar problem to self-positioning has been handled in @cite_16 using spatial RAY features. It tries to predict the position of the vehicle on the road from an input video stream. However, they predict at most 3 lanes at a time either on the left or right hand side, whereas we have upto 6 lanes in our database with upto 3 lanes being on each side. In addition, there experiments are spread over only 2 days with similar road conditions. Our experiments are more rigorous, spreading over 5 days and the data includes 6 different road conditions. They have low traffic density whereas we have moderate-to-high traffic density. Thus direct comparison between these two approaches may not be possible.
@cite_16
2019933614 2110405746 2745467776 2342840547 2070708339 2029841124 2740071316 2757356831 2964332990 2790329984 2802545380 2110235183 2592008540 1989185123 2048960138 1978260476 2157076480 2761595090 2076943964 2335306738
Assigning the ego-vehicle to a lane is not only beneficial for navigation but will be an essential element in future Advanced Driver Assistance Systems. This paper describes an approach for ego-lane index estimation using only a monocular camera and no additional sensing equipment like, e.g., the typically employed GPS and Inertial Measurement Unit. Key aspect of the approach are SPatial RAY (SPRAY) features which represent the spatial layout of the road in the visual scene. The proposed method perceives a variety of local visual properties of the scene by means of base classifiers operating on patches extracted from camera images. The spatial arrangement of these local visual properties are captured using SPRAY features. With a boosting classifier trained on these features the ego-lane index is obtained. The system is evaluated on low traffic density and complementary to an object-based approach suitable for heavy traffic. In the conducted experiments, the proposed approach reaches recognition rates of 93 to 97 on individual highway images without applying any kind of temporal filtering. Learning and then recognizing a route, whether travelled during the day or at night, in clear or inclement weather, and in summer or winter is a challenging task for state of the art algorithms in computer vision and robotics. In this paper, we present a new approach to visual navigation under changing conditions dubbed SeqSLAM. Instead of calculating the single location most likely given a current image, our approach calculates the best candidate matching location within every local navigation sequence. Localization is then achieved by recognizing coherent sequences of these “local best matches”. This approach removes the need for global matching performance by the vision front-end - instead it must only pick the best match within any short sequence of images. The approach is applicable over environment changes that render traditional feature-based techniques ineffective. Using two car-mounted camera datasets we demonstrate the effectiveness of the algorithm and compare it to one of the most successful feature-based SLAM algorithms, FAB-MAP. The perceptual change in the datasets is extreme; repeated traverses through environments during the day and then in the middle of the night, at times separated by months or years and in opposite seasons, and in clear weather and extremely heavy rain. While the feature-based method fails, the sequence-based algorithm is able to match trajectory segments at 100 precision with recall rates of up to 60 . In recent years, autonomous driving algorithms using low-cost vehicle-mounted cameras have attracted increasing endeavors from both academia and industry. There are multiple fronts to these endeavors, including object detection on roads, 3-D reconstruction etc., but in this work we focus on a vision-based model that directly maps raw input images to steering angles using deep networks. This represents a nascent research topic in computer vision. The technical contributions of this work are three-fold. First, the model is learned and evaluated on real human driving videos that are time-synchronized with other vehicle sensors. This differs from many prior models trained from synthetic data in racing games. Second, state-of-the-art models, such as PilotNet, mostly predict the wheel angles independently on each video frame, which contradicts common understanding of driving as a stateful process. Instead, our proposed model strikes a combination of spatial and temporal cues, jointly investigating instantaneous monocular camera observations and vehicle's historical states. This is in practice accomplished by inserting carefully-designed recurrent units (e.g., LSTM and Conv-LSTM) at proper network layers. Third, to facilitate the interpretability of the learned model, we utilize a visual back-propagation scheme for discovering and visualizing image regions crucially influencing the final steering prediction. Our experimental study is based on about 6 hours of human driving data provided by Udacity. Comprehensive quantitative evaluations demonstrate the effectiveness and robustness of our model, even under scenarios like drastic lighting changes and abrupt turning. The comparison with other state-of-the-art models clearly reveals its superior performance in predicting the due wheel angle for a self-driving car. We trained a convolutional neural network (CNN) to map raw pixels from a single front-facing camera directly to steering commands. This end-to-end approach proved surprisingly powerful. With minimum training data from humans the system learns to drive in traffic on local roads with or without lane markings and on highways. It also operates in areas with unclear visual guidance such as in parking lots and on unpaved roads. The system automatically learns internal representations of the necessary processing steps such as detecting useful road features with only the human steering angle as the training signal. We never explicitly trained it to detect, for example, the outline of roads. Compared to explicit decomposition of the problem, such as lane marking detection, path planning, and control, our end-to-end system optimizes all processing steps simultaneously. We argue that this will eventually lead to better performance and smaller systems. Better performance will result because the internal components self-optimize to maximize overall system performance, instead of optimizing human-selected intermediate criteria, e.g., lane detection. Such criteria understandably are selected for ease of human interpretation which doesn't automatically guarantee maximum system performance. Smaller networks are possible because the system learns to solve the problem with the minimal number of processing steps. We used an NVIDIA DevBox and Torch 7 for training and an NVIDIA DRIVE(TM) PX self-driving car computer also running Torch 7 for determining where to drive. The system operates at 30 frames per second (FPS). In order to support driver assistance systems in unconstrained environments, we propose to extend local appearance-based road classification with a spatial feature generation and classification. Therefore, a hierarchical approach consisting of multiple low level base classifiers, the novel spatial feature generation, as well as a final road terrain classification, is used. The system perceives a variety of local properties of the environment by means of base classifiers operating on patches extracted from monocular camera images, each represented in a metric confidence map. The core of the proposed approach is the computation of spatial ray features (SPRAY) from these confidence maps. With this, the road-terrain classifier can decide based on local visual properties and their spatial layout in the scene. In order to show the feasibility of the approach, the extraction and evaluation of the metric ego-lane driving corridor on an inner city stream is demonstrated. This is a challenging task because on a local appearance level, ego-lane is not distinguishable from other asphalt parts on the road. However, by incorporating the proposed SPRAY features the distinction is possible without requiring an explicit lane model. Due to the parallel structure of this bottom-up approach, the implemented system operates in real-time with approximately 25 Hz on a GPU. Next generation driver assistance systems require precise self localization. Common approaches using global navigation satellite systems (GNSSs) suffer from multipath and shadowing effects often rendering this solution insufficient. In urban environments this problem becomes even more pronounced. Herein we present a system for six degrees of freedom (DOF) ego localization using a mono camera and an inertial measurement unit (IMU). The camera image is processed to yield a rough position estimate using a previously computed landmark map. Thereafter IMU measurements are fused with the position estimate for a refined localization update. Moreover, we present the mapping pipeline required for the creation of landmark maps. Finally, we present experiments on real world data. The accuracy of the system is evaluated by computing two independent ego positions of the same trajectory from two distinct cameras and investigating these estimates for consistency. A mean localization accuracy of 10 cm is achieved on a 10 km sequence in an inner city scenario. Predicting the movement of other traffic participants is an integral part in the motion planning of most automated road vehicles. While simple predictions, e.g. based on assuming constant velocity, may suffice for deciding a driving strategy, predicting the set of all possible behaviors is required to ensure safe motion plans. In this work, we propose a novel tool for the latter problem based on reachability analysis: Set-Based Prediction Of Traffic Participants (SPOT). Our tool can predict the future occupancy of other traffic participants, including all possible maneuvers (e.g. full acceleration, full braking, and arbitrary lane changes), by considering physical constraints and assuming that the traffic participants abide by the traffic rules. However, we remove assumptions for each traffic participant individually as soon as a violation of a traffic rule is detected. Removal of assumptions automatically results in larger occupancies and thus a smaller drivable area for the ego vehicle, ensuring that the ego vehicle does not cause a collision during the time horizon of the prediction. Experimental results show that we obtain the set of future occupancies within a fraction of the prediction horizon. Our tool is available at spot.in.tum.de. In this paper, we propose a unified end-to-end trainable multi-task network that jointly handles lane and road marking detection and recognition that is guided by a vanishing point under adverse weather conditions. We tackle rainy and low illumination conditions, which have not been extensively studied until now due to clear challenges. For example, images taken under rainy days are subject to low illumination, while wet roads cause light reflection and distort the appearance of lane and road markings. At night, color distortion occurs under limited illumination. As a result, no benchmark dataset exists and only a few developed algorithms work under poor weather conditions. To address this shortcoming, we build up a lane and road marking benchmark which consists of about 20,000 images with 17 lane and road marking classes under four different scenarios: no rain, rain, heavy rain, and night. We train and evaluate several versions of the proposed multi-task network and validate the importance of each task. The resulting approach, VPGNet, can detect and classify lanes and road markings, and predict a vanishing point with a single forward pass. Experimental results show that our approach achieves high accuracy and robustness under various conditions in real-time (20 fps). The benchmark and the VPGNet model will be publicly available. In this paper, we propose a unified end-to-end trainable multi-task network that jointly handles lane and road marking detection and recognition that is guided by a vanishing point under adverse weather conditions. We tackle rainy and low illumination conditions, which have not been extensively studied until now due to clear challenges. For example, images taken under rainy days are subject to low illumination, while wet roads cause light reflection and distort the appearance of lane and road markings. At night, color distortion occurs under limited illumination. As a result, no benchmark dataset exists and only a few developed algorithms work under poor weather conditions. To address this shortcoming, we build up a lane and road marking benchmark which consists of about 20,000 images with 17 lane and road marking classes under four different scenarios: no rain, rain, heavy rain, and night. We train and evaluate several versions of the proposed multi-task network and validate the importance of each task. The resulting approach, VPGNet, can detect and classify lanes and road markings, and predict a vanishing point with a single forward pass. Experimental results show that our approach achieves high accuracy and robustness under various conditions in realtime (20 fps). The benchmark and the VPGNet model will be publicly available Predicting driver behavior in general, and the problem of predicting an impending lane change in particular have been studied in the community under different aspects and setups. Mostly the problem is approached as a classification problem in which each class represents a possible action in the next few seconds. In this work, we re-define the task as a regression problem, in which the time until the ego-vehicle touches the other lane is predicted. In particular, we use Long-Short-Term-Memory (LSTM) networks to learn to predict the time to lane change. Even though a regression-based formulation of the problem captures more information and should thus be harder to learn, we can show that it provides slightly better results than a comparable classification-based approach. Moreover, the additional precision that we get about the exact time of the impending the lane change could be used to further increase a user's acceptance of advanced driver assistance systems. We also show that it is possible to individualize the network by fine-tuning it to a particular driver's behavior. The fine-tuning results in an improvement of F1-Score while observing about 20 minutes of driving data from that driver. The promise of self-driving cars promotes several advantages, e.g. they have the ability to outperform human drivers while being safer. Here we take a deeper look into some aspects from algorithms aimed at making this promise a reality. More specifically, we analyze an end-to-end neural network to predict a car's steering actions on a highway based on images taken from a single car-mounted camera. We focus our analysis on several aspects which could have a significant impact on the performance of the system. These aspects are: the input data format, the temporal dependencies between consecutive inputs, and the origin of the data. We show that, for the task at hand, regression networks outperform their classifier counterparts. In addition, there seems to be a small difference between networks that use coloured images and ones that use grayscale images as input. For the second aspect, by feeding the network three concatenated images, we get a significant decrease of 30 in mean squared error. For the third aspect, by using simulation data we are able to train networks that have a performance comparable to networks trained on real-life datasets. We also qualitatively demonstrate that the standard metrics that are used to evaluate networks do not necessarily accurately reflect a system's driving behaviour. We show that a promising confusion matrix may result in poor driving behaviour while a very ill-looking confusion matrix may result in good driving behaviour. Lane-level positioning and map matching are some of the biggest challenges for navigation systems. Additionally, in safety applications or in those with critical performance requirements (such as satellite-based electronic fee collection), integrity becomes a key word for the navigation community. In this scenario, it is clear that a navigation system that can operate at the lane level while providing integrity parameters that are capable of monitoring the quality of the solution can bring important benefits to these applications. This paper presents a pioneering novel solution to the problem of combined positioning and map matching with integrity provision at the lane level. The system under consideration hybridizes measurements from a global navigation satellite system (GNSS) receiver, an odometer, and a gyroscope, along with the road information stored in enhanced digital maps, by means of a multiple-hypothesis particle-filter-based algorithm. A set of experiments in real environments in France and Germany shows the very good results obtained in terms of positioning, map matching, and integrity consistency, proving the feasibility of our proposal. We propose a Dynamic-Spatial-Attention (DSA) Recurrent Neural Network (RNN) for anticipating accidents in dashcam videos (Fig. 1). Our DSA-RNN learns to (1) distribute soft-attention to candidate objects dynamically to gather subtle cues and (2) model the temporal dependencies of all cues to robustly anticipate an accident. Anticipating accidents is much less addressed than anticipating events such as changing a lane, making a turn, etc., since accidents are rare to be observed and can happen in many different ways mostly in a sudden. To overcome these challenges, we (1) utilize state-of-the-art object detector [3] to detect candidate objects, and (2) incorporate full-frame and object-based appearance and motion features in our model. We also harvest a diverse dataset of 678 dashcam accident videos on the web (Fig. 3). The dataset is unique, since various accidents (e.g., a motorbike hits a car, a car hits another car, etc.) occur in all videos. We manually mark the time-location of accidents and use them as supervision to train and evaluate our method. We show that our method anticipates accidents about 2 s before they occur with 80 recall and 56.14 precision. Most importantly, it achieves the highest mean average precision (74.35 ) outperforming other baselines without attention or RNN. Lane relative vehicle navigation and control requires accurate lane-relative positioning of the vehicle. This relative position can be computed by comparing the vehicle absolute position with analytic roadway maps, which requires both high-accuracy positioning of the vehicle and high-accuracy lane maps. Carrier Phase Differential GPS (CPDGPS) aided INS or CPDGPS aided encoders is capable of estimating vehicle absolute position (relative to earth center) with centimeter level accuracy; however, to the best of the author's knowledge, the accuracy of lane level maps is currently not sufficient. In this paper, we first consider the structure of lane level maps that are compatible with standard practices of GIS road modeling. Then, various analytic lane definition are discussed. We also present a method of building lane level maps from high-accuracy positioning data along the lane center. The data is segmented according to road intersections. Shape points (vertices) as a function of arclength are located based on changes in estimated curvature. For each segment, the parameters are estimated by least-square criteria and can be refined as new datasets become available. This process is shown by an example. We present a probabilistic framework for the joint estimation of pedestrian head and body orientation from a mobile stereo vision platform. For both head and body parts, we convert the responses of a set of orientation-specific detectors into a (continuous) probability density function. The parts are localized by means of a pictorial structure approach, which balances part-based detector responses with spatial constraints. Head and body orientations are estimated jointly to account for anatomical constraints. The joint single-frame orientation estimates are integrated over time by particle filtering. The experiments involved data from a vehicle-mounted stereo vision camera in a realistic traffic setting; 65 pedestrian tracks were supplied by a state-of-the-art pedestrian tracker. We show that the proposed joint probabilistic orientation estimation framework reduces the mean absolute head and body orientation error up to 15° compared with simpler methods. This results in a mean absolute head body orientation error of about 21° 19°, which remains fairly constant up to a distance of 25 m. Our system currently runs in near real time (8–9 Hz). In summer 2013, a Mercedes S-Class drove completely autonomously for about 100 km from Mannheim to Pforzheim, Germany, using only close-to-production sensors. In this project, called Mercedes Benz Intelligent Drive, stereo vision was one of the main sensing components. For the representation of free space and obstacles we relied on the so called Stixel World, a generic 3D intermediate representation which is computed from dense disparity images. In spite of the high performance of the Stixel World in most common traffic scenes, the availability of this technique is limited. For instance under adverse weather, rain or even spray water on the windshield results in erroneous disparity images which generate false Stixel results. This can lead to undesired behavior of autonomous vehicles. Our goal is to use the Stixel World for a robust free space estimation and a reliable obstacle detection even during difficult weather conditions. In this paper, we meet this challenge and fuse the Stixels incrementally into a reference grid map. Our new approach is formulated in a Bayesian manner and is based on existence estimation methods. We evaluate our new technique on a manually labeled database with emphasis on bad weather scenarios. The number of structures which are detected mistakenly within free space areas is reduced by a factor of two whereas the detection rate of obstacles increases at the same time. This paper describes a system for detecting and estimating the properties of multiple travel lanes in an urban road network from calibrated video imagery and laser range data acquired by a moving vehicle. The system operates in real-time in several stages on multiple processors, fusing detected road markings, obstacles, and curbs into a stable non-parametric estimate of nearby travel lanes. The system incorporates elements of a provided piecewise-linear road network as a weak prior. Our method is notable in several respects: it detects and estimates multiple travel lanes; it fuses asynchronous, heterogeneous sensor streams; it handles high-curvature roads; and it makes no assumption about the position or orientation of the vehicle with respect to the road. We analyze the system's performance in the context of the 2007 DARPA Urban Challenge. With five cameras and thirteen lidars, our method was incorporated into a closed-loop controller to successfully guide an autonomous vehicle through a 90 km urban course at speeds up to 40 km h amidst moving traffic. Steering a car through traffic is a complex task that is difficult to cast into algorithms. Therefore, researchers turn to training artificial neural networks from front-facing camera data stream along with the associated steering angles. Nevertheless, most existing solutions consider only the visual camera frames as input, thus ignoring the temporal relationship between frames. In this work, we propose a Convolutional Long Short-Term Memory Recurrent Neural Network (C-LSTM), that is end-to-end trainable, to learn both visual and dynamic temporal dependencies of driving. Additionally, We introduce posing the steering angle regression problem as classification while imposing a spatial relationship between the output layer neurons. Such method is based on learning a sinusoidal function that encodes steering angles. To train and validate our proposed methods, we used the publicly available Comma.ai dataset. Our solution improved steering root mean square error by 35 over recent methods, and led to a more stable steering by 87 . Spatio-temporal and geo-referenced datasets are growing rapidly, with the rapid development of some technology, such as GPS, satellite systems. At present, many scholars are very interested in the clustering of the trajectory. Existing trajectory clustering algorithms group similar trajectories as a whole and can't distinguish the direction of trajectory. Our key finding is that clustering trajectories as a whole could miss common sub-trajectories and trajectory has direction information. In many applications, discovering common sub-trajectories is very useful. In this paper, we present a trajectory clustering algorithm CTHD (clustering of trajectory based on hausdorff distance). In the CTHD, the trajectory is firstly described by a sequence of flow vectors and partitioned into a set of sub-trajectory. Next the similarity between trajectories is measured by their respective Hausdorff distances. Finally, the trajectories are clustered by the DBSCAN clustering algorithm. The proposed algorithm is different from other schemes using Hausdorff distance that the flow vectors include the position and direction. So it can distinguish the trajectories in different directions. The experimental result shows the phenomenon. The problem of maneuvering a vehicle through a race course in minimum time requires computation of both longitudinal (brake and throttle) and lateral (steering wheel) control inputs. Unfortunately, solving the resulting nonlinear optimal control problem is typically computationally expensive and infeasible for real-time trajectory planning. This paper presents an iterative algorithm that divides the path generation task into two sequential subproblems that are significantly easier to solve. Given an initial path through the race track, the algorithm runs a forward-backward integration scheme to determine the minimum-time longitudinal speed profile, subject to tire friction constraints. With this fixed speed profile, the algorithm updates the vehicle's path by solving a convex optimization problem that minimizes the resulting path curvature while staying within track boundaries and obeying affine, time-varying vehicle dynamics constraints. This two-step process is repeated iteratively until the predicted lap time no longer improves. While providing no guarantees of convergence or a globally optimal solution, the approach performs very well when validated on the Thunderhill Raceway course in Willows, CA. The predicted lap time converges after four to five iterations, with each iteration over the full 4.5 km race course requiring only thirty seconds of computation time on a laptop computer. The resulting trajectory is experimentally driven at the race circuit with an autonomous Audi TTS test vehicle, and the resulting lap time and racing line is comparable to both a nonlinear gradient descent solution and a trajectory recorded from a professional racecar driver. The experimental results indicate that the proposed method is a viable option for online trajectory planning in the near future.