Event detection is gaining increasing attention in smart cities research. Large-scale mobility data serves as an important tool to uncover the dynamics of urban transportation systems, and more often than not the dataset is incomplete. In this article, we develop a method to detect extreme events in large traffic datasets, and to impute missing data during regular conditions. Specifically, we propose a robust tensor recovery problem to recover low rank tensors under fiber-sparse corruptions with partial observations, and use it to identify events, and impute missing data under typical conditions. Our approach is scalable to large urban areas, taking full advantage of the spatio-temporal correlations in traffic patterns. We develop an efficient algorithm to solve the tensor recovery problem based on the alternating direction method of multipliers (ADMM) framework. Compared with existing @math norm regularized tensor decomposition methods, our algorithm can exactly recover the values of uncorrupted fibers of a low rank tensor and find the positions of corrupted fibers under mild conditions. Numerical experiments illustrate that our algorithm can exactly detect outliers even with missing data rates as high as 40 , conditioned on the outlier corruption rate and the Tucker rank of the low rank tensor. Finally, we apply our method on a real traffic dataset corresponding to downtown Nashville, TN, USA and successfully detect the events like severe car crashes, construction lane closures, and other large events that cause significant traffic disruptions.
The outliers we are interested in this work are due to outliers caused by extreme events. Another related problem considers methods to detect outliers caused by data measurement errors, such as sensor malfunction, malicious tampering, or measurement error @cite_13 @cite_2 @cite_38 . The latter methods can be seen as a part of a standard data cleaning or data pre-processing step. On the other hand, outliers caused by extreme traffic have valuable information for congestion management, and can provide agencies with insights into the performance of urban network. The works @cite_1 @cite_35 @cite_21 explore the problem of outlier detection caused by events, while the works @cite_6 @cite_3 @cite_50 @cite_4 focus on determining the root causes of the outlier.
@cite_38 @cite_35 @cite_4 @cite_21 @cite_1 @cite_6 @cite_3 @cite_50 @cite_2 @cite_13
1974476933 2093855404 2042850276 2117618130 2743812350 2014211872 2963024417 2021002141 2088400370 1544613517
Novel methods for implementation of detector-level multivariate screening methods are presented. The methods use present data and classify data as outliers on the basis of comparisons with empirical cutoff points derived from extensive archived data rather than from standard statistical tables. In addition, while many of the ideas of the classical Hotelling's T2-statistic are used, modern statistical trend removal and blocking are incorporated. The methods are applied to intelligent transportation system data from San Antonio and Austin, Texas. These examples show how the suggested new methods perform with high-quality traffic data and apparently lower-quality traffic data. All algorithms were implemented by using the SAS programming language. The increasing availability of large-scale trajectory data provides us great opportunity to explore them for knowledge discovery in transportation systems using advanced data mining techniques. Nowadays, large number of taxicabs in major metropolitan cities are equipped with a GPS device. Since taxis are on the road nearly 24h a day (with drivers changing shifts), they can now act as reliable sensors to monitor the behavior of traffic. In this article, we use GPS data from taxis to monitor the emergence of unexpected behavior in the Beijing metropolitan area, which has the potential to estimate and improve traffic conditions in advance. We adapt likelihood ratio test statistic (LRT) which have previously been mostly used in epidemiological studies to describe traffic patterns. To the best of our knowledge the use of LRT in traffic domain is not only novel but results in accurate and rapid detection of anomalous behavior. The recent availability of datasets on transportation networks with higher spatial and temporal resolution is enabling new research activities in the fields of Territorial Intelligence and Smart Cities. Among these, many research efforts are aimed at predicting traffic congestions to alleviate their negative effects on society, mainly by learning recurring mobility patterns. Within this field, in this paper we propose an integrated solution to predict and visualize non-recurring traffic congestion in urban environments caused by Planned Special Events (PSE), such as a soccer game or a concert. Predictions are done by means of two Machine Learning-based techniques. These have been proven to successfully outperform current state of the art predictions by 35 in an empirical assessment we conducted over a time frame of 7 months within the inner city of Cologne, Germany. The predicted congestions are fed into a specifically conceived visualization tool we designed to allow Decision Makers to evaluate the situation and take actions to improve mobility. HighlightsWe analyze in detail the impact of Planned Special Events (PSEs) on traffic.We propose two novel methods to predict upcoming congestions caused by PSEs.Results show that our prediction methods outperform the state of the art by 35 .We introduce a specifically designed tool for visualizing the prediction results.Visualization tool allows experts to evaluate upcoming situations ahead of time. The detection of outliers in spatio-temporal traffic data is an important research problem in the data mining and knowledge discovery community. However to the best of our knowledge, the discovery of relationships, especially causal interactions, among detected traffic outliers has not been investigated before. In this paper we propose algorithms which construct outlier causality trees based on temporal and spatial properties of detected outliers. Frequent substructures of these causality trees reveal not only recurring interactions among spatio-temporal outliers, but potential flaws in the design of existing traffic networks. The effectiveness and strength of our algorithms are validated by experiments on a very large volume of real taxi trajectories in an urban road network. As the development of crowdsourcing technique, acquiring amounts of data in urban cities becomes possible and reliable, which makes it possible to mine useful and significant information from data. Traffic anomaly detection is to find the traffic patterns which are not expected and it can be used to explore traffic problems accurately and efficiently. In this paper, we propose LoTAD to explore anomalous regions with long-term poor traffic situations. Specifically, we process crowdsourced bus data into TS-segments (Temporal and Spatial segments) to model the traffic condition. Later, we explore anomalous TS-segments in each bus line by calculating their AI (Anomaly Index). Then, we combine anomalous TS-segments detected in different lines to mine anomalous regions. The information of anomalous regions provides suggestions for future traffic planning. We conduct experiments with real crowdsourced bus trajectory datasets of October in 2014 and March in 2015 in Hangzhou. We analyze the varieties of the results and explain how they are consistent with the real urban traffic planning or social events happened between the time interval of the two datasets. At last we do a contrast experiment with the most ten congested roads in Hangzhou, which verifies the effectiveness of LoTAD. Anomaly detection (a.k.a., outlier or burst detection) is a well-motivated problem and a major data mining and knowledge discovery task. In this article, we study the problem of population anomaly detection, one of the key issues related to event monitoring and population management within a city. Through studying detected population anomalies, we can trace and analyze these anomalies, which could help to model city traffic design and event impact analysis and prediction. Although a significant and interesting issue, it is very hard to detect population anomalies and retrieve anomaly trajectories, especially given that it is difficult to get actual and sufficient population data. To address the difficulties of a lack of real population data, we take advantage of mobile phone networks, which offer enormous spatial and temporal communication data on persons. More importantly, we claim that we can utilize these mobile phone data to infer and approximate population data. Thus, we can study the population anomaly detection problem by taking advantages of unique features hidden in mobile phone data. In this article, we present a system to conduct Population Anomaly Detection (PAD). First, we propose an effective clustering method, correlation-based clustering, to cluster the incomplete location information from mobile phone data (i.e., from mobile call volume distribution to population density distribution). Then, we design an adaptive parameter-free detection method, R-scan, to capture the distributed dynamic anomalies. Finally, we devise an efficient algorithm, BT-miner, to retrieve anomaly trajectories. The experimental results from real-life mobile phone data confirm the effectiveness and efficiency of the proposed algorithms. Finally, the proposed methods are realized as a pilot system in a city in China. Non-recurring traffic congestion is caused by temporary disruptions, such as accidents, sports games, adverse weather, etc. We use data related to real-time traffic speed, jam factors (a traffic congestion indicator), and events collected over a year from Nashville, TN to train a multi-layered deep neural network. The traffic dataset contains over 900 million data records. The network is thereafter used to classify the real-time data and identify anomalous operations. Compared with traditional approaches of using statistical or machine learning techniques, our model reaches an accuracy of 98.73 percent when identifying traffic congestion caused by football games. Our approach first encodes the traffic across a region as a scaled image. After that the image data from different timestamps is fused with event- and time-related data. Then a crossover operator is used as a data augmentation method to generate training datasets with more balanced classes. Finally, we use the receiver operating characteristic (ROC) analysis to tune the sensitivity of the classifier. We present the analysis of the training time and the inference time separately. With the increasing amount of traffic information collected through floating car data, it is highly desirable to find meaningful traffic patterns such as congestion patterns from the accumulated massive historical dataset. It is however challenging due to the huge size of the dataset and the complexity and dynamics of traffic phenomena. A novel floating car data analysis method based on data cube for congestion pattern exploration is proposed in this paper. This method is different from traditional methods that depend only on numerical statistics of traffic data. The view of the event or spatial-temporal progress is adapted to model and measure traffic congestions. According to a multi-dimensional analysis framework, the traffic congestion event is first identified based on spatial-temporal related relationship of slow-speed road segment. Then, it is aggregated by a cluster style to get the traffic pattern on a different level of detail of spatial-temporal dimension. Aggregated location, time period and duration time for recurrent and important congestions are used to represent the congestion pattern. The authors evaluate methods using a historical traffic dataset collected from about 12000 taxi-based floating cars for one week in a large urban area. Results show that the method can effectively identify and summarize the congestion pattern with efficient computation and reduced storage cost. In order to improve the veracity and reliability of a traffic model built, or to extract important and valuable information from collected traffic data, the technique of outlier mining has been introduced into the traffic engineering domain for detecting and analyzing the outliers in traffic data sets. Three typical outlier algorithms, respectively the statistics-based approach, the distance-based approach, and the density-based local outlier approach, are described with respect to the principle, the characteristics and the time complexity of the algorithms. A comparison among the three algorithms is made through application to intelligent transportation systems (ITS). Two traffic data sets with different dimensions have been used in our experiments carried out, one is travel time data, and the other is traffic flow data. We conducted a number of experiments to recognize outliers hidden in the data sets before building the travel time prediction model and the traffic flow foundation diagram. In addition, some artificial generated outliers are introduced into the traffic flow data to see how well the different algorithms detect them. Three strategies-based on ensemble learning, partition and average LOF have been proposed to develop a better outlier recognizer. The experimental results reveal that these methods of outlier mining are feasible and valid to detect outliers in traffic data sets, and have a good potential for use in the domain of traffic engineering. The comparison and analysis presented in this paper are expected to provide some insights to practitioners who plan to use outlier mining for ITS data. In their goal to effectively manage the use of existing infrastructures, intelligent transportation systems require precise forecasting of near-term traffic volumes to feed real-time analytical models and traffic surveillance tools that alert of network links reaching their capacity. This article proposes a new methodological approach for short-term predictions of time series of volume data at isolated cross sections. The originality in the computational modeling stems from the fit of threshold values used in the stationary wavelet-based denoising process applied on the time series, and from the determination of patterns that characterize the evolution of its samples over a fixed prediction horizon. A self-organizing fuzzy neural network is optimized in its configuration parameters for learning and recognition of these patterns. Four real-world data sets from 3 interstate roads are considered for evaluating the performance of the proposed model. A quantitative comparison made with the results obtained by 4 other relevant prediction models shows a favorable outcome.