Spatial Big Data and Moving Objects: A Comprehensive Survey <s> D. ADP (ADAPTIVE DISTRIBUTED PROGRAMMING)-BASED LEARNING CONTROL <s> Transit agencies often implement operations control strategies so as to mitigate the effects of streetcar bunching along transit routes. Typical control strategies include vehicle holding, expressing, and short turning, which are usually implemented through manual means via field supervisors or central control centers. The objective of this study is to automate streetcar bunching control by means of multiple Reinforcement Learning (RL) agents that act on a series of successive signalized intersections. The multiple RL agents developed in this study include the “bunch–splitting,” “holding,” and “expressing” agents which work cooperatively to break up a streetcar bunch if one is detected and to build a reasonable headway between the paired streetcars. Various elements of the RL agents such as the action set, the reward, and the state space are set up under a traffic signal control context. Simulation results indicated that these multiple RL agents could split up a streetcar bunch and prevent it from forming... <s> BIB001 </s> Spatial Big Data and Moving Objects: A Comprehensive Survey <s> D. ADP (ADAPTIVE DISTRIBUTED PROGRAMMING)-BASED LEARNING CONTROL <s> The high growth rate of vehicles per capita now poses a real challenge to efficient urban traffic control (UTC). An efficient solution to UTC must be adaptive in order to deal with the highly-dynamic nature of urban traffic. In the near future, global positioning systems and vehicle-to-vehicle/infrastructure communication may provide a more detailed local view of the traffic situation that could be employed for better global UTC optimization. In this paper we describe the design of a next-generation UTC system that exploits such local knowledge about a junction's traffic in order to optimize traffic control. Global UTC optimization is achieved using a local adaptive round robin (ARR) phase switching model optimized using collaborative reinforcement learning (CRL). The design employs an ARR-CRL-based agent controller for each signalized junction that collaborates with neighbouring agents in order to learn appropriate phase timing based on the traffic pattern. We compare our approach to non-adaptive fixed-time UTC system and to a saturation balancing algorithm in a large-scale simulation of traffic in Dublin's inner city centre. We show that the ARR-CRL approach can provide significant improvement resulting in up to ~57% lower average waiting time per vehicle compared to the saturation balancing algorithm. <s> BIB002 </s> Spatial Big Data and Moving Objects: A Comprehensive Survey <s> D. ADP (ADAPTIVE DISTRIBUTED PROGRAMMING)-BASED LEARNING CONTROL <s> In this article, we introduce some recent research trends within the field of adaptive/approximate dynamic programming (ADP), including the variations on the structure of ADP schemes, the development of ADP algorithms and applications of ADP schemes. For ADP algorithms, the point of focus is that iterative algorithms of ADP can be sorted into two classes: one class is the iterative algorithm with initial stable policy; the other is the one without the requirement of initial stable policy. It is generally believed that the latter one has less computation at the cost of missing the guarantee of system stability during iteration process. In addition, many recent papers have provided convergence analysis associated with the algorithms developed. Furthermore, we point out some topics for future studies. <s> BIB003
Although, learning based techniques seem effective and the results are also quite promising, yet realizing the performance in a highly dynamic and uncertain environment is quite difficult. Several mathematical approaches are proposed to handle such problems BIB003 . In order to solve complex decision processes having huge and continuous states, Wang et al. BIB003 propose methods that use reinforcement learning (RL) with dynamic programming. The dynamic programming algorithms are classified into two classes. First class is the one that does not need any initial stable policy while the other class of algorithms requires an initial stable policy. Ling and Shalaby BIB001 study various elements of RL agents such as reward, action set, and state space in the context of traffic flow. One of the most promising features of RL is the ability of an agent to understand the relationship between controlled actions and their consequences on the environment. Abdulhai et al.  utilize this feature and propose a Q-learning method for network traffic control. Salkham et al. BIB002 utilize the information provided by GPS and V2V (vehicle to vehicle) communication for traffic optimization. They use a local round robin switching model based on RL. A lot of work still needs to be done but it is expected that dynamic programming variants, methods, and algorithms will provide the basic functionality for realizing an optimized and dynamic system for moving objects.