Autonomous vehicles are becoming more common in city transportation.   Companies will begin to find a need to teach these vehicles smart city fleet coordination.   Currently, simulation based modeling along with hand coded rules dictate the decision making of these autonomous vehicles. We believe that complex intelligent behavior can be learned by these agents through Reinforcement Learning. In this paper, we discuss our work for solving this system by adapting the Deep Q-Learning (DQN) model to the multi-agent setting.   Our approach applies deep reinforcement learning by combining convolutional neural networks with DQN to teach agents to fulfill customer demand in an environment that is partially observ-able to them. We also demonstrate how to utilize transfer learning to teach agents to balance multiple objectives such as navigating to a charging station when its en-ergy level is low. The two evaluations presented show that our solution has shown  hat we are successfully able to teach agents cooperation policies while balancing multiple objectives. Many business problems that exist in todays environment consist of multiple decisions makers either collaborating or competing towards a particular goal. In this work, the challenge is applying multi-agent systems for autonomous fleet control. As Autonomous Vehicles (AVs) are becoming more prevalent, companies controlling these fleets such as Uber/Lyft will need to teach these agents to make optimal decisions. The goal of this work is to train these agents/cars optimal relocation strategies that will maximize the efficiency of the fleet while satisfying customer trip demand. Traditional solutions will use discrete event simulation modeling to optimize over a chosen objective function. This approach requires various hand coded rules as well as assumptions to help the model converge on a solution. This becomes an extremely difficult problem when there are many outside environment dynamics that can influence an agents/cars decision making (E.g. Charging, Parking). Furthermore, a solution to a particular environment may become outdated with new incoming information (E.g. New Demand Distribution).An algorithm that can adapt and learn decision making organically is needed for these types of problems and recent works in Reinforcement Learning and particularly Deep Reinforcement Learning has shown to be effective in this space. Deep Minds recent success with Deep Q Learning (DQN) was proven to be very successful in learning human level performance for many Atari 2600 games which was difficult before this because of its highly dimension unstructured data. In this work, we will pull from prior work in Multi-Agent Deep Reinforcement Learning (MA-DRL) and extend this to our multi-agent system of cars and fleet coordination. We will represent the city environment that holds the cars and customers as an image-like state representation where each layer holds specific information about the environment. We then will introduce our work with applying this to a partially observable environment where agents can only see a certain distance from them and show how this helps with scaling up. Along with that, we will show how we took advantage of Transfer Learning to teach agents multiple objects in particular charging an import aspect of AVs. Our results show that we are successfully able to teach coordination strategies with other cars so that they can optimize the utility of each car. Finally, we are also able to teach agents the second object of keeping itself alive while not losing the previous objective of picking up customers. Deep Reinforcement Learning provides a great approach to teach agents how to solve complex problems that us as humans may never be able to solve. For instance, Deep Mind has been successful in teach an agent to defeat the world champion in Go. More specifically, multi-Agent Reinforcement Learning problems provide an interesting avenue to investigate agent to agent communication and decision protocols. Since agents must rationalize about the intentions of other agents the dimensionality of the problem space becomes difficult to solve. In our use case, we wanted to see if we can scale a DRL solution up to an actual ride sharing environment that maintains the same dynamics as it would in real life. For this to be possible, we were tasked with the problem of teaching these agents effective cooperation strategies that would optimize the reward of the system along with the problem of teaching these same agents multiple objectives. This work, demonstrated how we successfully applied a partially observable multi-agent deep reinforcement solution to this ride sharing problem. Along with that, we showed how we can effectively take advantage of transfer learning to adapt decision policies to account for multiple objectives.
Utilized Deep Reinforcement Learning to teach agents ride-sharing fleet style coordination.
{"src": "scitldr-AIC_train", "paper_id": "B1EGg7ZCb"}