The reproducibility of reinforcement-learning research has been highlighted as a key challenge area in the field. In this paper, we present a case study in reproducing the results of one groundbreaking algorithm, AlphaZero, a reinforcement learning system that learns how to play Go at a superhuman level given only the rules of the game. We describe Minigo, a reproduction of the AlphaZero system using publicly available Google Cloud Platform infrastructure and Google Cloud TPUs. The Minigo system includes both the central reinforcement learning loop as well as auxiliary monitoring and evaluation infrastructure. With ten days of training from scratch on 800 Cloud TPUs, Minigo can play evenly against LeelaZero and ELF OpenGo, two of the strongest publicly available Go AIs. We discuss the difficulties of scaling a reinforcement learning system and the monitoring systems required to understand the complex interplay of hyperparameter configurations. In March 2016, Google DeepMind's AlphaGo BID0 defeated world champion Lee Sedol by using two deep neural networks (a policy and a value network) and Monte Carlo Tree Search (MCTS) to synthesize the output of these two neural networks. The policy network was trained via supervised learning from human games, and the value network was trained from a much larger corpus of synthetic games generated by sampling game trajectories from the policy network. AlphaGo Zero BID1 , published in October 2017, described a continuous pipeline, which when initialized with random weights, could train itself to defeat the original AlphaGo system. The requirement for expert human data was replaced with a requirement for vast amounts of compute: approximately two thousand TPUs were used for 72 hours to train AlphaGo Zero to its full strength. AlphaZero BID2 presents a refinement of the AlphaGoZero pipeline, notably removing the gating mechanism for publishing new models.In many ways, AlphaGo Zero can be seen as the logical culmination of fully automating and streamlining the bootstrapping process: the original AlphaGo system was bootstrapped from expert human data and reached a final strength that was somewhat stronger than the best humans. Then, by generating new training data with the stronger AlphaGo system and repeating the bootstrap process, an even stronger system was created. By automating the bootstrapping process until it is continuous, a system is created that can train itself to surpass human levels of play, even when starting from random play.In this paper, we discuss our experiences creating Minigo. About half of our effort went into rebuilding the infrastructure necessary to coordinate a thousand selfplay workers. The other half of the effort went into monitoring infrastructure to test and verify that what we had built was bug-free. Despite having at hand a paper describing the final architecture of AlphaZero, we rediscovered the hard way which components of the system were absolutely necessary to get right, and which components we could be messy with. It stands to reason that without the benefit of pre-existing work, monitoring systems are even more important in the discovery process. We discuss in particular,
We reproduced AlphaZero on Google Cloud Platform
{"src": "scitldr-AIC_train", "paper_id": "H1eerhIpLV"}