Reinforcement learning methods that continuously learn neural networks by episode generation with game tree search have been successful in two-person complete information deterministic games such as chess, shogi, and Go. However, there are only reports of practical cases and there are little evidence to guarantee the stability and the final performance of learning process. In this research, the coordination of episode generation was focused on. By means of regarding the entire system as game tree search, the new method can handle the trade-off between exploitation and exploration during episode generation. The experiments with a small problem showed that it had robust performance compared to the existing method, Alpha Zero. The result that computer programs beat professional human players on chess, shogi and Go was a huge achievement in computer science. In particular, the development of highly general methods totally changed our perspective about two-person complete information deterministic games. Then, has this field already finished ? My answer is no. To deal with many games, more robust methods are required to free humans from hyperparameter tuning. Moreover, the challenge to the god of games won't be finished and we want algorithms that can achieve better final performance. This study attempts to bring suggestions for recent achievements in two-player complete information deterministic games from classical game tree search context. More specifically, this is a new approach in which the reinforcement learning system using game tree search itself is handled as game tree search. In this study, we examined a very simple task, Tic-tac-toe. First of all, it was shown that obtaining the optimal strategy is sometimes difficult depending on the parameters. The results suggest that reinforcement learning methods like Alpha Zero often suffer from naiveness against exploration. In the proposed method, it is possible to vary the beginning of the game in the episode generation by the master game tree. The results suggest that the proposed method has ability to control the search for the beginning of the game by adding proper noise. On the other hand, when PUCT using the strategy as it is applied to the master game tree (MbMNoNoise), the performance was lower than the baseline. The reason of this result is that the policy has converged earlier than the effect of exploration in the master game tree. Due to this point, it was not effective. In this report, PUCT is applied to the master game tree as same as ordinal game tree. However, it is necessary to examine a mechanism that makes it more exploratory. Lastly, in this study, we verified only one of the simplest games, Tic-tac-toe. From the experimental results in this paper, it is expected that the proposed method can produce robust results with respect to temperature parameters even for larger games. It will also be necessary to verify whether the speed of improvement in real time is better that previous methods. I hope that the combination of tree search and reinforcement learning will be used for a wider range of domains if there exists the method in which both stableness and speed are better performance.
Apply Monte Carlo Tree Search to episode generation in Alpha Zero
{"src": "scitldr-AIC_train", "paper_id": "HkgU3xBtDS"}