Parallel bandit architecture based on laser chaos for reinforcement learning
Accelerating artificial intelligence by photonics is an active field of study
aiming to exploit the unique properties of photons. Reinforcement learning is
an important branch of machine learning, and photonic decision-making
principles have been demonstrated with respect to the multi-armed bandit
problems. However, reinforcement learning could involve a massive number of
states, unlike previously demonstrated bandit problems where the number of
states is only one. Q-learning is a well-known approach in reinforcement
learning that can deal with many states. The architecture of Q-learning,
however, does not fit well photonic implementations due to its separation of
update rule and the action selection. In this study, we organize a new
architecture for multi-state reinforcement learning as a parallel array of
bandit problems in order to benefit from photonic decision-makers, which we
call parallel bandit architecture for reinforcement learning or PBRL in short.
Taking a cart-pole balancing problem as an instance, we demonstrate that PBRL
adapts to the environment in fewer time steps than Q-learning. Furthermore,
PBRL yields faster adaptation when operated with a chaotic laser time series
than the case with uniformly distributed pseudorandom numbers where the
autocorrelation inherent in the laser chaos provides a positive effect. We also
find that the variety of states that the system undergoes during the learning
phase exhibits completely different properties between PBRL and Q-learning. The
insights obtained through the present study are also beneficial for existing
computing platforms, not just photonic realizations, in accelerating
performances by the PBRL algorithms and correlated random sequences.