DRL was implemented on a cylinder flow by [1]}, using a Q-learning algorithm to learn how to stabilise the cylinder wake efficiently. Q-learning is one of the two main families of DRL methods, which consists in implicitly encoding the policy in a Q-network that provides the value of every state-action pair, the policy being the action that maximises this value for each given state. [2]} used Deep Deterministic Policy Gradient (DDPG) to steer the Kuramoto-Sivashinksy equation to its fixed points, and [3]} used Deep Q-Networks (DQN) to optimise the frequency of plasma actuator bursts to control airfoil flow separation. These methods make use of a self-consistency equation for their optimisation and thus display a well-known tendency to be unstable. Most of policy optimisation algorithms, the other main family of DRL training methods, derive from Proximal Policy Optimisation (PPO) introduced by [4]}. This algorithm directly implements the idea of promoting successful control action and avoiding detrimental ones. PPO was used by [5]} to control a low Reynolds number cylinder wake using surface-mounted jets. [6]} also resorted to PPO to control a low-Reynolds confined bi-dimensional airfoil flow using three suction-side synthetic jets to reduce drag. In this paper, a variant named Proximal Policy Optimisation with Covariance Matrix Adaptation (PPO-CMA) [7]} is used as the base learning algorithm.
