The ability to design biological structures such as DNA or proteins would have considerable medical and industrial impact. Doing so presents a challenging black-box optimization problem characterized by the large-batch, low round setting due to the need for labor-intensive wet lab evaluations. In response, we propose using reinforcement learning (RL) based on proximal-policy optimization (PPO) for biological sequence design. RL provides a flexible framework for optimization generative sequence models to achieve specific criteria, such as diversity among the high-quality sequences discovered. We propose a model-based variant of PPO, DyNA-PPO, to improve sample efficiency, where the policy for a new round is trained offline using a simulator fit on functional measurements from prior rounds. To accommodate the growing number of observations across rounds, the simulator model is automatically selected at each round from a pool of diverse models of varying capacity.   On the tasks of designing DNA transcription factor binding sites, designing antimicrobial proteins, and optimizing the energy of Ising models based on protein structure, we find that DyNA-PPO performs significantly better than existing methods in settings in which modeling is feasible, while still not performing worse in situations in which a reliable model cannot be learned. Driven by real-world obstacles in health and disease requiring new drugs, treatments, and assays, the goal of biological sequence design is to identify new discrete sequences x which optimize some oracle, typically an experimentally-measured functional property f (x). This is a difficult black-box optimization problem over a combinatorially large search space in which function evaluation relies on slow and expensive wet-lab experiments. The setting induces unusual constraints in black-box optimization and reinforcement learning: large synchronous batches with few rounds total. The current gold standard for biomolecular design is directed evolution, which was recently recognized with a Nobel prize (Arnold, 1998) and is a form of randomized local search. Despite its impact, directed evolution is sample inefficient and relies on greedy hillclimbing to the optimal sequences. Recent work has demonstrated that machine-learning-guided optimization (Section 3) can find better sequences faster. Reinforcement learning (RL) provides a flexible framework for black-box optimization that can harness modern deep generative sequence models. This paper proposes a simple method for improving the sample efficiency of policy gradient methods such as PPO (Schulman et al., 2017) for black-box optimization by using surrogate models that are trained online to approximate f (x). Our method updates the policy's parameters using sequences x generated by the current policy π θ (x), but evaluated using a learned surrogate f w (x), instead of the true, but unknown, oracle reward function f (x). We learn the parameters of the reward model, w, simultaneously with the parameters of the policy. This is similar to other model-based RL methods, but simpler, since in the context of sequence optimization, the state-transition model is deterministic and known. Initially the learned reward model, f w (x), is unreliable, so we rely entirely on f (x) to assess sequences and update the policy. This allows a graceful fallback to PPO when the model is not effective. Over time, the reward model becomes more reliable and can be used as a cheap surrogate, similar to Bayesian optimization methods (Shahriari et al., 2015) . We show empirically that cross-validation is an effective heuristic for assessing the model quality, which is simpler than the inference required by Bayesian optimization. We rigorously evaluate our method on three in-silico sequence design tasks that draw on experimental data to construct functions f (x) characteristic of real-world design problems: optimizing binding affinity of DNA sequences of length 8 (search space size 4 8 ); optimizing anti-microbial peptide sequences (search space size 20 50 ), and optimizing binary sequences where f (x) is defined by the energy of an Ising model for protein structure (search space size 20 50 ). These do not rely on wet lab experiments, and thus allow for large-scale benchmarking across a range of methods. We show that our DyNA-PPO method achieves higher cumulative reward for a given budget (measured in terms of number of calls to f (x)) than existing methods, such as standard PPO, various forms of the cross-entropy method, Bayesian optimization, and evolutionary search. In summary, our contributions are as follows: • We provide a model-based RL algorthm, DyNA-PPO, and demonstrate its effectiveness in performing sample efficient batched black-box function optimization. • We address model bias by quantifying the reliability and automatically selecting models of appropriate complexity via cross validation. • We propose a visitation-based exploration bonus and show that it is more effective than entropy-regularization in identifying multiple local optima. • We present a new optimization task for benchmarking methods for biological sequence design based on protein energy Ising models. We have shown that RL is an attractive alternative to existing methods for designing DNA and protein sequences. We have proposed DyNA-PPO, a model-based extension of PPO (Schulman et al., 2017) with automatic model selection that improves sample efficiency, and incorporates a reward function that promotes exploration by penalizing identical sequences. By approximating an expensive wet-lab experiment with a surrogate model, we can perform many rounds of optimization in simulation. While this work has been focused on showing the benefit of DyNA-PPO for biological sequence design, we believe that the large-batch, low-round optimization setting described here may well be of general interest, and that model-based RL may be applicable in other domains such as agriculture, education, and economics. A IMPLEMENTATION DETAILS
We augment model-free policy learning with a sequence-level surrogate reward functions and count-based visitation bonus and demonstrate effectiveness in the large batch, low-round regime seen in designing DNA and protein sequences.
{"src": "scitldr-AIC_train", "paper_id": "HklxbgBKvr"}