Recent advances in deep reinforcement learning have made significant strides in performance on applications such as Go and Atari games. However, developing practical methods to balance exploration and exploitation in complex domains remains largely unsolved. Thompson Sampling and its extension to reinforcement learning provide an elegant approach to exploration that only requires access to posterior samples of the model. At the same time, advances in approximate Bayesian methods have made posterior approximation for flexible neural network models practical. Thus, it is attractive to consider approximate Bayesian neural networks in a Thompson Sampling framework. To understand the impact of using an approximate posterior on Thompson Sampling, we benchmark well-established and recently developed methods for approximate posterior sampling combined with Thompson Sampling over a series of contextual bandit problems. We found that many approaches that have been successful in the supervised learning setting underperformed in the sequential decision-making scenario. In particular, we highlight the challenge of adapting slowly converging uncertainty estimates to the online setting. Recent advances in reinforcement learning have sparked renewed interest in sequential decision making with deep neural networks. Neural networks have proven to be powerful and flexible function approximators, allowing one to learn mappings directly from complex states (e.g., pixels) to estimates of expected return. While such models can be accurate on data they have been trained on, quantifying model uncertainty on new data remains challenging. However, having an understanding of what is not yet known or well understood is critical to some central tasks of machine intelligence, such as effective exploration for decision making.A fundamental aspect of sequential decision making is the exploration-exploitation dilemma: in order to maximize cumulative reward, agents need to trade-off what is expected to be best at the moment, (i.e., exploitation), with potentially sub-optimal exploratory actions. Solving this trade-off in an efficient manner to maximize cumulative reward is a significant challenge as it requires uncertainty estimates. Furthermore, exploratory actions should be coordinated throughout the entire decision making process, known as deep exploration, rather than performed independently at each state.Thompson Sampling (Thompson, 1933) and its extension to reinforcement learning, known as Posterior Sampling, provide an elegant approach that tackles the exploration-exploitation dilemma by maintaining a posterior over models and choosing actions in proportion to the probability that they are optimal. Unfortunately, maintaining such a posterior is intractable for all but the simplest models. As such, significant effort has been dedicated to approximate Bayesian methods for deep neural networks. These range from variational methods BID17 BID6 BID23 to stochastic minibatch Markov Chain Monte Carlo (Neal, 1994; Welling & Teh, 2011; BID25 BID1 BID26 , among others. Because the exact posterior is intractable, evaluating these approaches is hard. Furthermore, these methods are rarely compared on benchmarks that measure the quality of their estimates of uncertainty for downstream tasks.To address this challenge, we develop a benchmark for exploration methods using deep neural networks. We compare a variety of well-established and recent Bayesian approximations under the lens of Thompson Sampling for contextual bandits, a classical task in sequential decision making. All code and implementations to reproduce the experiments will be available open-source, to provide a reproducible benchmark for future development. 1 Exploration in the context of reinforcement learning is a highly active area of research. Simple strategies such as epsilon-greedy remain extremely competitive (Mnih et al., 2015; Schaul et al., 2016) . However, a number of promising techniques have recently emerged that encourage exploration though carefully adding random noise to the parameters (Plappert et al., 2017; BID12 BID13 or bootstrap sampling (Osband et al., 2016) before making decisions. These methods rely explicitly or implicitly on posterior sampling for exploration.In this paper, we investigate how different posterior approximations affect the performance of Thompson Sampling from an empirical standpoint. For simplicity, we restrict ourselves to one of the most basic sequential decision making scenarios: that of contextual bandits.No single algorithm bested the others in every bandit problem, however, we observed some general trends. We found that dropout, injecting random noise, and bootstrapping did provide a strong boost in performance on some tasks, but was not able to solve challenging synthetic exploration tasks. Other algorithms, like Variational Inference, Black Box α-divergence, and minibatch Markov Chain Monte Carlo approaches, strongly couple their complex representation and uncertainty estimates. This proves problematic when decisions are made based on partial optimization of both, as online scenarios usually require. On the other hand, making decisions according to a Bayesian linear regression on the representation provided by the last layer of a deep network offers a robust and easy-to-tune approach. It would be interesting to try this approach on more complex reinforcement learning domains.In Section 2 we discuss Thompson Sampling, and present the contextual bandit problem. The different algorithmic approaches that approximate the posterior distribution fed to Thompson Sampling are introduced in Section 3, while the linear case is described in Section 4. The main experimental results are presented in Section 5, and discussed in Section 6. Finally, Section 7 concludes. Overall, we found that there is significant room for improvement in uncertainty estimation for neural networks in sequential decision-making problems. First, unlike in supervised learning, sequential decision-making requires the model to be frequently updated as data is accumulated. As a result, methods that converge slowly are at a disadvantage because we must truncate optimization to make the method practical for the online setting. In these cases, we found that partially optimized uncertainty estimates can lead to catastrophic decisions and poor performance. Second, and while it deserves further investigation, it seems that decoupling representation learning and uncertainty estimation improves performance. The NeuralLinear algorithm is an example of this decoupling. With such a model, the uncertainty estimates can be solved for in closed form (but may be erroneous due to the simplistic model), so there is no issue with partial optimization. We suspect that this may be the reason for the improved performance. In addition, we observed that many algorithms are sensitive to their hyperparameters, so that best configurations are problem-dependent.Finally, we found that in many cases, the inherit randomness in Stochastic Gradient Descent provided sufficient exploration. Accordingly, in some scenarios it may be hard to justify the use of complicated (and less transparent) variations of simple methods. However, Stochastic Gradient Descent is by no The suffix of the BBB legend label indicates the number of training epochs in each training step. We emphasize that in this evaluation, all algorithms use the same family of models (i.e., linear). While PrecisionDiag exactly solves the mean field problem, BBB relies on partial optimization via SGD. As the number of training epochs increases, BBB improves performance, but is always outperformed by PrecisionDiag.means always enough: in our synthetic exploration-oriented problem (the Wheel bandit) additional exploration was necessary.Next, we discuss our main findings for each class of algorithms.Linear Methods. Linear methods offer a reasonable baseline, surprisingly strong in many cases. While their representation power is certainly a limiting factor, their ability to compute informative uncertainty measures seems to payoff and balance their initial disadvantage. They do well in several datasets, and are able to react fast to unexpected or extreme rewards (maybe as single points can have a heavy impact in fitted models, and their updates are immediate, deterministic, and exact). Some datasets clearly need more complex non-linear representations, and linear methods are unable to efficiently solve those. In addition, linear methods obviously offer computational advantages, and it would be interesting to investigate how their performance degrades when a finite data buffer feeds the estimates as various real-world online applications may require (instead of all collected data).In terms of the diagonal linear approximations described in Section 3, we found that diagonalizing the precision matrix (as in mean-field Variational Inference) performs dramatically better than diagonalizing the covariance matrix.NeuralLinear. The NeuralLinear algorithm sits near a sweet spot that is worth further studying. In general it seems to improve the RMS neural network it is based on, suggesting its exploration mechanisms add concrete value. We believe its main strength is that it is able to simultaneously learn a data representation that greatly simplifies the task at hand, and to accurately quantify the uncertainty over linear models that explain the observed rewards in terms of the proposed representation. While the former process may be noisier and heavily dependent on the amount of training steps that were taken and available data, the latter always offers the exact solution to its approximate parent problem. This, together with the partial success of linear methods with poor representations, may explain its promising results. In some sense, it knows what it knows. In the Wheel problem, which requires increasingly good exploration mechanisms, NeuralLinear is probably the best algorithm. Its performance is almost an order of magnitude better than any RMS algorithm (and its spinoffs, like Bootstrapped NN, Dropout, or Parameter Noise), and all greedy linear approaches. On the other hand , it is able to successfully solve problems that require non-linear representations (as Statlog or Covertype) where linear approaches fail. In addition, the algorithm is remarkably easy to tune, and robust in terms of hyper-parameter configurations. While conceptually simple, its deployment to large scale systems may involve some technical difficulties; mainly, to update the Bayesian estimates when the network is re-trained. We believe, however , standard solutions to similar problems (like running averages) could greatly mitigate these issues. In our experiments and compared to other algorithms, as shown in Table 8 , NeuralLinear is fast from a computational standpoint.Variational Inference. Overall, Bayes By Backprop performed poorly, ranking in the bottom half of algorithms across datasets TAB0 . To investigate if this was due to underestimating uncertainty (as variational methods are known to BID5 ), to the mean field approximation, or to stochastic optimization, we applied BBB to a linear model, where the mean field optimization problem can be solved in closed form FIG5 . We found that the performance of BBB slowly improved as the number of training epochs increased, but underperformed compared to the exact mean field solution. Moreover, the difference in performance due to the number of training steps dwarfed the difference between the mean field solution and the exact posterior. This suggests that it is not sufficient to partially optimize the variational parameters when the uncertainty estimates directly affect the data being collected. In supervised learning, optimizing to convergence is acceptable, however in the online setting, optimizing to convergence at every step incurs unreasonable computational cost.Expectation-Propagation. The performance of Black Box α-divergence algorithms was poor. Because this class of algorithms is similar to BBB (in fact, as α → 0, it converges to the BBB objective), we suspect that partial convergence was also the cause of their poor performance. We found these algorithms to be sensitive to the number of training steps between actions, requiring a large number to achieve marginal performance. Their terrible performance in the Mushroom bandit is remarkable, while in the other datasets they perform slightly worse than their variational inference counterpart. Given the successes of Black Box α-divergence in other domains BID19 , investigating approaches to sidestep the slow convergence of the uncertainty estimates is a promising direction for future work.Monte Carlo. Constant-SGD comes out as the winner on Covertype, which requires non-linearity and exploration as evidenced by performance of the linear baseline approaches TAB0 ). The method is especially appealing as it does not require tuning learning rates or exploration parameters. SGFS, however, performs better on average . The additional injected noise in SGFS may cause the model to explore more and thus perform better, as shown in the Wheel Bandit problem where SGFS strongly outperforms Constant-SGD.Bootstrap. The bootstrap offers significant gains with respect to its parent algorithm (RMS) in several datasets. Note that in Statlog one of the actions is optimal around 80% of the time, and the bootstrapped predictions may help to avoid getting stuck, something from which RMS methods may suffer. In other scenarios, the randomness from SGD may be enough for exploration, and the bootstrap may not offer important benefits. In those cases, it might not justify the heavy computational overhead of the method. We found it surprising that the optimized versions of BootstrappedNN decided to use only q = 2 and q = 3 networks respectively (while we set its value to q = 10 in the manually tuned version, and the extra networks did not improve performance significantly). Unfortunately, Bootstrapped NNs were not able to solve the Wheel problem, and its performance was fairly similar to that of RMS. One possible explanation is that -given the sparsity of the rewardall the bootstrapped networks agreed for the most part, and the algorithm simply got stuck selecting action a 1 . As opposed to linear models, reacting to unusual rewards could take Bootstrapped NNs some time as good predictions could be randomly overlooked (and useful data discarded if p 1).Direct Noise Injection. When properly tuned, Parameter-Noise provided an important boost in performance across datasets over the learner that it was based on (RMS), average rank of ParamNoise-SL is 20.9 compared to RMS at 28.7 TAB0 . However, we found the algorithm hard to tune and sensitive to the heuristic controlling the injected noise-level. On the synthetic Wheel problem -where exploration is necessary-both parameter-noise and RMS suffer from underexploration and perform similarly, except ParamNoise-MR which does a good job. In addition, developing an intuition for the heuristic is not straightforward as it lacks transparency and a principled grounding, and thus may require repeated access to the decision-making process for tuning.Dropout. We initially experimented with two dropout versions: fixed p = 0.5, and p = 0.8 . The latter consistently delivered better results, and it is the one we manually picked. The optimized versions of the algorithm provided decent improvements over its base RMS (specially Dropout-MR).In the Wheel problem, dropout performance is somewhat poor: Dropout is outperformed by RMS, while Dropout-MR offers gains with respect to all versions of RMS but it is not competitive with the best algorithms. Overall, the algorithm seems to heavily depend on its hyper-parameters (see cum-regret performance of the raw Dropout, for example). Dropout was used both for training and for decision-making; unfortunately, we did not add a baseline where dropout only applies during training. Consequently, it is not obvious how to disentangle the contribution of better training from that of better exploration. This remains as future work.Bayesian Non-parametrics. Perhaps unsurprisingly, Gaussian processes perform reasonably well on problems with little data but struggle on larger problems. While this motivated the use of sparse GP, the latter was not able to perform similarly to stronger (and definitively simpler) methods. In this work, we empirically studied the impact on performance of approximate model posteriors for decision making via Thompson Sampling in contextual bandits. We found that the most robust methods exactly measured uncertainty (possibly under the wrong model assumptions) on top of complex representations learned in parallel. More complicated approaches that learn the representation and its uncertainty together seemed to require heavier training, an important drawback in online scenarios, and exhibited stronger hyper-parameter dependence. Further exploring and developing the promising approaches is an exciting avenue for future work.
An Empirical Comparison of Bayesian Deep Networks for Thompson Sampling
{"src": "scitldr-AIC_train", "paper_id": "SyYe6k-CW"}