We propose a new output layer for deep neural networks that permits the use of logged contextual bandit feedback for training. Such contextual bandit feedback can be available in huge quantities (e.g., logs of search engines, recommender systems) at little cost, opening up a path for training deep networks on orders of magnitude more data. To this effect, we propose a Counterfactual Risk Minimization (CRM) approach for training deep networks using an equivariant empirical risk estimator with variance regularization, BanditNet, and show how the resulting objective can be decomposed in a way that allows Stochastic Gradient Descent (SGD) training. We empirically demonstrate the effectiveness of the method by showing how deep networks -- ResNets in particular -- can be trained for object recognition without conventionally labeled images. Log data can be recorded from online systems such as search engines, recommender systems, or online stores at little cost and in huge quantities. For concreteness, consider the interaction logs of an ad-placement system for banner ads. Such logs typically contain a record of the input to the system (e.g., features describing the user, banner ad, and page), the action that was taken by the system (e.g., a specific banner ad that was placed) and the feedback furnished by the user (e.g., clicks on the ad, or monetary payoff). This feedback, however, provides only partial information -"contextual-bandit feedback" -limited to the actions taken by the system. We do not get to see how the user would have responded, if the system had chosen a different action (e.g., other ads or banner types). Thus, the feedback for all other actions the system could have taken is typically not known. This makes learning from log data fundamentally different from traditional supervised learning, where "correct" predictions and a loss function provide feedback for all actions.In this paper, we propose a new output layer for deep neural networks that allows training on logged contextual bandit feedback. By circumventing the need for full-information feedback, our approach opens a new and intriguing pathway for acquiring knowledge at unprecedented scale, giving deep neural networks access to this abundant and ubiquitous type of data. Similarly, it enables the application of deep learning even in domains where manually labeling full-information feedback is not viable.In contrast to online learning with contextual bandit feedback (e.g., BID11 BID0 ), we perform batch learning from bandit feedback (BLBF) BID1 BID5 and the algorithm does not require the ability to make interactive interventions. At the core of the new output layer for BLBF training of deep neural networks lies a counterfactual training objective that replaces the conventional cross-entropy objective. Our approach -called BanditNet -follows the view of a deep neural network as a stochastic policy. We propose a counterfactual risk minimization (CRM) objective that is based on an equivariant estimator of the true error that only requires propensity-logged contextual bandit feedback. This makes our training objective fundamentally different from the conventional cross-entropy objective for supervised classification, which requires full-information feedback. Equivariance in our context means that the learning result is invariant to additive translations of the loss, and it is more formally defined in Section 3.2. To enable large-scale training, we show how this training objective can be decomposed to allow stochastic gradient descent (SGD) optimization.In addition to the theoretical derivation of BanditNet, we present an empirical evaluation that verifies the applicability of the theoretical argument. It demonstrates how a deep neural network architec-ture can be trained in the BLBF setting. In particular, we derive a BanditNet version of ResNet (He et al., 2016) for visual object classification. Despite using potentially much cheaper data, we find that Bandit-ResNet can achieve the same classification performance given sufficient amounts of contextual bandit feedback as ResNet trained with cross-entropy on conventionally (full-information) annotated images. To easily enable experimentation on other applications, we share an implementation of BanditNet. 1 2 RELATED WORK Several recent works have studied weak supervision approaches for deep learning. Weak supervision has been used to pre-train good image features (Joulin et al., 2016) and for information retrieval BID3 . Closely related works have studied label corruption on CIFAR-10 recently BID12 . However, all these approaches use weak supervision/corruption to construct noisy proxies for labels, and proceed with traditional supervised training (using crossentropy or mean-squared-error loss) with these proxies. In contrast, we work in the BLBF setting, which is an orthogonal data-source, and modify the loss functions optimized by deep nets to directly implement risk minimization.Virtually all previous methods that can learn from logged bandit feedback employ some form of risk minimization principle BID9 over a model class. Most of the methods BID1 BID2 BID5 employ an inverse propensity scoring (IPS) estimator (Rosenbaum & Rubin, 1983) as empirical risk and use stochastic gradient descent (SGD) to optimize the estimate over large datasets. Recently, the self-normalized estimator BID8 ) has been shown to be a more suitable estimator for BLBF BID7 . The self-normalized estimator, however, is not amenable to stochastic optimization and scales poorly with dataset size. In our work, we demonstrate how we can efficiently optimize a reformulation of the self-normalized estimator using SGD.Previous BLBF methods focus on simple model classes: log-linear and exponential models (Swaminathan & Joachims, 2015a) or tree-based reductions BID1 ). In contrast, we demonstrate how current deep learning models can be trained effectively via batch learning from bandit feedback (BLBF), and compare these with existing approaches on a benchmark dataset (Krizhevsky & Hinton, 2009 ).Our work, together with independent concurrent work BID4 , demonstrates success with off-policy variants of the REINFORCE BID11 algorithm. In particular, our algorithm employs a Lagrangian reformulation of the self-normalized estimator, and the objective and gradients of this reformulation are similar in spirit to the updates of the REINFORCE algorithm. This connection sheds new light on the role of the baseline hyper-parameters in REINFORCE: rather than simply reduce the variance of policy gradients, our work proposes a constructive algorithm for selecting the baseline in the off-policy setting and it suggests that the baseline is instrumental in creating an equivariant counterfactual learning objective. We proposed a new output layer for deep neural networks that enables the use of logged contextual bandit feedback for training. This type of feedback is abundant and ubiquitous in the form of interaction logs from autonomous systems, opening up the possibility of training deep neural networks on unprecedented amounts of data. In principle, this new output layer can replace the conventional cross-entropy layer for any network architecture. We provide a rigorous derivation of the training objective, linking it to an equivariant counterfactual risk estimator that enables counterfactual risk minimization. Most importantly, we show how the resulting training objective can be decomposed and reformulated to make it feasible for SGD training. We find that the BanditNet approach applied to the ResNet architecture achieves predictive accuracy comparable to conventional full-information training for visual object recognition.The paper opens up several directions for future work. First, it enables many new applications where contextual bandit feedback is readily available. Second, in settings where it is infeasible to log propensity-scored data, it would be interesting to combine BanditNet with propensity estimation techniques. Third, there may be improvements to BanditNet, like smarter search techniques for S, more efficient counterfactual estimators beyond SNIPS, and the ability to handle continuous outputs. DISPLAYFORM0 If the optimaŵ a andŵ b are not equivalent in the sense thatR DISPLAYFORM1 where g(w ) corresponds to the value of the control variate S. Sinceŵ a andŵ b are not equivalent optima, we know that DISPLAYFORM2 Adding the two inequalities and solving implies that DISPLAYFORM3 B APPENDIX: CHARACTERIZING THE RANGE OF S TO EXPLORE.Theorem 2. Let p ≤ π 0 (y | x) be a lower bound on the propensity for the logging policy, then constraining the solution of Eq. (11) to the w with control variate S ∈ [1 − , 1 + ] for a training set of size n will not exclude the minimizer of the true risk w * = arg min w∈W R(π w ) in the policy space W with probability at least DISPLAYFORM4 Proof. For the optimal w * , let DISPLAYFORM5 be the control variate in the denominator of the SNIPS estimator. S is a random variable that is a sum of bounded random variables between 0 and DISPLAYFORM6 We can bound the probability that the control variate S of the optimum w * lies outside of [1− , 1+ ] via Hoeffding's inequality: DISPLAYFORM7 The same argument applies to any individual policy π w , not just w * . Note , however, that it can still be highly likely that at least one policy π w with w ∈ W shows a large deviation in the control variate for high-capacity W , which can lead to propensity overfitting when using the naive IPS estimator. Suppose we have a dataset of n BLBF samples D = {(x 1 , y 1 , δ 1 , p 1 ) . . . (x n , y n , δ n , p n )} where each instance is an i.i.d. sample from the data generating distribution. In the sequel we will be considering two datasets of n + 1 samples, D = D ∪ {(x , y , δ , p )} and D = D ∪ {(x , y , δ , p )} where (x , y , δ , p ) = (x , y , δ , p ) and (x , y , δ , p ), (x , y , δ , p ) / ∈ D.For notational convenience, let DISPLAYFORM8 π0(yi|xi) , andġ i := ∇ w g i . First consider the vanilla IPS risk estimate of Eq. (5). DISPLAYFORM9 To maximize this estimate using stochastic optimization, we must construct an unbiased gradient estimate. That is, we randomly select one sample from D and compute a gradient α((x i , y i , δ i , p i )) and we require that DISPLAYFORM10 Here the expectation is over our random choice of 1 out of n samples. Observe that α((x i , y i , δ i , p i )) =ḟ i suffices (and indeed, this corresponds to vanilla SGD): DISPLAYFORM11 Other choices of α(·) can also produce unbiased gradient estimates, and this leads to the study of stochastic variance-reduced gradient optimization. Now let us attempt to construct an unbiased gradient estimate for Eq. (8): DISPLAYFORM12 Suppose such a gradient estimate exists, β((x i , y i , δ i , p i )). Then, DISPLAYFORM13 This identity is true for any sample of BLBF instances -in particular, for D and D : DISPLAYFORM14 1 n + 1 β((x i , y i , δ i , p i )) + β((x , y , δ , p )) n + 1 , DISPLAYFORM15 1 n + 1 β((x i , y i , δ i , p i )) + β((x , y , δ , p )) n + 1 .Subtracting these two equations, DISPLAYFORM16 = β((x , y , δ , p )) − β((x , y , δ , p )) n + 1 .The LHS clearly depends on {(x i , y i , δ i , p i )} n i=1 in general, while the RHS does not! This contradiction indicates that no construction of β that only looks at a sub-sample of the data can yield an unbiased gradient estimate ofR SNIPS (π w ).
The paper proposes a new output layer for deep networks that permits the use of logged contextual bandit feedback for training. 
{"src": "scitldr-AIC_train", "paper_id": "SJaP_-xAb"}