We derive an unbiased estimator for expectations over discrete random variables based on sampling without replacement, which reduces variance as it avoids duplicate samples. We show that our estimator can be derived as the Rao-Blackwellization of three different estimators. Combining our estimator with REINFORCE, we obtain a policy gradient estimator and we reduce its variance using a built-in control variate which is obtained without additional model evaluations. The resulting estimator is closely related to other gradient estimators. Experiments with a toy problem, a categorical Variational Auto-Encoder and a structured prediction problem show that our estimator is the only estimator that is consistently among the best estimators in both high and low entropy settings. Put replacement in your basement! We derive the unordered set estimator: an unbiased (gradient) estimator for expectations over discrete random variables based on (unordered sets of) samples without replacement. In particular, we consider the problem of estimating (the gradient of) the expectation of f (x) where x has a discrete distribution p over the domain D, i.e. This expectation comes up in reinforcement learning, discrete latent variable modelling (e.g. for compression), structured prediction (e.g. for translation), hard attention and many other tasks that use models with discrete operations in their computational graphs (see e.g. Jang et al. (2016) ). In general, x has structure (such as a sequence), but we can treat it as a 'flat' distribution, omitting the bold notation, so x has a categorical distribution over D given by p(x), x ∈ D. Typically, the distribution has parameters θ, which are learnt through gradient descent. This requires estimating the gradient ∇ θ E x∼p θ (x) [f (x)], using a set of samples S. A gradient estimate e(S) is unbiased if The samples S can be sampled independently or using alternatives such as stratified sampling which reduce variance to increase the speed of learning. In this paper, we derive an unbiased gradient estimator that reduces variance by avoiding duplicate samples, i.e. by sampling S without replacement. This is challenging as samples without replacement are dependent and have marginal distributions that are different from p(x). We further reduce the variance by deriving a built-in control variate, which maintains the unbiasedness and does not require additional samples. Related work. Many algorithms for estimating gradients for discrete distributions have been proposed. A general and widely used estimator is REINFORCE (Williams, 1992) . Biased gradients based on a continuous relaxations of the discrete distribution (known as Gumbel-Softmax or Concrete) were jointly introduced by Jang et al. (2016) and Maddison et al. (2016) . These can be combined with the straight through estimator (Bengio et al., 2013) if the model requires discrete samples or be used to construct control variates for REINFORCE, as in REBAR (Tucker et al., 2017) or RELAX (Grathwohl et al., 2018) . Many other methods use control variates and other techniques to reduce the variance of REINFORCE (Paisley et al., 2012; Ranganath et al., 2014; Gregor et al., 2014; Mnih & Gregor, 2014; Gu et al., 2016; Mnih & Rezende, 2016) . Some works rely on explicit summation of the expectation, either for the marginal distribution (Titsias & Lázaro-Gredilla, 2015) or globally summing some categories while sampling from the remainder (Liang et al., 2018; Liu et al., 2019) . Other approaches use a finite difference approximation to the gradient (Lorberbom et al., 2018; 2019) . Yin et al. (2019) introduced ARSM, which uses multiple model evaluations where the number adapts automatically to the uncertainty. In the structured prediction setting, there are many algorithms for optimizing a quantity under a sequence of discrete decisions, using (weak) supervision, multiple samples (or deterministic model evaluations), or a combination both (Ranzato et al., 2016; Shen et al., 2016; He et al., 2016; Norouzi et al., 2016; Bahdanau et al., 2017; Edunov et al., 2018; Leblond et al., 2018; Negrinho et al., 2018) . Most of these algorithms are biased and rely on pretraining using maximum likelihood or gradually transitioning from supervised to reinforcement learning. Using Gumbel-Softmax based approaches in a sequential setting is difficult as the bias accumulates because of mixing errors (Gu et al., 2018) . We introduced the unordered set estimator, a low-variance, unbiased gradient estimator based on sampling without replacement, which can be used as an alternative to the popular biased GumbelSoftmax estimator (Jang et al., 2016; Maddison et al., 2016) . Our estimator is the result of RaoBlackwellizing three existing estimators, which guarantees equal or lower variance, and is closely related to a number of other estimators. It has wide applicability, is parameter free (except for the sample size k) and has competitive performance to the best of alternatives in both high and low entropy regimes. In our experiments, we found that REINFORCE with replacement, with multiple samples and a built-in baseline as inspired by VIMCO (Mnih & Rezende, 2016) , is a simple yet strong estimator which has performance similar to our estimator in the high entropy setting. We are not aware of any recent work on gradient estimators for discrete distributions that has considered this estimator as baseline, while it may be often preferred given its simplicity. This means that F φ (g) is the CDF and f φ (g) the PDF of the Gumbel(φ) distribution. Additionally we will use the identities by Maddison et al. (2014): Also, we will use the following notation, definitions and identities (see Kool et al. (2019c) ): For a proof of equation 30, see Maddison et al. (2014) . We can sample the set S k from the Plackett-Luce distribution using the Gumbel-Top-k trick by drawing Gumbel variables G φi ∼ Gumbel(φ i ) for each element and returning the indices of the k largest Gumbels. If we ignore the ordering, this means we will obtain the set S k if min i∈S k G φi > max i∈D\S k G φi . Omitting the superscript k for clarity, we can use the Gumbel-Max trick, i.e. that G φ D\S = max i ∈S G φi ∼ Gumbel(φ D\S ) (equation 30) and marginalize over G φ D\S : Here we have used a change of variables u = F φ D\S (g φ D\S ). This expression can be efficiently numerically integrated (although another change of variables may be required for numerical stability depending on the values of φ). Exact computation in O(2 k ). The integral in equation 31 can be computed exactly using the identity i∈S Computation of p D\C (S \ C). When using the Gumbel-Top-k trick over the restricted domain D \ C, we do not need to renormalize the log-probabilities φ s , s ∈ D \ C since the Gumbel-Top-k trick applies to unnormalized log-probabilities. Also, assuming This means that we can compute p D\C (S \ C) similar to equation 31: Computation of R(S k , s). Note that, using equation 10, it holds that This means that, to compute the leave-one-out ratio for all s ∈ S k , we only need to compute p D\{s} (S k \ {s}) for s ∈ S k . When using the numerical integration or summation in O(2 k ), we can reuse computation, whereas using the naive method, the cost is O(k · (k − 1)! ) = O(k!), making the total computational cost comparable to computing just p(S k ), and the same holds when computing the 'second-order' leave one out ratios for the built-in baseline (equation 17). Details of numerical integration. For computation of the leave-one-out ratio (equation 35) for large k we can use the numerical integration, where we need to compute equation 34 with C = {s}. For this purpose, we rewrite the integral as Here we have used change of variables v = u exp(−b) and a = b − φ D\S . This form allows to compute the integrands efficiently, as where the numerator only needs to computed once, and, since C = {s} when computing equation 35, the denominator only consists of a single term. The choice of a may depend on the setting, but we found that a = 5 is a good default option which leads to an integral that is generally smooth and can be accurately approximated using the trapezoid rule. We compute the integrands in logarithmic space and sum the terms using the stable LOGSUMEXP trick. In our code we provide an implementation which also computes all second-order leave-one-out ratios efficiently.
We derive a low-variance, unbiased gradient estimator for expectations over discrete random variables based on sampling without replacement
{"src": "scitldr-AIC_train", "paper_id": "rklEj2EFvB"}