Ensembling multiple predictions is a widely used technique for improving the accuracy of various machine learning tasks. One obvious drawback of ensembling is its higher execution cost during inference. In this paper, we first describe our insights on the relationship between the probability of prediction and the effect of ensembling with current deep neural networks; ensembling does not help mispredictions for inputs predicted with a high probability even when there is a non-negligible number of mispredicted inputs. This finding motivated us to develop a way to adaptively control the ensembling. If the prediction for an input reaches a high enough probability, i.e., the output from the softmax function, on the basis of the confidence level, we stop ensembling for this input to avoid wasting computation power. We evaluated the adaptive ensembling by using various datasets and showed that it reduces the computation cost significantly while achieving accuracy similar to that of static ensembling using a pre-defined number of local predictions. We also show that our statistically rigorous confidence-level-based early-exit condition reduces the burden of task-dependent threshold tuning better compared with naive early exit based on a pre-defined threshold in addition to yielding a better accuracy with the same cost.
@cite_11 showed an important observation related to ours. They showed that the large part of the gain of ensembling came from the ensembling of the first few local predictions. Our observation discussed in the previous section enhances Opitz's observation from a different perspective: most gain of the ensembling comes from inputs with low probabilities in the prediction.
@cite_11
1988342708 1779497736 2419099043 2973707709
By introducing a quantitative degree of commutativity'' in terms of the angle between spin observables we present two tight quantitative trade-off relations in the case of two qubits. First, for entangled states, between the degree of commutativity of local observables and the maximal amount of violation of the Bell inequality: if both local angles increase from zero to @math (i.e., the degree of local commutativity decreases), the maximum violation of the Bell inequality increases. Secondly, a converse trade-off relation holds for separable states: if both local angles approach @math the maximal value obtainable for the correlations in the Bell inequality decreases and thus the non-violation increases. As expected, the extremes of these relations are found in the case of anticommuting local observables where, respectively, the bounds of @math and @math hold for the expectation value of the Bell operator. The trade-off relations show that noncommmutativity gives a more than classical result'' for entangled states, whereas a less than classical result'' is obtained for separable states. The experimental relevance of the trade-off relation for separable states is that it provides an experimental test for two qubit entanglement. Its advantages are twofold: in comparison to violations of Bell inequalities it is a stronger criterion and in comparison to entanglement witnesses it needs to make less strong assumptions about the observables implemented in the experiment. A microscopic calculation of ground state entanglement for the XY and Heisenbergmodels shows the emergence of universal scaling behavior at quantum phase transitions,Entanglement is thus controlled by conformal symmetry, Away from the critical point,entanglement gets saturated by a mass scale, Results borrowed from conformal fieldtheory imply irreversibility of entanglement loss along renormalization group trajecto-ries. Entanglement does not saturate in higher dimensions which appears to limit thesuccess of the density matrix renormalization group technique. A possible connection be-tween majorization and renormalization group irreversibility emerges from our numericalanalysis. We consider the following basic learning task: given independent draws from an unknown distribution over a discrete support, output an approximation of the distribution that is as accurate as possible in L1 distance (equivalently, total variation distance, or "statistical distance"). Perhaps surprisingly, it is often possible to "de-noise" the empirical distribution of the samples to return an approximation of the true distribution that is significantly more accurate than the empirical distribution, without relying on any prior assumptions on the distribution. We present an instance optimal learning algorithm which optimally performs this de-noising for every distribution for which such a de-noising is possible. More formally, given n independent draws from a distribution p, our algorithm returns a labelled vector whose expected distance from p is equal to the minimum possible expected error that could be obtained by any algorithm, even one that is given the true unlabeled vector of probabilities of distribution p and simply needs to assign labels---up to an additive subconstant term that is independent of p and goes to zero as n gets large. This somewhat surprising result has several conceptual implications, including the fact that, for any large sample from a distribution over discrete support, prior knowledge of the rates of decay of the tails of the distribution (e.g. power-law type assumptions) is not significantly helpful for the task of learning the distribution. As a consequence of our techniques, we also show that given a set of n samples from an arbitrary distribution, one can accurately estimate the expected number of distinct elements that will be observed in a sample of any size up to n log n. This sort of extrapolation is practically relevant, particularly to domains such as genomics where it is important to understand how much more might be discovered given larger sample sizes, and we are optimistic that our approach is practically viable. In sparse approximation theory, the fundamental problem is to reconstruct a signal A∈ℝn from linear measurements 〈Aψi〉 with respect to a dictionary of ψi's. Recently, there is focus on the novel direction of Compressed Sensing [9] where the reconstruction can be done with very few—O(k logn)—linear measurements over a modified dictionary if the signal is compressible, that is, its information is concentrated in k coefficients with the original dictionary. In particular, these results [9, 4, 23] prove that there exists a single O(k logn) ×n measurement matrix such that any such signal can be reconstructed from these measurements, with error at most O(1) times the worst case error for the class of such signals. Compressed sensing has generated tremendous excitement both because of the sophisticated underlying Mathematics and because of its potential applications In this paper, we address outstanding open problems in Compressed Sensing. Our main result is an explicit construction of a non-adaptive measurement matrix and the corresponding reconstruction algorithm so that with a number of measurements polynomial in k, logn, 1 e, we can reconstruct compressible signals. This is the first known polynomial time explicit construction of any such measurement matrix. In addition, our result improves the error guarantee from O(1) to 1 + e and improves the reconstruction time from poly(n) to poly(k logn) Our second result is a randomized construction of O(kpolylog (n)) measurements that work for each signal with high probability and gives per-instance approximation guarantees rather than over the class of all signals. Previous work on Compressed Sensing does not provide such per-instance approximation guarantees; our result improves the best known number of measurements known from prior work in other areas including Learning Theory [20, 21], Streaming algorithms [11, 12, 6] and Complexity Theory [1] for this case Our approach is combinatorial. In particular, we use two parallel sets of group tests, one to filter and the other to certify and estimate; the resulting algorithms are quite simple to implement