Proper Scoring and Sufficiency

Logarithmic score and information divergence appear in both information theory, statistics, statistical mechanics, and portfolio theory. We demonstrate that all these topics involve some kind of optimization that leads directly to the use of Bregman divergences. If a sufficiency condition is also fulfilled the Bregman divergence must be proportional to information divergence. The sufficiency condition has quite different consequences in the different areas of application, and often it is not fulfilled. Therefore the sufficiency condition can be used to explain when results from one area can be transferred directly from one area to another and when one will experience differences.


INTRODUCTION
The use of scoring rules has a long history in statistics. An early contribution was the idea of minimizing the sum of square deviations that dates back to Gauss and works perfectly for Gaussian distributions. In the 1920's Ramsay and de Finetti proved versions of the Dutch book theorem where determination of probability distributions were considered as dual problems to maximizing a payoff function. Later it was proved that any consistent inference corresponds to optimizing with respect to some payoff function. A more systematic study of scoring rules was given by McCarthy [1] and has recently been studied by Dawid, Lauritzen and Parry [2] where the notion of a local scoring rule has been extended. The basic result is that the only strictly local proper scoring rule is logarithmic score.
Thermodynamics is the study of concepts like heat, temperature and energy. A major objective is to extract as much energy from a system as possible. Concepts like entropy and free energy play a significant role. The idea in statistical mechanics is to view the macroscopic behavior of a thermodynamic system as a statistical consequence of the interaction between a lot of microscopic components where the interacting between the components are governed by very simple laws. Here the central limit theorem and large deviation theory play a major role. One of the main achievements is the formula for entropy as a logarithm of a probability.
One of the main purposes of information theory is to compress data so that data can be recovered exactly or approximately. One of the most important quantities was called entropy because it is calculated according to a formula that mimics the calculation of entropy in statistical mechanics. Another key concept in information theory is information divergence (KL-divergence) that was introduced by Kullback and Leibler in 1951 in a paper entitled information and sufficiency. The link from information theory back to statistical physics was developed by E.T. Jaynes via the maximum entropy principle. The link back to statistics is now well established [3,4,5].
The relation between information theory and gambling was established by Kelly [6]. Logarithmic terms appear because we are interested in the exponent in an exponential growth rate of of our wealth. Later Kelly's approach has been generalized to training of stocks although the relation to information theory is weaker [7].
Related quantities appear in statistics, statistical mechanics, information theory and finance, annd we are interested in a theory that describes when these relations are exact and when they just work by analogy. First we introduce some general results about optimization on convex sets. This part applies exactly to all the topics under consideration and lead to Bregman divergences. Secondly, we introduce a notion of sufficiency and show that this leads to information divergence and logarithmic score. This second step is not always applicable which explains when the different topics are really different.
Proofs of the theorems in this short paper can be found in an appendix that is part of the arXiv version of the paper.

STATE SPACE
The present notion of a state space is based on [8], and is mainly relevant for quantum systems.
Before we do anything we prepare our system. Let P denote the set of preparations. Let p 0 and p 1 denote two preparations. For t ∈ [0, 1] we define (1 − t) · p 0 + t · p 1 as the preparation obtained by preparing p 0 with probability 1−t and t with probability t. A measurement m is defined as an affine mapping of the set of preparations into a set of probability measures on some measurable space. Let M denote a set of feasible measurements. The state space S is defined as the set of preparations modulo measurements. Thus, if p 1 and p 2 are preparations then they represent the same state if m (p 1 ) = m (p 2 ) for any m ∈ M.
In statistics the state space equals the set of preparations and has the shape of a simplex. The symmetry group of a simplex is simply the group of permutations of the extreme points. In quantum theory the state space has the shape of the density matrices on a complex Hilbert space and the state space has a lot of symmetries that a simplex does not have. For simplicity we will assume that the state space is a finite dimensional convex compact space.

OPTIMIZATION
Let A denote a subset of the feasible measurements M such that a ∈ A maps S into a distribution on the real numbers i.e. a random variable. The elements of A may represent actions like the score of a statistical decision, the energy extracted by a certain interaction with the system, (minus) the length of a codeword of the next encoded input letter using a specific code book, or the revenue of using a certain portfolio. For each s ∈ S we define F (s) = sup a∈A E [a (s)]. We note that F is convex but F need not be strictly convex. We say that a sequence of actions (a n ) n is asymptotically optimal for the state s if E [a n (s)] → F (s) for n → ∞.
If the state is s 1 but one acts as if the state were s 2 one suffers a regret that equals the difference between what one achieves and what could have been achieved.
where the supremum is taken over all sequences (a n ) n that are asymptotically optimal over s 2 .
Proposition 2. The regret D F has the following properties: If the state space is finite dimensional and there exists a unique action a 2 such that F ( If unique optimal actions exists for any state then F is differentiable which implies that the regret can be written as a Bregman divergence in the following form In the context of forecasting and statistical scoring rules the use of Bregman divergences dates back to [9]. We note that D F1 (s 1 , s 2 ) = D F2 (s 1 , s 2 ) if and only if F 1 (s) − F 2 (s) is an affine function of s. If the state s 2 has the unique optimal action a 2 then so the function F can be reconstructed from D F except for an affine function of s 1 . The closure of the convex hull of the set of functions s → E [a (s)] is uniquely determined by the convex function F.

SUFFICIENCY
Let (s θ ) θ denote a family of states and let Φ denote a completely positive transformation S → T where S and T denote state spaces. Then Φ is said to be sufficient for (s θ ) θ if there exists a completely positive transformation We say that the regret D F on the state space S satisfies the sufficiency property if D F (Φ (s 1 ) , Φ (s 2 )) = D F (s 1 , s 2 ) for any completely positive transformation S → S that is sufficient for (s 1 , s 2 ) . The notion of sufficiency as a property of divergences was introduced in [10]. The crucial idea of restricting the attention to transformations of the state space into itself was introduced in [11].
Theorem 3. Assume that S is a state space. If the divergence D F satisfies the sufficiency property then for any state s and any completely positive transformation Φ : If the alphabet size is two the above condition on F is sufficient to conclude that Theorem 4. Assume that the state space S is a classical or quantum state space on three or more letters. If the regret D F satisfies the sufficiency property, then F is proportional to the entropy function and D F is proportional to information divergence (relative entropy).
This theorem can be proved via a numer of partial results as explained in the next section.

Statistics
Consider an experiment with X = {1, 2, . . . , } as sample space. A scoring rule f is defined as a function with domain X × M + 1 (X ) → R such that the score is f (x, Q) when the prediction was given by Q and x ∈ X has been observed. A scoring rule is proper if for any probability measure P ∈ M + 1 (X ) the score x∈X P (x) · f (x, Q) is minimal when Q = P. Lemma 7. On a finite space a Bregman divergence that satisfies the sufficiency condition gives a strictly local scoring rule.
The following theorem was given in [11] with a much longer proof.
Theorem 8. On a finite alphabet with at least three letters a Bregman divergence that satisfies the sufficiency condition is proportional to information divergence.
Proof. Since any strictly local proper scoring rule corresponds to separable divergence a divergence that is Bregman and satisfies sufficiency must also be separable. If the alphabet size is at least three the only separable divergences that are Bregman divergences are the ones proportional to information divergence [10].

Information theory
Let b 1 , b 2 , . . . , b n denote the letters of an alphabet and let (κ (b i )) denote the length of the codeword κ (b i ) according to some code book κ. If the code is uniquely decodable then 2 − (κ(bi)) ≤ 1. Note that (κ (b i )) is an integer. If only integer values of are allowed then h is piecewise linear and sufficiency is not fulfilled. If arbitrary real numbers are allowed then it obvious we get a proper local scoring rule.

Statistical mechanics
Statistical mechanics can be stated based on classical mechanics or quantum mechanics. For our purpose this makes no difference because Theorem 4 can be applied for both classical systems and quantum systems.
Proof of Theorem 4. If we restrict to any commutative subalgebra the divergence is proportional to information divergence as stated in Theorem 8 so that F is proportional to the entropy function H restricted to the sub-algebra. Any state generates a commutative sub-algebra so the function F is proportional to H on all states and the divergence is proportional to information divergence.
Assume that a heat bath of temperature T is given and that all the states are close to the state of the heat bath. An action a ∈ A is some interaction with the thermodynamic system that extracts some energy from the system. In thermodynamics the quantity F (s) = sup a∈A E [a (s)] is normally called the free energy. If the temperature is kept fixed under all interactions F is called Helmholtz free energy. Any sufficient transformation Φ for s 1 and s 2 is quasi-static and can be approximately realized by a physical process Ψ that is reversible in the thermodynamic sense of the word.
≤ a 2 (s 2 ) = a 2 (Ψ (Φ (s 2 ))) = (a 2 • Ψ) (Φ (s 2 )) ≤ a Φ(s2) (Φ (s 2 )) . (6) Hence a Φ(s2) = a 2 • Ψ so that The amount of extractable energy Ex is proportional to information divergence. The quotient between extractable energy and information divergence depends on the temperature and one may even define the absolute temperature via the formula where k = 1.381 · 10 −23 J /K is Boltzmann's constant. Equation (8) was derived already in [12] by a similar argument. According to Equation (8) any bit of information can be converted into an amount of energy! One may ask how this is related to the mixing paradox (a special case of Gibbs' paradox). Consider a container divided by a wall with a blue and a yellow gas on each side of the wall. The question is how much energy can be extracted by mixing the gasses?
We loose one bit of information about each molecule by mixing the gasses, but if the color is the only difference no energy can be extracted. This seems to be in conflict with Equation (8), but in this case different states cannot be converted into each other by reversible processes. For instance one cannot convert the blue gas into the yellow gas. To get around this problem one can restrict the set of preparations and one can restrict the set of measurements. For instance one may simply ignore measurements of the color of the gas. What should be taken into account and what should be ignored, can only be answered by an experienced physicist. Formally this solves the mixing paradox but from a practical point of view nothing has been solved. If for instance the molecules in one of the gasses are much larger than the molecules in the other gas then a semi-permeable membrane can be used to create an osmotic pressure that can be used to extract some energy. It is still an open question which differences in properties of the two gasses that can be used to extract energy.

Portfolio theory
Let X 1 , X 2 , . . . , X k denote price relatives for a list of stocks. For instance X 5 = 1.04 means that stock no. 5 increases its value by 4 %. A portfolio is a probability vector b = (b 1 , b 2 , . . . , b k ) where for instance b 5 = 0.3 means that 30 % of your money is invested in stock no. 5. The total price relative is X 1 ·b 1 +X 2 ·b 2 +· · ·+X k ·b k = X · b. We now consider a situation where the stocks are traded once every day. For a sequence of price relative vectors X 1 , X 2 , . . . X n and a constant re-balancing portfolio b the wealth after n days is According to law of large numbers Here E log X, b is proportional to the doubling rate and is denoted W b, P where P indicates the probability distribution of X. Our goal to maximize W b, P by choosing an appropriate portfolio b. Let b P denote the portfolio that is optimal for P . As proved in [7] Theorem 9. The Bregman divergence satisfies the equation if and only if the measure P on k distinct vectors of the form (a 1 , 0, 0, . . . 0), (0, a 2 , 0, . . . 0) , until (0, 0, . . . a k ) .

CONCLUSION
On the level of optimization the theory works out in exactly the same way in statistics, information theory, statistical mechanics, and portfolio theory. The sufficiency condition is more complicated to apply. It requires that we restrict to a certain class of mappings of the state space into itself. In the case where the state space can be identified with a set of density matrices one should restrict to completely positive maps. In case the state space has a different structure it is not obvious which mappings one should restrict to. The basic problem is that we have to introduce a notion of tensor product for convex sets and it is not obvious how to do this, but this will be the topic of further investigations and results on this topic may have some impact on our general understanding of quantum theory. The original paper of Kullback and Leibler [13] was called "On Information and Sufficiency". In the present paper we have made the relation between information divergence and the notion of sufficiency more explicit. The idea of sufficiency has different consequences in different applications but in all cases information divergence prove to be the quantity that convert the general notion of sufficiency into a number. For specific applications one cannot identify the sufficient variables without studying the specific application in detail. For problems like the the mixing paradox there is still no simple answer to the question about what the sufficient variables are, but if the sufficient variables have been specified we have the mathematical framework to develop the rest of the theory in a consistent manner.

Proof of Theorem 5
If f is given in terms of a regret function D F then Assume that f is proper. Then we may define a divergence by (15) Since f is assumed to be proper D (P, Q) ≥ 0 with equality if and only if P = Q. The equality t i ·D F (P i , Q) = t i · D F P i ,P + D F P , Q follows by straight forward calculations. With these two results we see that D equals a Bregman divergence D F and that

Proof of Lemma 7
Let D F denote a regret function that satisfies the sufficiency condition. Then where we have made a cyclic permutation of indices. Next we use the sufficient transformation that projects a mixture of δ 1 and a uniform distribution.
Note that the projection can be obtained by taking a mixture of all permutations of the extreme points that leave the first extreme point unchanged. Hence the scoring rule is given by the local scoring rule

Proof of Theorem 9
We have Next we use Jensen's inequality to get is constant P -almost surely. Equivalently dQ dP is proportional to X, b Q X, b P for any probability measure Q on the support of X. The set of vectors b Q lie in a k − 1 dimensional convex set. Therefore the set of probability measures on the support of P is at most k − 1 dimensional. Hence P is supported on at most k vectors in R k 0,+ . The inequalitŷ holds with equality if b Q