Combinatorial Information Theory: I. Philosophical Basis of Cross-Entropy and Entropy

This study critically analyses the information-theoretic, axiomatic and combinatorial philosophical bases of the entropy and cross-entropy concepts. The combinatorial basis is shown to be the most fundamental (most primitive) of these three bases, since it gives (i) a derivation for the Kullback-Leibler cross-entropy and Shannon entropy functions, as simplified forms of the multinomial distribution subject to the Stirling approximation; (ii) an explanation for the need to maximize entropy (or minimize cross-entropy) to find the most probable realization; and (iii) new, generalized definitions of entropy and cross-entropy - supersets of the Boltzmann principle - applicable to non-multinomial systems. The combinatorial basis is therefore of much broader scope, with far greater power of application, than the information-theoretic and axiomatic bases. The generalized definitions underpin a new discipline of ``{\it combinatorial information theory}'', for the analysis of probabilistic systems of any type. Jaynes' generic formulation of statistical mechanics for multinomial systems is re-examined in light of the combinatorial approach. (abbreviated abstract)


I. INTRODUCTION
The concept of entropy -a measure of the lack of order of a system -is one of the most profound human discoveries, with implications for virtually all disciplines of human study. The thermodynamic entropy was first defined by Clausius [1] in terms of the exact differential dS, given by the quantity of heat transferred reversibly to a system, dQ, scaled by the absolute temperature T of the system: Consideration of irreversible (non-equilibrium) processes -as expressed by the second law of thermodynamics -gives the Clausius [2] inequality: The combinatorial basis of entropy was revealed by Boltzmann [3] and Planck [4,5] in the famous equation: where S N is the total thermodynamic entropy of the system, N is the number of entities (discrete particles or agents) present, S is the thermodynamic entropy per entity, W is number of ways in which a specified realization 1 of a system can occur, known as its statistical weight, and k is the Boltzmann constant (1.38×10 23 J K −1 entity −1 ). Whilst the combinatorial definition is well-known in physics, most thermodynamicists, information theorists and mathematicians use the information entropy of Shannon [6] (or a simple multiple thereof, as given by Boltzmann [3]): where p i is the probability of occurrence of the ith distinguishable outcome or state, from s such states, and p = {p i } is the probability distribution (probability mass function). In the maximum entropy principle ("MaxEnt") developed by Jaynes [7], (4) is maximized subject 1 Here the state refers to each different category (e.g. boxes, energy levels, elements or results) accessible to a system; a configuration is a distinguishable permutution or pattern of entities amongst the states of a system (complexion, microstate, sequence); and a realization is an externally identifiable set of such configurations, grouped in a specified manner (macrostate, outcome, type).
to the constraints on a system, giving a probability distribution p * which is considered to be the most uncertain, the least informative or the least committed to information not given, and is therefore used to represent the system [7,8,9,10,11,12,13,14,15]. It has been shown that the central tenets of thermodynamics can be derived directly (and more naturally) using the MaxEnt principle without recourse to any other laws [7,8,9]. The Shannon entropy is itself a subset of the Kullback-Leibler directed divergence, cross-entropy or relative entropy 2 function [10,14,16,17,18], in discrete form: where q i and p i are respectively the prior and posterior probability of occurrence of the ith result, p = {p i }, q = {q i } and the solidus | is the Bayesian "subject to". In the minimum cross-entropy principle ("MinXEnt") -a superset of MaxEnt - (5) is minimized subject to the constraints on a system, to give the distribution p * which contains the least informa- tion, yet is closest to q [14]. Collectively, definitions (1)-(5) underpin present-day statistical physics, thermodynamics, information theory and encoding, optimization and data analysis, whilst the MaxEnt and MinXEnt methods have found widespread application to a vast assortment of fields, including information technology, communications, mathematics, science, engineering, economics, decision theory, geography, linguistics and the social sciences [e.g. 9,13,14,19,20].
Attention must, however, be directed towards two points. Firstly, although the MaxEnt and MinXEnt methods are widely considered to fall within the scope of Bayesian inferential reasoning [e.g. 21,22], no such Bayesian derivation appears to be evident in the literature, and the two methods remain largely underpinned by axiomatic arguments [23,24]. Since these arguments can be varied and are subject to debate, the circumstances in which the MaxEnt and MinXEnt methods remain valid, or require modification, are not clear. Secondly, there is now widespread controversy in many of the above-mentioned fields -especially in statistical physics -due to the promulgation of a variety of alternative entropy functions, inconsistent with the above definitions, for specific applications. These include the Fisher information measure [25,26]; the Bose-Einstein and Fermi-Dirac entropies of quantum mechanics [27,28,29,30,31,32,33]; the Rényi [34], Tsallis [35,36], Sharma-Mittal [37,38] 2 The relative entropy is usually defined as the negative of (5); a few authors define the cross-entropy differently.
and other entropies of non-extensive (correlated) statistics; Beck-Cohen superstatistics [39]; the Kaniadakis entropy of relativity theory [40,41]; the "exact" entropies of the author [42,43], and many others [e.g. 20,44,45,46,47,48,49,50,51]. This discussion highlights the fact that the entropy concept has many different philosophical bases. In addition to (i) the combinatorial basis of entropy, other bases include: (ii) The information-theoretic basis [6,52,53,54,55,56,57,58], in which entropy is defined in terms of the number of bits of information needed to describe a particular system, and/or in terms of coding theory; (iii) The axiomatic basis [6], in which the desired properties of an entropy measure -its axioms -are listed and used for its derivation; (iv) The inverse modelling approach of Kapur and Kesavan [13,14,59,60,61,62], in which one works backwards from an observed probability distribution p * , a priori distribution q (if available) and any constraints, to derive the measure of cross-entropy or entropy applicable to a system; (v) The game-theoretic basis [63,64,65,66,67,68,69], in which an entropy function is derived by analysis of a game between two or more players; and (vi) The information-geometric or statistical manifold basis [70,71,72,73], in which an information measure is analysed using a geometric representation.
Bases (ii) and (iii) are popular in information theory, (v) in economics, business and military strategy, (vi) in statistics, probability theory and mathematics, whilst (iv) is less well known.
Whilst each basis has a following in its own discipline, the relationships between the different bases are still largely unexplored. Furthermore, whether any basis can claim supremacy over the other bases, or whether they are of equal philosophical standing, is a question which has not been adequately addressed.
The aim of this article -which follows two previous studies [42,43] -is to critically examine the philosophical bases of the entropy and cross-entropy concepts, with particular attention to the information-theoretic, axiomatic and combinatorial interpretations. Using the combinatorial basis, it is shown (following a well-trodden road) that both the cross-entropy and entropy functions are simplified forms of the logarithm of the multinomial distribution; they are therefore only shorthand functions to determine the "most probable" (MaxEnt or MinX-Ent) realization of a system which follows the multinomial distribution, without the necessity of invoking this distribution itself. The Kullback-Leibler cross-entropy and Shannon entropy functions are therefore secondary concepts, based firmly on simple combinatorial principles.
This perspective lies in stark contrast to the axiomatic and information-theoretic bases, both of which take the cross-entropy or (especially) the entropy function as the fundamental concept and starting point for analysis. Since it rests upon a more definitive philosophical foundation, the combinatorial basis is the most fundamental (most primitive) of these three bases. It is also of much broader scope, leading naturally to new, generalized combinatorial definitions of cross-entropy and entropy -each a superset of the Boltzmann principle (1)  The revised definitions underpin the development of a new, broad discipline of combinatorial information theory, spanning the entirety of present-day statistical physics, information theory and probability theory, for the analysis of probabilistic systems of every type.
After early drafts of this work were completed [74], the author's attention was alerted to several works of Grendar and Grendar [75,76,77,78,79], who adopt a similar philosophical argument, albeit with somewhat different aims and a different scope. In fact, the central premise of this study has been known since the time of Boltzmann [3], played a critical role in the discovery of Bose-Einstein and Fermi-Dirac (quantum) statistics [27,28,29,30,31] This work is organised as follows. In §II A-II C, the main elements of the informationtheoretic, axiomatic and combinatorial bases of entropy and cross-entropy are critically examined, leading to combinatorial derivations of the Shannon and Kullback-Leibler measures, which reveal their purpose to determine the "most probable" (modal) probability distribution of a multinomial system. Several technical aspects are then scrutinized in detail: zero reference states for entropy or information; ensemble theory and multicomponent systems; and the "generic" formulation of statistical mechanics developed by Jaynes [e.g. 7,8,9,10,11,12,13,14,15]. The latter is reinterpreted and extended for a multinomial system in light of the combinatorial approach, with the derivation of new concepts including a generalized Clausius inequality, a generalized free energy ("free information") function, a generalized Gibbs-Duhem relation and phase rule, and a reappraisal of fluctuation theory and Jaynes' entropy concentration theorem. In §III, the significance of the multinomial distribution is reviewed, leading to the proposition of generalized definitions of entropy and cross-entropy for non-multinomial systems. A connection to Bayesian statistical inference, and the other bases of entropy, are also discussed.
In the following, an entity is taken to be any discrete particle, object or agent within a system, which acts separately but not necessarily independently of the other entities present (note this definition encompasses human beings). The entity therefore constitutes the unit of analysis of the system, although of course some entities can be further examined in terms of their constituent sub-entities, if desired.

II. THEORETICAL ROOTS OF THE INFORMATION ENTROPY CONCEPT
What is entropy? This question has certainly occupied (or been dismissed from) the minds of millions of college and university students for one and a half centuries -predominantly in physics, chemistry, engineering and informatics -and undoubtedly tens of thousands more of their professional elders in all disciplines.
To endeavour to answer this question, in this section the first three theoretical or philosophical roots of the entropy and cross-entropy concepts listed in §I are examined. The first two, information-theoretic and axiomatic, are so closely intertwined in the literature that it is not possible to distinguish them clearly. The third origin, based on combinatorial analysis, is somewhat distinct, and occupies much of this work. Discussion of the remaining three bases of entropy (inverse modelling, game-theoretic and information-geometric) is postponed until later in the text ( §III C). A rival approach to the analysis of probabilistic systems, which invokes the continuous Fisher information [25,26,94], is examined in detail elsewhere [95].
A. The Information-Theoretic (Bits) Approach The first theoretical basis of the Shannon entropy -although not the first in historical development -concerns the number of bits of information required to specify a particular system or outcome [6,52,53,54,55,56,57,58]. Consider the binary entropy or B-entropy: related to the Shannon entropy (defined using the natural logarithm, (4)) by H = B ln 2.
Now consider a random variable which may take one of two states, of equal probability Initially, the state of the variable is not known. After a binary decision (a process of selection or measurement) it is found to be in one of these states (say p 1 = 1) and not the other (p 2 = 0). The initial and final binary entropies are therefore: (Here and subsequently, we take 0 log 0 = log 0 0 = log 1 = 0 for all logarithmic bases). The change in entropy is then: If we define the change in information as the negative of the change in entropy (i.e., entropy lost = information gained) [53,54,55,83,96,97], the gain in information -reflecting our improved state of knowledge -is: Thus for a simple binary decision, the information gained (entropy lost) corresponds to one bit of information. The decrease in entropy therefore provides a quantitative measure of the information gained by observation of a system.
If we adopt a scaled binary entropy S B = −k s i=1 p i log 2 p i , the information gained by a binary decision is k, measured in the units of k. For a scaled entropy based on the natural logarithm, S = −k s i=1 p i ln p i , the gain in information is k ln 2 [6,52]. For thermodynamic systems for which k is the Boltzmann constant, 1 bit of information corresponds to an energy transfer of 9.57 × 10 −24 J K −1 entity −1 . To access information carried by photons, and distinguish them from the background (thermal) radiation, it is necessary to account for the effect of temperature [54,55]; in this case, 1 bit of information corresponds to kT ln 2 energy units per entity.
A second variant of the information-theoretic definition -which overlaps with the axiomatic approach ( §II B) -is to consider a random variable which may take s equally probable states. We define a measure of uncertainty as [9,98]: As the states are equally probable, s = 1/p i , ∀i, hence U = − ln p i . The mathematical expectation of the uncertainty is U = − s i=1 p i ln p i = H, i.e. the Shannon entropy. As the states are equally probable, this reduces to U = U.
For states which are not equally probable, we may thus adopt the Shannon entropy as a measure of the expectation of the uncertainty [6]. We can further define the surprisal or self-information associated with each result [8,9,45]: The entropy is therefore the expectation of the surprisal.
The surprisal has also been defined relative to the prior probability of that result, δ i = ln(p i /q i ), i.e. as the amount of information gained by a decision or message [9,16,45].
This is better referred to as the cross-surprisal. The expectation of the cross-surprisal gives the cross-entropy (5). The cross-entropy is therefore a measure of the expected information relative to what is known. Another useful term is the function To obtain an instantaneous or readily decipherable code, it is common practice to seek a prefix-free code, in which no codeword is a prefix of any other; the codeword lengths are then subject to the Kraft inequality [58]: Minimization of (12) with respect to κ i subject to (13) by the Lagrangian method (see §II C 2), with normalization ( s i=1 p i = 1), yields a discontinuous binary entropy: where ⌈x⌉ is the ceiling of x (the smallest integer greater than or equal to x), which arises since κ i must be an integer. By repeated m-fold sampling of (14), the two entropies converge: The entropy B therefore indicates the minimum mean (possibly fractional) number of bits per symbol, whilst B 0 is the equivalent quantity based on integer codeword lengths.
The above three information-theoretic roots of the Shannon entropy are of tremendous utility, primarily to information theory and coding applications. However, the first two variants suffer from the deficiency that they assume that measures of information (or entropy) should be of logarithmic form, an assumption in part derived from the axiomatic approach ( §II B). Certainly, other functions could yield one bit of information for a binary decision (9). The third variant assumes that the mean code length is the appropriate quantity to be minimized; this is reasonable for coding applications, but does not necessarily apply to other situations. Furthermore, the Kraft inequality -which gives rise to the logarithm in the binary entropy -is not universal in application (e.g. to fixed-length codes, codes incorporating redundancy, etc), and warrants further examination. In consequence, the information-theoretic definitions of entropy and cross-entropy have a narrow philosophical basis, which does not necessarily apply outside their domain of application.

B. The Axiomatic Approach
The second theoretical basis of the entropy concept, developed by Shannon [6], proceeds by listing the desired properties of a measure of uncertainty -its axioms or desiderataand finding the mathematical function which satisfies these axioms. Shannon [6] considered three axioms: continuity, monotonicity and recursivity (the branching principle), from which the Shannon entropy (4) is uniquely obtained. To Shannon's original list, many additional axioms have been added: e.g. uniqueness, permutational symmetry (invariance), non-negativity, non-impossibility, inclusivity, decisivity, concavity, maximum entropy at uniformity (normality), additivity, strong additivity, subadditivity, system independence and subset independence [e.g. 6,9,14,20,23,44,47,50,101]. The Shannon entropy is the only function which satisfies these axioms. Indeed, it may be deduced from several small subsets of these axioms, implying that they are not independent [e.g. 14, 21,47].
It must be noted that the definition of thermodynamic entropy (3) by Planck [5, §118] is derived by an axiomatic argument, assuming multiplicity of the weights and additivity of the entropy function. Similarly, in the "plausible reasoning" treatises of Cox [102, p37] and Jaynes [15, §2.1], the Shannon entropy (4) is obtained axiomatically, assuming entropy is additive and multiply differentiable.
The cross-entropy or directed divergence function D can also be obtained using the axiomatic approach [14,16,17,23]. Its governing axioms are broadly similar to those for the Shannon entropy, except that it is convex, and the equilibrium distribution p * = q in the absence of other constraints [14]. Both the MaxEnt and MinXEnt principles themselves have also been justified axiomatically [e.g. 23,24].
Whilst mathematically sound and of tremendous utility, the axiomatic approach is intellectually unsatisfying in that it presents an austere, sterile basis for the entropy and may be more appropriate for particular or more general circumstances. To resolve these circular arguments, we now turn to consideration of the combinatorial basis of the entropy and cross-entropy functions, which as will be shown, should be recognized as their primary (most primitive) philosophical basis.
C. The Combinatorial (Statistical Mechanical) Approach

Statistics of Multinomial Systems
The combinatorial approach was first developed in statistical thermodynamics, to examine the distribution of molecules amongst energy levels or phase space elements [e.g.  distribution [104,105,106]: where again q i is the prior probability of a ball falling in the ith box, and n = {n i }. If the prior distribution q is equated to the uniform distribution u (i.e. q i = u = 1/s, ∀i) this reduces to: Since the total number of configurations of a multinomial distribution is s N [107], the number of ways in which any particular realization in (17) can be produced, or its statistical weight, is [109,110]: For constant N, the above equations are subject to the natural constraint: and usually one or several moment constraints [c.f. 7]: C1 to CR : where f ri is the value of the function f r in the ith state and f r is the mathematical expectation of f ri . An example of (20) is an energy constraint, in which each state is of energy f 1i = ε i and the expectation of the energy is f 1 = ε .
Now consider a sequence of v independent and identically distributed (i.i.d.) probabilistic events, within each of which w trials or selections are made between s distinguishable states, as represented in Figure 1b. Examples include tosses of a coin or coins, throws of a die or dice, spins of a roulette wheel, choices of symbols to make up a communications signal, or the sexual liaisons of leading film star. So long as we are only interested in the statistical nature of the selections, and not their order, the probability of any realization or type (without regard to order, assuming each event is independent) also follows the multinomial distribution (16) with N = vw. When only one selection is made in each event (i.e. w = 1), then N = v. When the prior probabilities q i of each state within each selection are identical, the weight also follows (18).

The Most Probable Realization
We now use first combinatorial principles to determine the most probable realization of the multinomial systems considered. As mentioned, the following derivation is common in statistical thermodynamics [e.g. 3,4,5,32,33,82,83,84,85,86,87,88,89], although such workers base their derivations on the weight W. As it is based on P rather than W, the following derivation incorporates the prior distribution q, and is therefore more comprehensive [91].
Clearly, the most probable realization is that for which P (16) is a maximum, subject to the constraints C0-CR on the system ( (19), (20)). As the natural logarithm ln x increases monotonically with x, but transforms a product into a sum, it is convenient -and equivalentto maximize ln P rather than P itself, a convention adopted (implicitly) throughout statistical mechanics [3,89]. (The use of logarithms is therefore merely a matter of convenience, not a requirement.) The most probable realization is given by: where d( ) is the total derivative or variational operator. Now (21) can be constructed using Lagrange's method of undetermined multipliers [9,14,89], involving extremization of the Lagrangian L: From the multinomial distribution (16): in which (for reasons which will be explained) the leading ln N! term is brought inside the summation using the natural constraint (19). From the constraints (19)- (20), the Lagrangian is: where λ r , r = 0, ..., R, are the Lagrangian multipliers, and λ 0 −1 is chosen rather than λ 0 for mathematical convenience. For constant N, q i and f r , and for f ri independent of n i , we need only consider the variation of (24) with respect to n i , i.e. ∂L/∂n i dn i = 0, ∀i, whence: The above equations are expressed in terms of n i , and can be said to be in "n i form." At this stage the near-universal approach taken in the literature (see previous statistical mechanics references) is to employ a truncated form of the approximation for factorials derived by Stirling [92] and de Moivre [93]: This is accurate to within 1% of ln x! for x > 90; a more precise form, ln , is accurate to within 1% of ln x! for x > 4 [105]. (Strictly speaking, the limit must be taken using upper and lower bounds, as treated in large deviations theory [80,81], but for the purpose of the present study, the Stirling approximation yields the same result).
Thus ∂ ln n i !/∂n i ≈ ln n i and ln N! ≈ N ln N − N, and so the most probable realization, here designated with an asterisk, is obtained from (25) in conjunction with C0 (19) as [c.f. 7,8,9,14,20]: or where p i is the proportion or probability of entities in each state i. Equations (27)-(28) can be termed the generalized Maxwell-Boltzmann distribution, whilst Z q is the generalized partition function and λ 0 = ln Z q is the generalized Massieu function (strictly speaking, its negative [9,111]). The Lagrangian multipliers are obtained from the constraints Cr (20) and/or more readily from moment calculations (see §II C 6).
If q = u, (28) reduces to: This is the more commonly reported, generalized Maxwell-Boltzmann distribution of statistical thermodynamics and information theory, and Z is the usual generalized partition function [7,9,14]. Eq. (30) is obtained directly if either ln P u (17) or ln W (18) is used in the Lagrangian (24) instead of ln P.
In the information literature, it is customary to cast the analysis in terms of p i rather than n i , thus in "p i form" [7,8,9,14]. The constraints are: C0 : hence the Lagrangian (24) is: where µ r , r = 0, ..., R, are the new Lagrangian multipliers, and (µ 0 − N) is used for convenience. Taking the variation and applying the Stirling approximation gives: with This is identical to (28)- (29), with λ r = µ r /N, r = 0, ..., R and Z ′ q = Z q . The Lagrangian multipliers are again obtained from the constraints (32).
It is worth commenting that if the leading ln N! term is not brought inside the summation in (23), but discarded -the approach of all previous workers -the resulting distribution p * i contains an additional dependence on N −1 , which cancels out when forming the partition function Z ′′ q = Ne λ 0 . It therefore has no effect on traditional statistical mechanics. The distinction is, however, important in the development of exact (finite-N) statistical mechanics, which does not use the Stirling approximation [42,43]. Furthermore, by virtue of the natural constraint (19), the n i form of a problem contains knowledge of N, and so it is unnecessary to include N as a separate parameter. In contrast, a problem specified in its p i form does not contain -of itself -any information about N; if needed, this must be specified separately.
From the foregoing it is clear that the "most probable" probability distribution for a multinomial system, subject to arbitrary moment constraints, can be obtained without making use of an entropy or cross-entropy function. One can instead analyse a probabilistic system directly using first combinatorial principles. This aspect of entropy theory is not clearly spelt out in the information theory literature, with only a few exceptions [e.g. 10,11,15,18,90,91]. The direct combinatorial approach is extended further in §III B, to encompass systems not of multinomial character.

Definition of the Cross-Entropy (Directed Divergence) and Entropy
Where do the cross-entropy and entropy functions come into the above analyses? Clearly, they are merely convenient mathematical tools to enable construction of the Lagrangian equation in p i form (33). In fact we can define the cross-entropy as "that function which, when inserted into the Lagrangian in place of ln P, and the extremum of the Lagrangian is obtained, yields the most probable distribution of the system". The entropy may be similarly defined as "that function which, when inserted into the Lagrangian in place of ln P u (or ln W), and the extremum of the Lagrangian is obtained, yields the most probable distribution of the system." Consider ln P, expressed in p i form: whence from the Stirling approximation (26) [91]: Thus the cross-entropy or directed divergence D (5) is simply the negative of the logarithm of the governing probability distribution, expressed per number of entities present [91].
Maximizing ln P for a multinomial system subject to the Stirling limits is therefore equivalent to maximizing −D, or minimizing D, subject to the constraints on the system. (It does not matter whether we adopt a positive function, whose minimum yields the most probable realization, or its negative, whose maximum also yields this realization. By convention, the cross-entropy is taken here as a positive function to be minimized, although this choice is arbitrary.) Similarly if we consider ln P u , from (31) and (37) the Stirling form is [3,10,91]: This is proportional to the Shannon entropy (4), shifted by a constant. Maximizing ln P u subject to the Stirling limits and constraints is therefore equivalent to maximizing H, subject to the same constraints [91]. Indeed, from (18): This definition of entropy for a multinomial system accords with the probabilistic expressions of Boltzmann and Shannon (4).
It is therefore seen that the Kullback-Leibler cross-entropy and Shannon entropy functions are simplified forms of the logarithm of the multinomial distribution (16), expressed per unit entity. The MinXEnt and MaxEnt principles therefore provide simplified methods to determine the most probable realization of a multinomial system, subject to its constraints (succinctly termed "MaxProb" by Grendar and Grendar [76]). The cross-entropy is the more generic of the two functions, in that it contains the prior probabilities q i .
Of the three theoretical roots of the entropy and cross-entropy functions, the combinatorial approach is therefore the most intellectually satisfying in that it provides a direct answer to the question: what is entropy? There is no circular argument: entropy and cross-entropy are firmly based on simple combinatorial principles. In consequence, there is no need to imbue either the MinXEnt or MaxEnt principles, or the cross-entropy or entropy functions themselves, with the kind of mystique with which they have been associated for well over a century. There is no mystery at all. In later sections, the foregoing analysis is generalized to any probabilistic system, irrespective of whether it is of multinomial character.

Equivalence of Reference States
It is necessary to be extremely careful about the definitions of the cross-entropy and entropy functions, given in §II C 3. To this end, note that obtaining the extremum of the Lagrangian ((24) or (33)) necessitates extremization, whether it contains ln P or its substitute, −D (or whether ln W or H, if q = u). The relationship between these quantities is therefore: Correspondence between the ith terms of D and ln P gives: where Integration with respect to p i and summation gives: where C is a constant of integration. In consequence, the multinomial cross-entropy (5) and entropy (4) could have been given respectively as (or any multiple of): However, the axiomatic definitions of these functions require that they obey the decisivity producing the recognized forms of the above functions ( (5), (4)). This causes the N ln s term to be dropped from the definition of H (38). Note, however, that the choice of C has no impact on the application of D or H to determine the most probable realization.
(In other words, as is recognized throughout science and engineering, all zero reference or datum positions for the cross-entropy and entropy -and hence for information and energyare mathematically equivalent.)

Ensemble Theory and Multicomponent Systems
In its application to thermodynamics, one aspect of statistical mechanics has caused needless conceptual difficulty: the use of ensembles to represent particular types of systems [112].
Most common are the microcanonical ensemble, representing a closed system of fixed energy; the canonical ensemble, a closed system of fixed temperature; and the grand canonical ensemble, an open system of fixed temperature and mean composition. From the foregoing discussion, it is evident that an ensemble is simply the set of all possible realizations -each weighted by its number of permutations (or for unequal q i , by the probability of each realization) -consistent with a particular system specification; i.e. consistent with a specified governing probability distribution P, total number of entities N (or numbers of entities of different types), number of states s, and specified constraints f r or their equivalent Lagrangian multipliers λ r , r = 1, ..., R. An ensemble is therefore a mental construct, which does not require a physical manifestation.
As an example, consider a closed physical system in which the entities fluctuate between states (the elemental chaos of Planck [5]), such as gas molecules in a container. Such a system will migrate from one realization to another, and thence between different members of its ensemble (it will describe a trajectory in -for example -energetic, geometric or phase space).
However, there is no need to require that the system must access every realization within a particular time frame, nor even that it should come arbitrary close to every realization; the only requirement of probability theory is that each realization included in the ensemble be realizable, to the extent given by its assigned probability. As every gambler or insurance broker will testify, probabilities are not certainties. Unfortunately, a great deal of erroneous reasoning has been put forth on this topic, which still clouds our present-day understanding.
In contrast, consider a "multiple selection" system as defined in §II C 1, such as a set of throws of a coin or rolls of a die. In this case, the ensemble can only ever be a mental can be formed by reaction [115].) Since the system is open, each N l is permitted to vary between zero and (effectively) infinity. Expressed in terms of P rather than W, the governing distribution is generally assumed to be "multiply multinomial" [c.f. 84,86,87,88]: where q {N l },i is the prior probability of a system which contains N l particles of each lth type in the ith state. This is normally subject to natural, moment and mean number of each type of entity constraints: Cr : Cl : The combinatorial method ((42) and §II C 2) gives the Stirling-approximate cross-entropy and equilibrium distribution: with where p {N l },i = n {N l },i /N; λ r and ν l are Lagrangian multipliers; and Ξ q is the generalized grand partition function. The entropy forms follow. However, the cross-entropy will only be of Kullback-Leibler form if the governing distribution is multinomial (45). If P is of some other form, for example the product of independent distributions (extending [33]): or if we possess some other knowledge (such as of N l ), then clearly the resulting multicomponent cross-entropy and entropy functions and the equilibrium distribution could be quite different. It is insufficient to simply assert (45) or (49); its adoption must be based on sound reasoning, and ultimately, be demonstrated by successful predictions.
The generic structure has the powerful advantage of being applicable to any multinomial system, irrespective of the physical nature of the constraints, and is therefore not limited to energetic systems. It can however be even further extended. The following discussion is a synthesis and extension of previous treatments, building up to the development of major new concepts including a generalized Clausius inequality, a generalized free energy function, and a generalized Gibbs-Duhem relation and phase rule. Throughout the following (except where specified), λ 0 = ln Z q is assumed to be a function of each λ r ; the λ r are mutually independent; each f ri is independent of λ r ; and each f r is a function of λ r but not of the other multipliers λ m , m = r.
Using the Cauchy-Schwartz inequality a 2 b 2 − ab 2 ≥ 0 [116] with a = f r , b = 1 gives var(f r ) = −∂ f r /∂λ r ≥ 0, whence ∂ f r /∂λ r ≤ 0 [13]. Accordingly, λ r decreases monotonically with increasing f r . No equivalent relation is available for the mixed derivatives It is also possible to consider λ 0 and each f ri (hence also f r ) to be functions of parameters .., V . By differentiation of the partition function (29) [7,10,15], or more directly by rearrangement of p * i ((27)-(28)) and differentiation: Alternatively, differentiation of (56) with respect to any continuous function α v yields (necessarily in the vicinity of equilibrium, e.g. for a shifting equilibrium position): Eq. (59) with α v = t = time is a statement of Onsager's [117,118] reciprocal relations.
Various other higher derivative equations in λ r and/or α v are given by Jaynes [15].
Similarly, considering λ 0 and λ r to be functions of β j , j = 1, ..., J; or λ 0 alone as a function of N, n * i or p * i , from (27)- (29): From (61), λ 0 (and thus Z q ) is independent of N in the Stirling limit N → ∞. From (62), ∂λ 0 /∂n * i → 0 in the Stirling limit n * i → ∞, hence λ 0 is independent of the mean degree of filling of each state.
Using p * i ((27)-(28)), the constraints ((19)- (20) or (31)-(32)), the definitions of H, D and P ((4)-(5), (42)) and the multiplier relations ((53)), the minimum cross-entropy or maximum entropy position is obtained as [c.f. 7,9,14]: with probability: where A is a normalising constant (with P * ≤ 1), and we recall that H * is obtained from ln P u by dropping the ln s term (or directly from ln W) ((38)- (39)). Equation (64) whilst differentiation with respect to λ r -now considering f r to be a function of λ m , ∀mand use of (56) gives the Euler relation [c.f. 120]: where M and R are numerically equal. From (66) The variation in D * or H * due to variations in λ 0 , λ r and f r (and also N) is [c.f. 7,9,10,15]: where we can interpret d f r = dU r , df r = s i=1 p i df ri = dW r and d f r − df r = s i=1 f ri dp i = dQ r respectively as changes in the rth type of "energy", "generalized work" on the system and "generalized heat" delivered to the system, whence (as defined here) dU r = dQ r + dW r . Note that in the above derivation, the variations in λ r cancel out [10,15], hence (69) encompasses conditions of either constant or variable λ r . Equation (69) is a superset of the Clausius relation (1), and so for each type of "generalized heat" there exists a conjugate integrating factor λ r . As with the Clausius relation, the λ r are properties of the system of interest (i.e. the one into which positive generalized heat is delivered).
Equation (69) applies to a reversible process, i.e. to an incremental change in the equilibrium position. If we also include spontaneous irreversible processes (involving a system not necessarily at equilibrium), for which the cross-entropy can decrease (or entropy can increase) without generalized heat input, we see that: This is a superset of the Clausius inequality (2). Equation (70) can be rearranged, in the manner of Gibbs [112,115], to give the differential form of a generic dimensionless free energy function Φ, here termed the free information 6 : (whence dΦ * = 0 at a fixed equilibrium position), where the upper form incorporates the prior probabilities q. Now from (64): so if we set dD = dD * +dD irrev and dH = dH * +dH irrev (with dD irrev ≤ 0 and dH irrev ≥ 0), where superscript irrev denotes the irreversible component, then from (71)- (72): If -and only if -there is no change in λ r (i.e. no change in any contacting bath; see also (77) below), no reversible generalized work on the system (apart from that already included in the constraints) and no irreversible process, then: where Z q is the applicable partition function ((29) or (30)). Alternatively, from (73), if there is no change in λ 0 or λ r and no irreversible process: Φ therefore indicates the maximum available weighted generalized work per entity which can be obtained from a system.
Integration of (71) gives the state function: where Q r = dQ r = d f r − dW r defines each absolute generalized heat 7 . Comparing its differential with (71) gives: This is a superset of the Gibbs-Duhem equation [115]. For a system containing separate coexistent phases, or bodies which differ in composition or state (as defined by Gibbs [115]), there will be one such equation for each phase. For L independent constituents, r = R − L other constraints (not including the L constituents) and p phases, (77) thus yields a generalized Gibbs' phase rule for the number of degrees of freedom of a system [c.f. 9, 85,115]: In other words, the system will be fully determined by R − p independent parameters, from the set of R constraints or (more commonly) their corresponding Lagrangian multipliers.
Equations (64), (71) and (73)-(78) form the basis of present-day thermodynamics. For energetic systems, dΦ is normally divided by the energetic multiplier λ 1 = 1/kT ; e.g. for an energetic system which can exchange heat with its surroundings, but not work or mass, at constant volume, dQ 1 = dU, dS = kdH, dA = kT dΦ = dU − T dS ≤ 0 and dA * = −kT d ln Z, where U is the mean internal energy per entity, A is the Helmholtz free energy per entity and Z is the microcanonical or canonical partition function 8 . For a grand canonical system with L independent constituents which can exchange heat and mass with its surroundings, but not work except for P V -work, at constant pressure, dQ 1 = dU, λ 1 = 1/kT , dQ 2 = dV , λ 2 = P/kT , dQ 2+l = dm l , λ 2+l = −µ l /kT = − ln α l , dG = kT dΦ = dU − T dS + P dV − l µ l dm l ≤ 0, dG * = −kT d ln Ξ and f = L + 2 − p, where P is pressure, V is mean volume per entity, µ l is the chemical potential and α l is the "absolute" (unscaled) chemical activity of the lth constituent, m l is the mean number of entities of lth type per entity, G is the Gibbs free energy per entity and Ξ is the grand canonical partition function. The essergy Y = kT 0 Φ = E − T 0 S + P 0 V − l µ l0 m l is a scaled Φ of a system with total internal energy E, in contact with a bath of reference temperature T 0 , pressure P 0 and chemical potentials {µ l0 } [121]. Essergy is thus an extended free energy calculated with reference to the bath (e.g. the external environment), not to the system.
The exergy X = Y − Y 0 is the difference between the essergy of a system (by early authors, with the chemical potential terms omitted), and of the same system in equilibrium with the bath [e.g. 121,122,123,124,125,126,127,128]. Exergy therefore represents the maximum work deliverable to the environment, by allowing a system to reach equilibrium with that environment. The statistical extropy [129,130,131] is a modified free information defined with respect to the bath -with all generalized work terms set to zero (i.e. Q r ≈ f r ) -less the modified free information at equilibrium. Exergy forms the nucleus of the interrelated fields of thermoeconomics and exergo-economics for resource management and process optimization [127,132,133], whilst both exergy and extropy have been used as measures of environmental impact, i.e. as quantitative tools within and/or complementary to the framework of environmental life cycle assessment [128,129,130,134,135].
Notwithstanding the historical development of this field, it must be emphasized that the use of Φ is not restricted to thermodynamic, industrial or environmental systems. Just as with the information entropy, we can define the free information of any multinomial system -for example in communications, transport, urban planning, biology, geography, social science, politics, economics, linguistics, image analysis or any other field -and use it to examine its (probabilistic) stability. The entire armoury of state functions, cyclic integrals, efficiency ratios, Gibbs-Duhem and phase relations, Maxwell-like relations and Jaynes relations -currently considered the exclusive domain of thermodynamics -can then be brought to bear to the analysis of such systems.

"Fluctuations" and Entropy Concentration Theorem
Although the MinXEnt or MaxEnt distribution is the "most probable" one, it cannot be a priori assumed to be the exclusive outcome. The sharpness of the predicted distribution has historically been examined by two methods: the fluctuation criterion of Gibbs [112] and Einstein [119], and the entropy concentration theorem of Jaynes [11,12,15,136], in part foreshadowed by Boltzmann [137] and Einstein [138]. The detailed asymptotic convergence behaviour of the distribution forms the subject of large deviations theory, based on various mathematical limit theorems [58,80,139], and will not be examined further here.
The first method examines the coefficient of variation δ of each constraining variable (or its square), commonly termed its "fluctuation" 9 . For a microcanonical system, this can be written as [c.f. 112,119]: where we are careful with notation to consider the variability about the total extensive quantity Nf r for a system of N entities, not the variability of the fixed quantity per entity f r . (Of course, δ does not capture the full picture of the distribution of N{f ri }, e.g. the skewness, kurtosis, etc, for which higher order moments must be considered.) The criterion for sharpness is normally stated as δ ≪ 1 [10,119]. From (54) and (79): The term inside the second square root is positive, and in many cases of order unity, whereupon δ(Nf r ) ≈ N −1/2 → 0 in the Stirling limit N → ∞. For example, for a microcanonical system with f 1i = ε i , f 1 = ε = U, λ 1 = 1/kT , containing an ideal monatomic noninteracting gas with U = 3 2 kT and C v = ∂U/∂T = 3 2 k, where C v is the isovolumetric heat capacity per entity, we obtain δ(Nε) = ( 3 2 N) −1/2 ≈ N −1/2 → 0 [e.g. 32, 33, 83, 84, 85, 86, 88] 10 . Although this result is not general (e.g. in the vicinity of phase changes [88]) it applies to many physical phenomena, producing what is widely regarded as the overwhelming precision of thermodynamics. If valid, the "N −1/2 rule" applies only as N → ∞; at very small N, a second effect must also be considered.
For the canonical and other ensembles, the variability of the (superset) {f ri } within each ensemble member is examined by (see above references): whence from (53)- (54) and (74): Whether or not this vanishes as N → ∞ depends on the physical variable r and the importance of interactions [32,33,84,103, c.f. previous footnote]. The variability of {f ri } for the total ensemble can be examined using δ(Nf r ), where N is the number of ensemble members, giving a relation analogous to (80). It is commonly asserted that N → ∞ (e.g. [83]), a rather questionable assumption. If correct, the total ensemble will be heavily concentrated at its ensemble means f r , ∀r.
Jaynes' [11,12,136] entropy concentration theorem considers the relative importance of the equilibrium probability distribution p * = {p * i } and some other distribution p ′ = {p ′ i }. From (37) or (65), the ratio of the probability of occurrence of p * to that of p ′ is: where P * , P ′ are the governing probability distributions and D * , D ′ are the cross-entropies corresponding respectively to p * and p ′ . This was originally formulated as the ratio of the 10 All the listed authors consider δ(E) for a canonical ensemble, where E is the "mean of the total energies", but then take E = N ε = 3 2 N kT for N non-interacting particles -thus assuming the system is microcanonical -giving the same result. number of ways in which p * and p ′ can be realized [11,138]: where W * , W ′ are the weights and H * , H ′ are the entropies corresponding to p * and p ′ . As shown by Jaynes [11,12,136], for N → 1000 even a small difference in H gives an enormous ratio, revealing the combinatorial dominance of the maximum entropy position.
Assuming p * , p ′ satisfy the constraints ((31)-(32)), and taking the Stirling limits N → ∞ and n i → ∞, an analysis similar to Kapur & Kesavan [14,§2.4.6] yields: i.e. simply the directed divergence of p ′ from p * , from which q vanish (being incorporated into p * ). Eqs. (83)-(84) then give: If we now put p ′ i = p * i (1 + ε i ), take a series expansion of ln p ′ i about ε i = 0, and discard all polynomial terms higher than ε 2 i , it is shown by Kapur & Kesavan [14,§2.4.7] that (a quite different derivation is given by Jaynes [136]): where n ′ i = p ′ i N is the number of entities in state i due to p ′ ; n * i = p * i N is the expected number of entities in state i; and we recognize χ 2 as the chi-squared distribution of statistics [25,26,140,141]. In other words, we can determine the "goodness of fit" of a distribution p ′ -or of some function F (p) which generates p ′ -to a multinomial system, by comparing the calculated χ 2 to the table value χ 2 (ν, 1 − α), where ν = s − R − 1 is the number of degrees of freedom and α is the significance level (upper tail or rejection area) [136].
As is well known [141,142,143,144] and dramatically illustrated by Jaynes [15, chap 9], the χ 2 statistic is an unreliable test for goodness of fit, being highly (and erroneously) sensitive to the occurrence of unlikely events. There is no need to conduct the simplification of (87); instead, from (85): where η is the correct test statistic for the goodness of fit of p ′ or its generator F (p) to a multinomial system, subject to the Stirling limits (η is given by Hoel [142, §10.1]; and by Jaynes [15, §9.11.1] in the form ψ = 10η/ ln(10), using an obscure decibel notation.) The calculated η can be compared to the "table value" η(ν, 1−α); alternatively, two distributions p ′ and p ′′ can be ranked by comparing their corresponding η ′ and η ′′ . Eqs. (86) and (88) finally give: Lagrangian distributions (and many others) have been obtained from the Shannon entropy subject to various constraints [14,20]. Similarly, in continuous statistics, the uniform, normal (Gaussian), Laplace, generalized Cauchy, generalized logistic, generalized extreme value, exponential, Pareto, gamma, beta (of first or second kind), generalized Weibull, lognormal, Poisson, power-function and many new distributions, and various multivariate forms, can be obtained from the continuous form of the Shannon entropy subject to various constraints [14,20]. Many additional distributions can be obtained from the Kullback-Leibler crossentropy in discrete or continuous form, subject to various prior distributions and constraints [14]. All these functions therefore constitute particular examples of multinomial statistics, and collectively form the multinomial family of statistical distributions. The broad applicability of the multinomial distribution, produced by the (fascinating) isomorphism of many probabilistic problems -such as of the "balls-in-boxes" and "multiple selection" systems described in §II C 1 -is responsible for the wide utility of the Kullback-Leibler cross-entropy and Shannon entropy functions.

B. Non-Multinomial Statistics
Notwithstanding the success of multinomial statistics, it is important to emphasize that a number of statistical functions are incompatible with the Shannon entropy and/or Kullback-Leibler cross-entropy, and are therefore not of multinomial character. Several of these (e.g. Bose-Einstein, Fermi-Dirac, Rényi, Tsallis and Kaniadakis entropies) reduce to the Shannon entropy as a limiting case [32,33,34,35,36,40,41,91]; such systems may therefore be approximated by multinomial statistics only when these limiting conditions are attained.
More thorough analyses of non-multinomial statistics must be deferred to later studies; however, their importance is here noted.
From the preceding analysis, it is clear that the definition of entropy (3) promulgated by Boltzmann [3] and Planck [4,5] can be used irrespective of whether the distribution is of multinomial character. A more comprehensive version, in which P now represents the governing probability distribution of any type and not only the multinomial distribution, is given in (42). The corresponding entropy is: where C, C ′ and K are arbitrary constants. (Note that the Boltzmann [3] -Planck [4] formula (3) is often misleadingly quoted as S = k ln W; this is correct only if S refers to the total entropy of the system, not the entropy per unit entity.) Indeed, it is not necessary to where again C, C ′ and κ are arbitrary, whilst "..." allows for other parameters or prior information. In many cases P will be a product-like function of s local probability distributions h i (p i , q i , N, ...); the appropriate choice of φ is the logarithm-like operator which transforms P neatly into a sum of terms in φ(h i (p i , q i , N, ...)), simplifying its extremization 11 (for parallel discussions of deformed logarithms, see [149,150]). Similarly, it may be convenient to choose φ and κ which define an entropy function with a "nice" asymptotic limiting form, in the sense of large deviations theory [80,81]. Clearly, the information entropy (4) given by Shannon [6] -although derived from sound axiomatic postulates, and of quite broad scope -is strictly valid only for multinomial systems subject to the Stirling approximation. This may be appropriate for communication signals of infinite length, but is surely insufficient to underpin the vast field of information theory in general.

C. Further Discussion
In his many works, Jaynes expounds the "Bayesian" or "subjective" view of probabilities, which represent assignments of one's belief based on the available information, and argues against the "frequentist" view in which probabilities are interpreted strictly as frequency assignments [7,11,151,152]. Separately, Jaynes demonstrates the equivalence of MaxEnt based on the Shannon entropy, and combinatorial analysis using the multinomial weight (the so-called Wallis derivation) [10,11]. At this point, however, he considers the combinatorial approach to represent a frequency interpretation, stating [11,15]: "the probability distribution which maximizes the entropy is numerically identical with the frequency distribution which can be realized in the greatest number of ways" [his emphasis]. This identification of the combinatorial approach with the frequentist view is unfortunate; in fact, by applying MaxEnt based on the Shannon entropy, one assumes (implicitly) that the phenomenon being examined follows the multinomial distribution, and one uses one's prior knowledge to infer (hypothesize) the available states i (for a parallel discussion, see Bhandari [153]) 12 . The calculated probability distribution p * i is therefore valid only in the "subjective" sense (i.e. exists only as an inference of the observer) until verified by experiment. Even if so "verified", there will always be room for doubt over its validity. 11 The recent derivation of the Tsallis [35] entropy by Suyari and co-workers [145,146,147,148] using a transformation of the form φ = ln 2−q (W 2−q ), where ln q is the q-logarithmic function and W q is a q-multinomial coefficient, provides a fascinating example of an alternative transformation function. 12 Jaynes appears to reach essentially this viewpoint in his final work [15, chaps. 9, 11; especially §9.5-9.6,

11.4].
Indeed, the calculated MinXEnt probability (e.g. (28)-(29)) can be expressed in a reversed form of Bayes' theorem: where P is a probability, P * is a most probable (modal) probability, i is the ith distinguishable outcome (datum) within a set of s such outcomes, M is the ith manifestation of the hypothesised model P, not necessarily of multinomial form, and I is the prior information.
For the problems considered here, I includes the constraints, any approximation or limit assumptions (e.g. the Stirling approximation) and any other relevant prior knowledge; if desired, these can be itemised separately. We immediately recognise the denominator in (94) as the partition function Z q (29) or its equivalent, whilst P (i|I) = P * (i|I) = q i is the prior probability. The generalized MinXEnt or MaxEnt methods therefore provide a method, in the absence of any sampling data, to "bootstrap" a sampling distribution {P * (i|M, I)} (we could call it the posterior pre-sampling distribution) from some hypothesis distribution {P * (M|i, I)} and prior distribution q. The latter two distributions are necessarily embedded within the governing distribution P, being obtainable from it by extremization of (91) or (92) subject to I 13 .
In consequence, the generalized definitions of cross-entropy and entropy given here ((91)-(92)) fit seamlessly into a Bayesian inferential framework [c.f. 21,102,152]. In such cases, q represents a "Bayesian prior distribution", "Jeffrey's uninformative prior" [154,155] or "Jaynes' measure distribution" [10,11], whilst P represents one's postulated understanding of the probabilistic structure of the phenomenon at hand. The broader Jaynesian program of maximum entropy analysis as a method of statistical inference is therefore untouched (in fact, enhanced) by the present analysis.

Now considering the other bases of entropy listed in §I:
(a) In the inverse modelling basis developed by Kapur, Kesevan and co-workers [13,14,59], one works backwards from a hypothesized or observed probability distribution (p), prior distribution (q) and constraints (C0-CR), to obtain the measure of cross-entropy or entropy applicable to the process. Using (91) or (92), such inverse methods could 13 A quite different connection between Bayesian and combinatorial perspectives was given recently [78].
then be used to determine the governing probability distribution P of the process.
In effect, this approach was adopted in statistical mechanics by Planck [4], Einstein [28,29,156] and Bose [27], in which measurements of the thermodynamic entropy were used to determine the statistical weight. Alternatively, one can work "sideways" from the observed, prior and governing distributions (p, q and P) to determine the constraints [c.f. 60,61,62]. Such methods offer powerful extensions to existing theory, yet have barely been examined in the literature.
(b) In the game-theoretic basis founded by von Neumann and Morgenstern [63], a decision tree or matrix is built up by analysis of a game between two or more players. The statistical structure of the game and optimal playing strategies are then determined.
Originally devised for gambling and economics, this basis has found diverse application to political science, biological evolution, military strategy, counter-terrorism, queuing theory, operations research, system dynamics and many other fields. In many game-playing scenarios, there exists a game-theoretic equilibrium (in economics: "Nash equilibrium") at which no player can benefit by changing his/her strategy [63,64]; such equilibrium concepts have more recently been related to information measures, cod- (c) In the information-geometric or statistical manifold basis, a statistical or probabilistic model is represented as a geometric structure (manifold) in some mathematical space, e.g. the space of probabilities, the space of population parameters or the space of possible distributions for the system [70,71,72,73]. This structure can then be analysed geometrically, using measures of distance (metrics), shape and connectivity (topology), tangency and differentiability. As an example, a cross-entropy or divergence measure can be interpreted as the probabilistic distance of distribution p from q [16,17,45,70]. Information-geometric arguments have recently been applied to alternative entropy concepts [e.g. 157], but much more work is required in this field.
Since the information-geometric basis serves as a representation of a system, rather than a cause, it is subsidiary to the other bases; however, it provides a valuable tool for the analysis of combinatorial concepts and distributions, and their connections to other physical theory (e.g. general relativity).
The analysis to this point has followed a long path, only to arrive more or less at its starting point: the combinatorial entropy of Boltzmann and Planck (although the idea is taken somewhat further than they had imagined). The fact that this discussion is still necessary in the 21st century reflects the great gulf between present-day statistical mechanics and thermodynamics -still taught much as they were 50 or even 100 years ago -and the more recent but surprisingly narrow field of information theory initiated by Shannon [6]. The gulf persists despite the efforts of Bose, Einstein, Fermi and Dirac, amongst others, in statistical mechanics, and of Jaynes, Tribus, Kapur, Kesavan and many others in information theory and maximum entropy methods. The two fields are, in fact, one. Appreciation of this fact (by both sides) would permit the development of a much broader discipline of "combinatorial information theory" than at present, applicable to many different types of problems.
In our search for meaning in the universe and our existence, the sentiment of John Wheeler is frequently quoted: "it from bit" [158]; i.e. the entirety of existence (it) arises from information-theoretic principles (bit). This statement implies a universe built up by observer-participancy, individual measurement by measurement. However, from the present analysis [see also 42,43], each combinatorial family has a different kind of "bit", producing a vast array of different observational frameworks and observer-dependent (subjective?) realities. This can be paraphrased as "both it and bit from prob.", i.e. both existence and information theory arise from raw probabilistic constructs. This principle clearly underpins the probabilistic definition of the second law of thermodynamics, "a system tends towards its most probable form", and may well explain the peculiarities of quantum mechanics [27,28,29,30,31,32,33,42,43].

IV. CONCLUSIONS
The philosophical bases of the entropy and cross-entropy concepts are critically examined, with particular attention to the information-theoretic, axiomatic and combinatorial interpretations. It is shown that the combinatorial basis, as first promulgated by Boltzmann and Planck, is the most fundamental (most primitive) of these three bases. Not only does it provide (i) a derivation of the Kullback-Leibler cross-entropy and Shannon entropy functions, as simplified forms of the multinomial distribution subject to the Stirling approximation; the combinatorial approach also yields (ii) an explanation for the need to maximize entropy (or minimize cross-entropy) to find the most probable realization; and (iii) generalized definitions of entropy and cross-entropy for systems which do not satisfy the multinomial distribution, i.e. which fall outside the domain of the Kullback-Leibler and Shannon measures. The information-theoretic and axiomatic bases of cross-entropy and entropy -whilst of tremendous importance and utility -are therefore seen as secondary viewpoints, which lack the breadth of the combinatorial approach. The view of Shannon, Jaynes and their followers -in which the Shannon entropy or Kullback-Leibler cross-entropy is taken as the starting point and universal tool for analysis -is not seen as incorrect, but simply incomplete.
On the other hand, the viewpoint of many scientists -who consider statistical mechanics to be a branch of classical mechanics or quantum physics, rather than of statistical inferenceis also incomplete. A more detailed understanding of the combinatorial basis will enable development of a powerful body of "combinatorial information theory", as a tool for statistical inference in all fields.
The generic formulation of statistical mechanics developed by Jaynes [7,8,9,10,11,13,14,15,20] is re-examined in light of the combinatorial approach. The analysis yields several new concepts including a generalized Clausius inequality, a generalized free energy ("free information") function, a generalized Gibbs-Duhem relation and phase rule, and a reappraisal of fluctuation theory and Jaynes' entropy concentration theorem. The free information concept provides a framework for the application of thermodynamic-like tools (e.g. state functions, cyclic integrals, efficiency ratios, Gibbs-Duhem and phase relations, Maxwell-like relations and Jaynes relations) for the analysis of probabilistic systems of any type. Finally, the combinatorial basis is shown to be embedded within a Bayesian statistical framework.
with UNSW@ADFA colleagues, with participants at the 2005 NEXT Sigma Phi conference, Kolymbari, Crete, Greece, and (since 2006) with Marian Grendar.