Models vs. Inductive Inference for Dealing With Probabilistic Knowledge

Two different approaches to dealing with probabilistic knowledge are examined -models and inductive inference. Examples of the first are: influence diagrams [1], Bayesian networks [2], log-linear models [3, 4]. Examples of the second are: games-against nature [5, 6] varieties of maximum-entropy methods [7, 8, 9], and the author's min-score induction [10]. In the modeling approach, the basic issue is manageability, with respect to data elicitation and computation. Thus, it is assumed that the pertinent set of users in some sense knows the relevant probabilities, and the problem is to format that knowledge in a way that is convenient to input and store and that allows computation of the answers to current questions in an expeditious fashion. The basic issue for the inductive approach appears at first sight to be very different. In this approach it is presumed that the relevant probabilities are only partially known, and the problem is to extend that incomplete information in a reasonable way to answer current questions. Clearly, this approach requires that some form of induction be invoked. Of course, manageability is an important additional concern. Despite their seeming differences, the two approaches have a fair amount in common, especially with respect to the structural framework they employ. Roughly speaking, this framework involves identifying clusters of variables which strongly interact, establishing marginal probability distributions on the clusters, and extending the subdistributions to a more complete distribution, usually via a product formalism. The product extension is justified on the modeling approach in terms of assumed conditional independence; in the inductive approach the product form arises from an inductive rule.

Ex amp 1 es of the second are: .games-against nature [5, 6J, varieties of maximum ent�opy methods [7, 8, 9J, and the author's min-sco�e induction In the mode ling approach, the basic issue is manage abi lity, with �espect to data elicitation and computatic • n.
Thus., it is assumed that the per·tinent set of u s er·s in sc• me sense Knows the �e levant p�obabi l ities, and the p�ob lem is to forma.t that Know ledge in a wa>' that is convenient to input and sto�e and that allows computation of the answe�s to current questions in an expeditious fashion.
The basic issue fo� the inductive approach appears at fi�st sight tc• be ver·y differ·er.t.
In this approach it is p�esumed that the �e levant p�obabi l ities a�e on ly partia lly Kr.owr., ar.d the pr·ob lem is to exter.d that incomp lete inf c • r· mation in a �easonab le way to answe� cu��ent questions.
Clear ly, this approach �equi�es that some form of induction be invoKed. Of course, rnanageabi l ity is an impo�tant addi tiona l cc• ncer·n.
Despite their seeming dif fe�ences, the two approaches have a fai� amount in common, especia ll>' t.vi th r·espect tc• the structura l f�ameworK they emp loy. Rough ly speaKing, this f�ameworK invo lves identifying cluste�s of variab les which strong ly inte�act, establishing margina l probabi lity distri butions on the clusters, and extending the subdistributions to a more comp lete distribution, usua lly via a produc t forma l ism.
The product extension is justified on the mode ling approach in terms of assumed conditiona l independence; in the inductive app�oach the p�oduct form a�ises from an inductive ru le .  The correspqnding probabll ity distributions on members of C PC = P<Y1) , •• , ,P<Y1,.) is called a pr·obabi 1 i ty system Cor· system for short.) In this notation, Y may be either abso lute or· conditional.
A system PC is called consistent if there is a proba bility distribution P<X> that fulfills (1) or (1'") fc·r· all cc• mponents Y ir1 PC. Ir1 gener·al, if PC i:. con:.isterrt, ther·e will be a set K< PC> of distributions compatible with PC.
In the model approach, it is assumed that a system PC represents the clustering of descriptors with respect to dependence; i.e., within a component\' , the descr·iptors have  A web is a structure which fulfills the recursion: 1.
Any absolute compone-nt is a web.

2.
If Y is a terminal component, and C-CY} is a web, the-n C is a \AJE>b.
From the definition, a we-b contains at least one abso- A web is somewhat more general than influence diagrams or Bayesian networks.
These can be characterized as condi tional webs where for any conditional component Y = <ZIW>, Z cons.ists of a singfe descriptor.
Conditional webs are signicant for modeling probabil is tic Knowledge as a result of two basic properties: a.
The product P •<x> = n P<Y> is a joint probabi 1 i ty c distribution on X. b.
Proofs for these assertions are readily constructed by induction on the number of components in a web.
What these two properties. entai 1, in effect, is that if you can represent your knowledge concerning a distribution P<X) by the sub-distributions PC for a web C plus assuming conditional independence for descriptors not in common components, then the product P <X> "automat i ca. 11 y" expresses. that know! edge.
From the model! ing point of view, then, a web is a rela tively manageable representation of probabilistic Knowledge.
All that need be input are the subdistributions PC. The product is quite convenient for computations; e.g., the mani pulations feasible for influence diagrams are directly ex tendable to webs.

Inductiorr and maximum-entr·opy
Tur·rr i ng to the inductive apprc• ach, in an ear·l i er· publ i cation I demonstrated that for a subspecies of web, namely a forest, the product.extension is the maximum entropy exten sion of PC.
(11] A forest is a web in which all terminal components Y = <Z: W) fulfill the additional restr·iction that W is contained in some componer .. Knowledge will be at least as informative as a conclusion t�-•ithout that Knowledge. [10] Thus, if all you Knot"'' is a set of subdistributions PC, and PC is a forest, then the product extension is a supportable estimate of the total distribution.
One of the motivations for studying webs was the expec tation that the product extension would also turn out to b e the maximum entropy extension for a general web. The expec tation was based on a purported result of P. M. Lewis fre quent 1 y cited in the 1 i tera ture tc• the effect that for· a structure with a product extension, the product is the maximum entropy extension. [12] Unfortunately, the Lewis "result" happens to be incorrect.  Table I; where "1" means occurrance and "0" means non-occurrance in the 1 ist of cases. Table I x, x.,.., P<X3 IX 1 .Xl.) The product distribution p•(X) is along with another distribution P 0 (X).
of PC--which can easily be verified by clearly a higher entropy distribution. Table I I x, x'l. X 3 P •oo The numerical differ-ence in entropy is small, but the difference between .25 and 1/6 for· P<1 ,1 ,1), e.g., may not seem tr·ivial.
The elementary structure C of the example is actually a substructure of any web that is not a forest. Hence a similar counter-example can be constructed for any such web. The example is also a counter to the Lewis "result".
The upshot of this inquiry, then, is that a forest is the most general structure for which the product extension is always the maximum entropy extension.

Discussion
At first glance, the fact that the product extension of a web is not in general maximum entropy may appear benign.
From the standpoint of the model approach, the basic properties of a web--the product is a probability and an extension of PC--maKe webs a highly convenient representation of probabilistic Knowledge.
All that is lost is a desirable, but by no means essential, fallbacK.
In the case of a forest, for example, if the assumption of conditional independence for separated descriptors is shaky, then it can still be contended that the product is a reasonable estimate of the joint distribution, given PC. It would be a valuable safety feature if the same could be claimed for a web.
Fr·om the standpoint of the inductive approach, it is perhaps unfortunate that the product extension of a web is not maxi mum entropy.
However, the maxi mum entropy ex ten�. ion can be sought by other means.
[l!J What is lost is the convenience of the pr·oduc t form.
For· the comp 1 ex systems of many descriptors common in expert systems, maximum entropy formal isms are 1 ikely to be cumbersome.
On a somewhat deeper level, however, the result is thought-provoKing.
One route that can be taKen is to "prune" the structure to a forest.
Lemmer [15 J has adopted this suggestion, fol A basic e 1 emen t missing fr·om this program is a measure of the information that is lost by the pruning process.
Concomm i tan t 1 y, there is no systematic procedur·e fc• r· deter· mining the most informative forest contained in the knowledge available to the analyst. Given a general probability system PC, if PC is consistent, the amount of information in PC can be defined as max Entropy( P) . At present, there is no PE.K<PC) way to determine this quar.tity directly from PC--or·, fc• r· that matter, determining whether PC is consistent.
These issues appear to be one area of potentially fruitful research.