<jupyter_start><jupyter_text>Kangaroos example Gull and Skilling (1984) gave the following example, which they describe as a physicist's perversion of the formal mathemetical analysis of Shore and Johnson 1980 showing that maximizing the entropy function is required for logical consistency.Information:> 1. A third of all kangaroos are blue-eyed> 2. A quarter of all kangaroos are left-handedOn the basis of this information alone, what proportion $x$ of kangaroos are both blue-eyed and left-handed? left-handedright-handed$\sum$blue-eyed$x$?$\frac{1}{3}$green-eyed??$\sum$$\frac{1}{4}$1 The normalization conditions imply that we have: left-handedright-handed$\sum$blue-eyed$0 \leq x \leq \frac{1}{4}$$\frac{1}{3} - x$$\frac{1}{3}$green-eyed$\frac{1}{4} - x$$\frac{5}{12} + x$$\frac{2}{3}$$\sum$$\frac{1}{4}$$\frac{3}{4}$1 parameterized in terms of $x$.  Are handedness and eye colour independent?Most people would intuitively guess that $x = \frac{1}{4} \times \frac{1}{3} = \frac{1}{12}$. Why? Any other value than $x = \frac{1}{12}$ would indicate that knowing the kangaroo's eye-colour tells us something about its handedness. But we have no information that there is a link, or in which direction any such link might be. So our intuition is that the most reasonable assignment assumes no link.Intuitively, this is the least informative choice, the flattest subject to the information we know. It is actually the choice that **maximizes entropy** of the probability distribution. Below we show this numerically.  Entropy The "information entropy" $H(p)$ of a probability distribution $p$ is a measure of its disorder or flatness. For a discrete distribution $p$ it is defined as:$$H(p) = -\sum p_i \log(p_i)$$whenever $p_i > 0$.<jupyter_code>from scipy.stats import entropy<jupyter_output><empty_output><jupyter_text>What properties does it have?  Example: a deterministic die (with no randomness)Suppose a die is rigged so it **always** comes up 1. Then it has this probability distribution:$i$123456$\sum p_i$$p_i$1000001Its entropy is **zero**.<jupyter_code>import numpy as np

p_rigged = np.zeros(6)
p_rigged[0] = 1
entropy(p_rigged)<jupyter_output><empty_output><jupyter_text>Example: fair die$i$123456$\sum p_i$$p_i$$\frac{1}{6}$$\frac{1}{6}$$\frac{1}{6}$$\frac{1}{6}$$\frac{1}{6}$$\frac{1}{6}$1<jupyter_code>p_fair = np.ones(6) / 6
entropy(p_fair)<jupyter_output><empty_output><jupyter_text>Example: unfair die<jupyter_code>p_unfair = np.ones(6)
# Assign arbitrary values:
p_unfair[0] = 5
p_unfair[5] = 4
p_unfair /= p_unfair.sum()   # normalize
print(p_unfair)
entropy(p_unfair)<jupyter_output><empty_output><jupyter_text>Exercise: show that the fair die probability distribution has higher entropy than any other distribution (for any unfair die) Brute-force approach: try calculating entropies for each of, say, 1 million different assignments of $p_1,\dots,p_6$ (normalized to sum to 1).<jupyter_code>ps = np.random.uniform(size=(10**6, 6))
entropies = entropy(ps.T)
entropies
np.all(entropies < entropy(p_fair))<jupyter_output><empty_output><jupyter_text>Back to kangaroos Recall this table of probabilities: left-handed_right-handed$\sum$blue-eyed$0 \leq x \leq \frac{1}{4}$$\frac{1}{3} - x$$\frac{1}{3}$green-eyed$\frac{1}{4} - x$$\frac{5}{12} + x$$\frac{2}{3}$$\sum$$\frac{1}{4}$$\frac{3}{4}$1 Here is a function that returns the matrix of probabilities for a given $x$ value:<jupyter_code>def kangaroo_prob_matrix(x):
    assert 0 <= x <= 0.25
    p = np.zeros((2, 2))
    p[0, 0] = x
    p[0, 1] = 1/3 - x
    p[1, 0] = 1/4 - x
    p[1, 1] = 5/12 + x
    return p
kangaroo_prob_matrix(0.2)
params = np.linspace(0, 0.25, num=100)
entropies = [entropy(kangaroo_prob_matrix(x).flatten()) for x in params]<jupyter_output><empty_output><jupyter_text>We can determine the entropy-maximizing parameter $x$ empirically as:<jupyter_code>x_opt = params[np.argmax(entropies)]
x_opt
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline
plt.plot(params, entropies)
plt.xlabel('$x$')
plt.ylabel('entropy $H(p)$')
plt.scatter(x_opt, np.max(entropies), color='red')
plt.text(x_opt, np.max(entropies)+0.01, f'$x={x_opt:.3f}$')
plt.title('Entropy $H(p)$ for different probability distributions $p$ parametrized by $x$');<jupyter_output><empty_output><jupyter_text>Solution with maxentropy package<jupyter_code>from itertools import product

samplespace = list(product(['left-handed', 'right-handed'], ['blue-eyed', 'green-eyed']))
samplespace
def f0(x):
    return x in samplespace

def f1(x):
    return x[1] == 'blue-eyed'

def f2(x):
    return x[0] == 'left-handed'

f = [f0, f1, f2]
from maxentropy import MinDivergenceModel
model = MinDivergenceModel(f, samplespace, vectorized=False, verbose=False)<jupyter_output><empty_output><jupyter_text>Here is the feature matrix that has been computed on all elements in the sample space:<jupyter_code>model.F.A
# Now set the desired feature expectations
K = np.array([1., 1./3, 1./4]).reshape(1, 3)
# Fit the model
model.fit(K)
model.F.todense()
model.params
model.probdist()
np.allclose(K, model.expectations())
# Output the distribution
print("\nFitted model parameters are:\n" + str(model.params))
print("The fitted distribution is:")
model.show_dist()
model.expectations()
np.isclose(x_opt, model.probdist()[0])
model.entropydual()<jupyter_output><empty_output><jupyter_text>Our earlier version gave this matrix of probabilities:<jupyter_code>kangaroo_prob_matrix(x_opt)<jupyter_output><empty_output><jupyter_text>Using the `maxentropy` package we can retrieve this as:<jupyter_code>model.probdist().reshape(2, 2).T
np.isclose(model.probdist().sum(), 1)
g = sns.barplot(np.arange(len(samplespace)), model.probdist())
g.set_xticklabels(samplespace, rotation=90);
sns.heatmap(kangaroo_prob_matrix(x_opt), annot=True, cmap='Reds')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14258783204798628, 'fraction_numerical': 0.02913453299057412, 'mean_word_length': 6.187192118226601, 'pattern_counts': {'":': 0, '<': 48, '<?xml version=': 0, '>': 48, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '61240', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 2175, 'n_tokens_neox': 2034, 'n_words': 555, 'path': 'notebooks/Kangaroos example.ipynb', 'repo_name': 'cagrell/maxentropy', 'size': '5835'}