<jupyter_start><jupyter_text>Quick Refresher from information theoryEntropy $ H(X) := -\sum_{x \in \mathcal{X}} p(x) \log p(x)$Conditional Entropy $ H(Y \big| X) := \sum_{x \in \mathcal{X}} p(x) H(Y\big|X=x) = \sum_{x \in \mathcal{X}} p(x) \sum_{y\in \mathcal{Y}} p(y|x) \cdot \log p(y|x) = \sum_{x \in \mathcal{X}} \sum_{y\in \mathcal{Y}} p(x,y) \log(\frac{p(x,y)}{p(x)})$Joint Entropy $ H(Y, X) := -\sum_{x \in \mathcal{X}} \sum_{y\in \mathcal{Y}} p(x,y) \log p(x,y)$Mutual Information$ I(X;Y) := \sum_{x \in \mathcal{X}} \sum_{y\in \mathcal{Y}} p(x,y)log(\frac{p(x,y)}{p(x)p(y)}) $Observe: 1. $H(X,Y) \leq H(X)+H(Y)$1. $H(Y|X) = 0 \iff$ Y is completely determined by X 1. $H(Y|X) = H(Y) \iff $ X and Y are independent1. $H(Y|X) = H(Y,X) - H(X)$1. $I(X;Y) = H(X) + H(Y) - H(X,Y)$  Important note: Don't get confused!!In the following you'll see X being used for the prior distribution that's the digit in case of MNIST and Y for the pixels. That is the opposite of what you know from any MNIST tutorial. The reason is that we're looking at the problem from an analytic view. We want to understand how the entropy of the result (the image) depends on the prior (the intended digit). It's actually the typical neural network that reverts the natural order by "guessing" what might have been meant by a certain image  Let's look at the distribution of a single pixel's values over a minibatch, subdivided into 10 arrays - one for each label:<jupyter_code># Pixel Nr. P
P = 300
pixels = i[:, P]
N_BINS=5
MIN_VAL=0
MAX_VAL=255
ln = np.argmax(l, axis=1)
# A list of lists: distributions of above pixel's values by digit
distr_by_digit=[list(map(lambda p: p[0], 
                         filter(lambda p: p[1]==digit, 
                                zip(pixels, ln)))) for digit in range(10)]<jupyter_output><empty_output><jupyter_text>Let's see the pixel's distribution on various digits<jupyter_code>plt.subplot(2,4,1)
plt.hist(distr_by_digit[0], bins=N_BINS)
plt.subplot(2,4,2)
plt.hist(distr_by_digit[1], bins=N_BINS)
plt.subplot(2,4,3)
plt.hist(distr_by_digit[2], bins=N_BINS)
plt.subplot(2,4,4)
plt.hist(distr_by_digit[4], bins=N_BINS)
plt.subplot(2,4,5)
plt.hist(distr_by_digit[6], bins=N_BINS)
plt.subplot(2,4,6)
plt.hist(distr_by_digit[7], bins=N_BINS)
plt.subplot(2,4,7)
plt.hist(distr_by_digit[8], bins=N_BINS)
plt.subplot(2,4,8)
plt.hist(distr_by_digit[9], bins=N_BINS)<jupyter_output><empty_output><jupyter_text>As expected, most of the pixels are either black or white with few in the grey areas. And the distributions obviously differ from digit to digit. If you look at pixel 200, you will see that it's basically always zero. Obviously, some pixels are helpful, others aren't. To support statistics on the distribution we use a partitioning function that distributes the values into N_BINS<jupyter_code>def partition(arr, width_bin, n_bins=N_BINS):

    # for each element of arr, which bin is to be incremented?
    arr = np.array(arr)//width_bin
    # Count the number of entries in each bin. I know, this is a tough algorithm. But it's cool.
    return np.sum(np.eye(n_bins)[arr], axis=0)<jupyter_output><empty_output><jupyter_text>Let's verify by partitioning the pixel's distribution for all images of a 4. The top-right histogram above should match what we got here.<jupyter_code>partition(distr_by_digit[4], 255//N_BINS+1, N_BINS)<jupyter_output><empty_output><jupyter_text>Now let's define the entropy of an arbitary distribution. We do the statistics by distributing the values into N_BINS arrays.<jupyter_code>def entropy(distr):
    res, _ = _entropy(distr)
    return res

def _entropy(distr):
    """
        calculate the entropy based on number counts in N_BINS
    """
    bins = partition(distr, MAX_VAL//N_BINS+1, N_BINS)
    N = sum(bins)
    return sum(map(lambda n_i: 0 if n_i == 0 else -n_i*log(n_i), bins))/N + log(N), N<jupyter_output><empty_output><jupyter_text>Note that we used a little simplification here. Namely, with $ p(i) $ being the probability of finding the pixel's actual value in bin number $i$, $N$ being the total number of pixel values we estimate the probability $ p(i) = \frac{n_i}{N} $. The total number of pixel values is obviously the total number of images that are represented by the given distribution.$$ -\sum_{i=1}^N p_i \log p_i = \frac{1}{N} \sum_{i=1}^N n_i \log(\frac{N}{n_i}) $$$$ = \frac{1}{N} \sum_{i=1}^N n_i (\log N - \log n_i) = \log N - \frac{1}{N} \sum_{i=1}^N n_i \log n_i $$You have certainly already figured out that the very last expression is exactly what's being calculated in the _entropy() function. Let's see the entropies of the pixel's distribution for all 10 digits. Note that these are by definition the conditional entropies of the entire distribution, given a particular digit.<jupyter_code>entropies = [entropy(distr_by_digit[digit]) for digit in range(10)]
total_entropy = entropy(pixels)
entropies, total_entropy<jupyter_output><empty_output><jupyter_text>Remember how we defined the conditional entropy $H(Y|X)$ of the total distribution? It's the probability-weighted sum of the particular entropies over the values of the prior X:$$ H(Y \big| X) := \sum_{x \in \mathcal{X}} p(x) H(Y\big|X=x)$$$$ = \sum_{x \in \mathcal{X}} p(x) \sum_{y\in \mathcal{Y}} p(y|x) * \log p(y|x)$$ Let's use this equation now to calculate the conditional image of a single pixel's distribution over a mini-batch of images<jupyter_code>def particular_entropies(joint_distr):
    """
    calculate all particular conditional entropies
    :joint_distr a matrix of 10 x N_BINS bins counting the activations of a pixel 
    """
    res = []
    for x in range(3):
        nx = np.sum(joint_distr[x], axis=0)
        if nx != 0: 
            p_y_given_x = joint_distr[x] / nx
            p_y_given_x = list(filter(lambda p: p!=0, p_y_given_x))
            ent_x = -np.sum(p_y_given_x * np.log(p_y_given_x))
            res.append(ent_x)
    return res
    
def cond_entropy(pixels, labels):
    """
        calculate the conditional entropy of a pixel distribution in a mini-batch of images, 
        given the true labels.
    """
    X = labels
    Y = pixels
    X_probs = np.sum(np.eye(10)[X], axis=0) / len(X)
    
    by_digit=[list(map(lambda p: p[0], filter(lambda p: p[1]==digit, zip(Y, X)))) for digit in range(10)]

    bins =[partition(by_digit[digit], MAX_VAL//N_BINS+1, N_BINS) for digit in range(10)]

    entropies = particular_entropies(bins)
    
    zipped = zip(X_probs, entropies)
    return np.sum((list(map(lambda p_e: p_e[0]*p_e[1], zipped))))
    
    
cond_entropy(pixels, ln)<jupyter_output><empty_output><jupyter_text>Now let's compare the total entropy with the conditional one for a number of pixels. The higher the value the bigger the difference between total and conditional entropy. We expect to gain more information from those pixels.<jupyter_code>np.array([total_entropy - cond_entropy(i[:, P], ln) for P in range(200, 320)])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1288607958626634, 'fraction_numerical': 0.014509409567590864, 'mean_word_length': 4.310450038138826, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '240657', 'license': 'Apache-2.0', 'n_tokens_mistral': 2613, 'n_tokens_neox': 2459, 'n_words': 795, 'path': 'others/notebooks/Entropy_on_MNIST.ipynb', 'repo_name': 'Project-Ellie/capsnet-fashion', 'size': '6961'}