Entropy of a distribution over strings Suppose for some parameter $d$, we choose a string from the Hamming cube ($\{0,1\}^d$) by setting each bit to be $0$ with probability $p$ and $1$ with probability $1-p$. What is the entropy of this distribution on the Hamming cube? Clearly, if $p=\frac{1}{2}$, then the entropy would just be $\log \left(\frac{1}{2^d} \right) = d$. If $p = 0$, then the entropy would be $0$. What would be the entropy for the general case in terms of $p$? It would clearly be less than $d$....

The probability that we select a specific string with $k$ one bits is 
$$p^k(1-p)^{d-k}$$
Thus
$$\Bbb{E}[-\log_2 P] = -\sum_{i\in\{0,1\}^d}P(i)\log_2P(i) \\= -\sum_{k=0}^{d}\binom{d}{k}p^k(1-p)^{d-k}\log_2(p^k(1-p)^{d-k})\\
=-\sum_{k=0}^d\binom{d}{k}p^k(1-p)^{d-k}[k\log_2(p)+(d-k)\log_2(1-p)]\\
= -\log(p) \Bbb{E}[H] - \log(1-p)\Bbb{E}[T]$$
Where $H\sim \operatorname{Binom}(d,p)$ and $T\sim\operatorname{Binom}(d,1-p)$. 
These respectively have means $dp$ and $d(1-p)$. The resulting entropy is therefore
$$-d(p\log_2(p)+(1-p)\log_2(1-p))$$
