Intuitive explanation of entropy I have bumped many times into entropy, but it has never been clear for me why we use this formula:

If $X$ is random variable then its entropy is:

$$H(X) = -\displaystyle\sum_{x} p(x)\log p(x).$$

Why are we using this formula? Where did this formula come from? I'm looking for the intuition. Is it because this function just happens to have some good analytical and practical properties? Is it just because it works? Where did Shannon get this from? Did he sit under a tree and entropy fell to his head like the apple did for Newton? How do you interpret this quantity in the real physical world?
Here's one mildly informal answer. 

How surprising is an event? Informally, the lower probability you would've assigned to an event, the more surprising it is, so surprise seems to be some kind of decreasing function of probability. It's reasonable to ask that it be continuous in the probability. And if event $A$ has a certain amount of surprise, and event $B$ has a certain amount of surprise, and you observe them together, and they're independent, it's reasonable that the amount of surprise adds. 

From here it follows that the surprise you feel at event $A$ happening must be a positive constant multiple of $- \log \mathbb{P}(A)$ (exercise; this is related to the Cauchy functional equation). Taking surprise to just be $- \log \mathbb{P}(A)$, it follows that the entropy of a random variable is its expected surprise, or in other words it measures how surprised you expect to be on average after sampling it. 

Closely related is Shannon's source coding theorem, if you think of $- \log \mathbb{P}(A)$ as a measure of how many bits you need to tell someone that $A$ happened.
Let me try to answer to part of your question where you ask how it's related to the "physical world" taking the same direction as MichaelNgelo's answer.

Let's say you have $N$ objects, out of them there are $n_1$ many $A$s and $n_2$ many $B$s. 

How many ways can you order them? For example for $N = 4$, with one $A$ and three $B$s we have:
$$ABBB$$
$$BABB$$
$$BBAB$$
$$BBBA$$

This is a combinatorics problem and we can find the formula: $$\frac{N!}{n_1!n_2!}$$

Note that there are more cases with $N=4$ and a arbitrarly number of $A$s and $B$s ($16$ cases) or with $N=4$ and with two $A$s and two $B$s ($6$ cases).

Given the proportions of $A$s and $B$s, it reduces the number of possibilities and we just need a way to differentiate between them. 

This is where the logarithm comes in to tell the least amount of bits to differentiate between these $M$ cases. Explicitly we need $\log_2(M)$ bits.

Now divide by $N$ to find the number of bits needed per symbol in the sequence. So if put all together, the formula becomes:

$$\frac{\log_2(\frac{N!}{n_1!n_2!})}{N}$$

and for convenience instead of talking about $n_1$ and $n_2$ in absolute number, we talk about the fraction of $N$, respectively $f_1$ and $f_2$. Finally tending $N$ to infinity gives the shannon entropy.

Example with $f_1 = 0.25$ and $f_2 = 0.75$:



Someone experienced can probably derive the Shannon entropy formula from the formula above with $N$ tending to infinity.

So a possible interpretation of the Shannon formula would be: The number of bits needed per symbol given the proportions of each symbol in advance in an infinite sequence.