Where are all the citations going? I started noticing something that left me wondering a bit. My estimate is that the average paper (for example submitted to physical review journals, contains of the order of 50 citations). But if the average paper cites 50 other papers, then each paper should also receive 50 citations on average.

This is definitely not my track record. One could argue that cited papers are dominated by a few rare events (very important papers with thousands of citations), but by the very definition of an impact factor this cannot be the case, (unless one considers books, but I would assume they comprise max 20% of the citations): only top journals as nature reach an impact factor of this order, but these are far from the only papers cited. Many of them are other physical reviews, with an impact factor of order 3.

So that leaves me wondering, where are the missing citations going?
Is it just a matter of delay, or is there a flaw in my logic?
The impact factor is not the right measure. The impact factor counts the average number of citations a paper receives in the two years after publication. But in most papers, most of the references given are for papers (substantially) older than two years, and consequently they don't count for the impact factor of the journals cited. That means that the average impact factor over all journals is not equal to the average number of citations.

Second, as others have pointed out, the distribution is heavily skewed. That's true for papers in general, but also for papers by the same researcher. To take just my own papers (see this google scholar page), there are a couple with over a thousand citations, but #10 on the list is already below 100 citations and there is quite a long tail of papers with far fewer citations than they have references themselves.