Is reviewing papers mandatory? I am baffled by this question. Elsevier boycott aside, is one supposed to review a paper every time he/she is asked? And somebody will be upset if he/she declines the request to review? Or is this only maths thing?

Reviewing paper is unpaid, while it takes a lot of time and effort (at least for me). As such, I expect someone in editorial board to be thankful if one accepts to review a paper, and move on if his/her request for reviewing is declined.

To be angry with a (junior) researcher for not reviewing a paper is unacceptable behaviour for me. We are not slave.

Note: Until now, I am happy every time I am asked to review a paper.
No, reviewing is not mandatory, it's just good community service.

When I'm soliciting reviews, as an editor or program chair, I would actually much rather have people who don't want to review simply decline to review.  It does mean that I have to search for more reviewers, but it's much preferable to getting low-quality reviews or no review at all.

Moreover, there is always a limit to the amount of service that one should be expected to do.  If you're reviewing a good amount of papers (see discussion in this question about how much is enough), you should feel free to triage. Just be polite when you decline: even with things you don't want to deal with, it's usually better to simply say "I don't have time right now" (true enough) than to offend a person by completing the sentence with something like "... for your crappy journal" or "... for this horribly tedious-looking paper."

Of course, there is always the possibility of encountering an ego-centric jerk on a power-trip, but life is too short to spend all of fearing that you might encounter such.