How much time should one spend refereeing a paper? I finished my PhD about a year ago and every now and then I get asked to referee a paper (perhaps once every two months from a reputable journal). I tend to accept as many as possible (except from crappy so-called "predatory" publishers, which I decline immediately) because I find refereeing papers is a good way to force myself to learn new things. There is also the moral obligation to referee a peer's paper to compensate for the time your peers spend refereeing yours.

My problem is that, at this early stage in my career, I'm familiar with the work I did during my PhD and little else. Since I try to be as fair as possible (I hate it when I get a report from someone who has clearly not understood the paper or just skimmed through the text), this means I spend a lot of time reviewing literature and trying to understand a paper before submitting a report. For something very close to what I have done in the past, I could finish the report in half a day. For something further away, I could be looking at anywhere from one to three days, depending on how familiar I am with the methods used and the length of the manuscript. As everyone in the academic world knows, it sometimes gets very busy (just "normal" busy otherwise), and these breaks tend to disrupt my "paid" job routine (I understand reviewing is also part of my unpaid duties).

I guess reviewing will get easier and quicker as I gain experience, but is it usual to spend so much time refereeing papers? What would be an acceptable compromise between a rigorous review and reconciling it with your paid duties?
The slightly unsatisfactory answer is "enough time". You should provide a constructive review and that might take time. How much, depends on, for example, the quality of the paper, the complexity of the paper and, of course, your own experience. As new to reviewing, you probably spend longer than what you would with experience. Having said this, I would say that between close to a full workday to maybe two might be expected from an average research article.

Since you are a beginner, I would also recommend that you take your time. A common beginner's "mistake" is to focus on details more than the larger perspective. You need to focus on both. A paper may be poorly written but contain good science (and vice versa) so attending to all perspectives of an article is necessary, and that takes time.

This may all sound like a chore no-one would want to do but reviewing can be a really positive experience since you get to read a paper in such detail as you would probably not do otherwise, you see new science and you get to contribute to science with your expertise in ways other than producing your own papers.