Am I disadvantaged by declining authorship? I have seen many discussions of authorship conflicts, which usually relate to a substantial contribution not being recognised, or a more senior supervisor insisting on co-authorship.

However, I am in the opposite situation: I have been offered co-authorship on a paper (currently in preparation for journal submission), and my contribution so far was very small. I am aware of the ICMJE guidelines on authorship and in truth my contribution to the acquisition/analysis of data probably wasn't "substantial".

Ethically, the right thing to do is probably to decline co-authorship. 

(In fact, I am included on a previously published paper to which I made no contribution. My PI included me: he argued that I contribute to the group's work on this subject as a whole. But it does not feel right!)

But, I feel that many people would not decline... therefore if I decline, I will have one fewer paper on my CV than somebody who would accept. In the long run, presumably this puts me at a disadvantage to people who accept such co-authorships, e.g. this could include people I will be competing against the next time I apply for a job (I am an early postdoc).

Should I accept co-authorship and live with the slightly guilty feeling? Am I disadvantaged if I decline?

Note: I am particularly interested in the career implications of this decision/future decisions like this, and what is the "norm" - rather than answers that just apply the more "theoretical" criteria for authorship (such as ICMJE).
Declining authorship is quite common among my collaborators. There are cases when you disagree with the paper and you do not want to assume responsibility for what is written there. Declining authorship is the right thing to do. I assume that in your case, you agree with your coauthors. In this case, you did the right thing to decline authorship when you feel that your contribution is too small, even if this could be a disadvantage to you.

I declined authorship twice so far (and I don't have that many publications). In both cases, I felt that my contribution was too small to warrant authorship. In the second case, I didn't agree fully with the paper, but I was involved in the experiments leading to the paper. In both cases, however, my coauthor(s) insisted of including me. In this case, I tried to contribute more in ways I could, such as by reading the draft thoroughly and giving constructive feedback that would improve the quality of the paper.

There are different ways your potential employer assess your publications. Some base it on the total number of papers and citations you authored; in this case you could be at a disadvantage if you have fewer papers than you would have by not declining. Some do not care about papers you are not a first author of (in my field the first author is assumed to have performed most of the work); in this case it does not matter if you have more papers or not.

However, in a job interview, your potential employer might ask you about your specific role in a paper, especially if the paper turns out to be highly cited (I was asked this question before). If you are not able to give a satisfactory answer, this could be a disadvantage instead of an advantage to you.