How can I tell if I'm being too picky as a referee? I am a mathematician, frequently asked to referee papers. (As my career progresses, I now find myself frequently asked to referee good papers.)

I have found that I've gotten pickier and pickier as a referee. I just now finished a referee report (for an excellent paper, submitted to an excellent journal) with 53 bullet points on it: mistakes I found, requests for clarification, other suggestions.

On another occasion I believe I submitted six "revise and resubmit" reports for the same paper, before finally recommending acceptance.

In all these cases I am spending a lot of time reading the papers (which is worthwhile; they're interesting papers!), and I'm almost as meticulous as if it were my name on the paper.

How can I tell if I am going overboard with this? I have never heard any negative comments by anyone – including by journal editors, whom I asked for feedback on this matter after sending my reports. Indeed, editors have acted extremely happy that I've read these papers in such close detail. Nevertheless, I wonder if I am investing too much time in this, and/or annoying the authors.
In my humble opinion, it looks like you are the ideal reviewer actually! You give a lot of advice to improve the paper, and this directly benefits the authors and the journal. And apparently you give very precise advice, which is much more useful and actionable than general or vague remarks.

My main concern as a reviewer is to be fair in my final recommendation. As long as your meticulousness doesn't lead you to reject potentially good papers, you are doing a good job as a reviewer. However the question of whether you are spending too much time on it depends on your priorities, it's important to weigh the benefits and costs for yourself before you accept. It's perfectly acceptable to refuse a review from time to time in order to maintain the level of quality for the ones you accept.