Is it OK to let somebody know I refereed their paper? It is not uncommon, especially for narrow fields, that you review papers authored by people that you might, at some point, come to interact with more or less closely. This can range from meeting them at a conference to even start a collaboration together.

Some times, when discussing with these people a point directly related to one of their papers it could be relevant to make a comment based on a discussion that came up as part of the reviewing process.

I have recently felt tempted to tell someone "hey, I refereed you paper, by the way, so I am particularly familiar with this and that..." but refrained myself because peer review is supposed to be anonymous.

Is it appropriate to bring up that I refereed somebody's paper when having a discussion with this person, in particular if it adds constructively to the discussion? Would it become "more appropriate" as time goes by and the review becomes "a thing of the past"?
but refrained myself because peer review is supposed to be anonymous.


Continue to refrain yourself. You can certainly say that you are familiar with the work of somebody else -- published articles can be read by anyone, after all -- but openly declaring that you were one of the reviewers would not add anything to the discussion and could put you in an awkward situation, if the authors haven't actually liked your review.

I work in a narrow field where almost all of the authors know each other and where it is not so difficult to guess who the reviewers are, also because journals ask the authors to suggest possible reviewers. These authors collaborate quite frequently, but I've never heard of anyone disclosing their role.