Justification that 'good exposition' is uniquely defined What evidence is there that changes to exposition that improve clarity for one reader will necessarily improve clarity for another reader?

The context I'm thinking of here particularly is referee comments (although the answer will probably apply more broadly). In various places I've read that if a referee says something is unclear then they are automatically right, because at least one reader has had trouble understanding. It seems to me that this assumes that 'the clearest description' will be the same for all readers. It is not clear to me why it is fair to rule out the possibility that the text simply doesn't align with the referee's preferred viewpoint.
This is a false dilemma. Good exposition is not uniquely defined.
Yet, if a referee claims they don't understand what you write, chances are that they are right. Add some clarifications. This doesn't mean necessarily rewriting everything with the referee's favorite approach: maybe just a remark tailored to people with their background. ("If you're used to seeing concept X defined in this other way, note that this is equivalent because...") If there is a wrong approach that you wish to refute, do it, in the paper or in the response letter.

Good exposition doesn't mean the referee's preferred viewpoint, but it doesn't mean your preferred viewpoint either.