What to do when a referee asks for changes I am not willing to make? I have submitted a paper later that was reviewed by four reviewers. I submitted my revisions to all reviewers. Three accepted the review, while the last is still asking for modifications. 

The problem is that the reviewer is asking to measure the performance of our work using performance metrics that were never taken into consideration before in the literature (and our problem is well studied - at 30 references !) .. 

These performance metrics would require a complete Master Thesis. 

Is it fair to judge a paper based on performance metrics that were not taken into consideration by the authors at the first hand ? (noting that none of the previous work considered these metrics) ..

I hate to say it, but the reviewer seems to be reading a paper about my problem for the first time ever, and some of the reviewer comments are contradicting.

Note: I am sorry if this have been asked before. I could not find any thing related.
In your question, there is no mention of an editor for the journal to which you have submitted your work. If you disagree with one or more of the reviewers, there is nothing remarkable about that; it is common, a review is an educated opinion about your work, supposedly based on facts. Normally you would provide an account for how you have met the comments (as you seem to have done) and in the cases where you disagree, you provide an account based on facts and reasoning why you think your way is better than that of the reviewer. It should then be up to the editor, not the reviewer, to decide whether your revisions make the manuscript acceptable or not. In some cases the editor will request a second round of reviews, this is normal, particularly if the revisions have been so substantial that the manuscript is quite different from the original.

This is what I see as a relatively typical way for a review process to take place.