How to respond to a perceived unfair editorial decision? We have an editorial decision after major revision: decline and resubmit.

Decisive is a reviewer who feels that the novelty of our work is too much limited into a specific area.

This reviewer gave concrete directions in the first round of review. We had to remove a full explanatory section because it should be well known to the audience. We had to add an additional analysis to support the relevance of our method development. We had to restructure the paper and change our line of reasoning and writing style. Otherwise, the submission would not qualify to be publishable. The reviewer even explicitly expressed his future support as the reviewer saw ‘promise in our intentions’.

We carefully followed all the reviewer’s critiques and basically rewrote our paper. For language we send the paper to a publishing service. The reviewer acknowledges and applauds our improved work, but still rejects the work. 

The editor encourages us to decline and resubmit (with the assurance that we will get new reviewers), which I interpret as the paper is considered to be publishable for this journal and its audience. But I also feel, addressing this reviewer’s critiques, has not improved its chances for acceptance (here or elsewhere) because the paper has become more specialized. 

If we had refuted the reviewer’s critiques in the first place, the paper would have been rejected. Now we followed the reviewer’s critiques carefully and the paper is still rejected. 

Why do certain reviewers recommend major revision if they never intend to accept a paper in the first place? Is it not much more fair to reject a paper immediately if you feel the topic is too specialized for the audience, although this journal publishes many other papers which are highly specialized in their domains?

Or should I ask why the editor chooses the easy route: decline and resubmit? The final reviewer’s comment is not repairable and the editor is in charge.

Or should I ask what I did wrong? I just do not know. How can I improve my work further if I just don’t know what to improve?

What should we do? Appeal, resubmit the revised version with this journal, resubmit the original version somewhere else?
That is a frustrating experience that many (all?) of us were forced to endure at some point in our careers.

Fact is, a "major revision" is not guaranteed acceptance upon receive. Papers that are not rejected in the first round of review usually are close enough that they can be accepted after revision(s), but that is in no way guaranteed.

Now, in your case, the reviewer made promising remarks for the future, yet failed to act on them. It can be that they misjudged the first time, that they are inexperienced (perhaps the review was delegated to a student), that they had malicious intents, etc. This line of reasoning, however, is a dark rabbit hole best avoided. You have no way of knowing for certain what was up with the reviewer at the time and little benefit even if you did, so it's best to let that go.

The next point is the editor's decision, "decline and resubmit". Again, there is zero guarantee that the paper will be accepted by a potentially different set of reviewers. Usually, journals prohibit submitting the same paper again once it has been rejected. So, if it is a particularly important venue, I'd consider this a better thing than straight out decline. Another important point is that (at least in my field) for a resubmit after decline, a predefined amount of time is required to pass, e.g. 6 months, and a significant rewrite is expected. Practically, close enough to a new paper. If similar practices apply, it is probably easier to resubmit somewhere else.

Appealing rarely works without substantial proof of malpractice during review. In fact, you may have already gotten the best result an appeal can get you: the right to resubmit for a new set of reviewers.

Bottom line, the lost time and the additional time it will take to resubmit is unrecoverable. You have the review feedback as it is, and you have your current and previous draft. If you feel that the reviewer's directions were misplaced, you can polish up the older draft and resubmit. My advice would be to try to get the best out of both versions, take into account any remarks the reviewers had, do a thorough proofread of the paper, and resubmit at some other venue.