Editor says "the paper will not be acceptable within a reasonable number of revisions": is it worth appealing the rejection? Given the editor's comments reported below, is it worth appealing this rejection or one should submit to another journal?


  At the rate of improvement that has occurred over the last revision,
  the paper will not be acceptable within a reasonable number of
  revisions. Consequently, your paper is being declined. The reviewer
  comments are very helpful and should assist you to prepare the paper
  for consideration in another journal. Reviewers' comments on your work
  have now been received. You will see that they are advising against
  publication of your work. Therefore I must reject it. For your
  guidance, I append the reviewers' comments below. Thank you for giving
  us the opportunity to consider your work.
Appeals are usually intended to be pursued when you think there was a serious procedural problem with the way your paper was handled, resulting in a decision that was not properly based on your paper's content.  Some examples I might think of:


Reviewers were not qualified to judge the paper, or their reports were of extremely low quality (suggesting they may not have read the paper or given it serious consideration)
Editor or reviewers have a conflict of interest
Editor or reviewers show evidence of bias or prejudice against the author
Other unprofessional behavior on the part of editors or reviewers


None of that seems to apply here (unless there is something you haven't mentioned).  It sounds like the reviewers read your initial submission and said it wasn't acceptable for publication as it was; it needed significant revision.  You made some revisions but they are not satisfied.  So you still don't have a version which they consider suitable for publication.  If the editor thought it was worth the time to keep revising it, he could let you do that, but he doesn't.  So your paper is rejected.

It sounds to me like this decision was properly made based on the reviewer's opinion of the paper's content, with which the editor concurs.  You may disagree with that opinion, but simple disagreement isn't grounds for appeal - you'd have to show the editor and/or reviewers did something objectively wrong.

So submit to another journal.  But first, go through the reviewers' comments carefully, and make the changes you feel are appropriate.  They might need to be extensive.  Reading between the lines, it sounds like the reviewers didn't think that your revision did a good enough job of addressing their concerns, so maybe you need to think more carefully about what they said.  The reviewers for the new journal may well have the same concerns.