Can frequent messages to an editor about the status of a paper under review be a cause for rejection? I have submitted a paper on mathematics to a journal and it was there for more than one year. I asked the editor many times about the reason of the review delay. At the end the paper was rejected with trivial comments. Some experts told me the reason of the rejection is that: I had asked the editor many times about the paper, which make him insist that the reviewers make their decision fast, and thus they gave trivial comments which led to the rejection. Can this really be a reason for rejection?
It really should not be a reason for rejection.  But the world of journals (even restricted to mathematics, which is my field as well) is vast and variable, and no one person gets to see more than a small part of it, so could it happen that repeated messages to the editor are a proximate cause of a paper's rejection?  Unfortunately it could and probably has at least once...but all kinds of crazy things have probably happened at least once.

What I want to convey to you is that you are perfectly within your rights to check in with the status of a paper at appropriate intervals.  In fact it is often wise (or even necessary) to do so, because a lot of processing jobs get done slowly on the part of the referees without specific reminders.  (I am speaking in particular both about my experiences as an author and as a referee.)

There are better and worse ways to deal with editors -- as a quick example, when I submit a paper I like to suggest that I will inquire about the paper after N months (I fill in a value of N, never less than 4) and see what kind of response I get.  I have never been told not to do this; at worst, I get a "contact us whenever you like, but don't expect expedited service because of this" type of response.  As ever, being polite and professional catches more flies.  But fundamentally: no, it is completely unprofessional to reject a paper because an author wants it to be processed in a timely manner.  Reputable journals and editors should take it as a point of honor not to do this.

It would be different if you gave the editor an ultimatum: e.g. "I need a report within three weeks or I will withdraw the paper".  Many editors will respond to that by cheerfully accepting your withdrawal.  But that does not seem to be the situation you describe.