After successfully publishing papers during my Post-Doc, why am I now having trouble publishing as a tenure track academic? I have had my PhD in mathematics for four and a half years now, completed two postdocs and found a tenure track job at a mostly teaching university. While I provide substantial background information below, my real, general question is:


  How can I identify the causes of a decline in acceptance rate of my papers? 


Background

I had a fairly good research program in geometry and topology, with 5 papers in relatively good journals and an undergraduate research paper published over the course of two postdocs.

I then took a tenure track job with 33 teaching credits a year. While I have little time for new research, I had several papers ready for submission.

These new papers are better than previous papers of mine published in the same journals; not only do I (subjectively) consider them better, but they have received citations in preprint form, and I was invited to a give a talk based on a stranger reading one of them, neither of which happened with my earlier papers.

Possible reasons

I can think of a few possible reasons why these papers are not being published:


They are just not up to shape, because I don't have the time to focus on polishing them.
The referees hate me. This may seem silly, but I am in a very specialized field, with only 3-4 people who have published more than one paper on the subject. One of these people have retired, and another openly said that he had been my referee before on more than one occasion, had rejected my largest paper, and said that he didn't like the direction I was going with the research.


Also, I had a toxic relationship with my postdoc advisor, to the point where it came down to personal insults. They said that they were furious at my past letter of reference writer for recommending me, knowing I was a 'bad egg'. Though we have since made up, I have heard reports from others about this advisor talking about me in a negative way. Also, none of the jobs I applied to with their letter of recommendation wrote me back, while 30-50% of the applications without his letter were replied to favorably, including my current job.


Chance. The reviewers of the first papers happened to be interested in those topics, and the reviewers of the current papers have not been interested.


Conclusion

How do I tell what is really the issue? A few notes:


It's very hard to do research with 33 credits of teaching, but all the work was done before I accepted this job.
The main paper (that was rejected by that referee) went through 5 revisions with the aid of an advisor before submission. I have revised and submitted to 3 or 4 other journals, one after another, incorporating each journals feedback into the new version and trying a weaker journal each time.


My goal is to just get these papers published; at my current institution, the publication of these two papers alone would be enough to satisfy all of the research requirement for tenure.
I feel for you.  I have been in the (pure math) game for a lot longer than you, and sometimes my work is still treated shabbily by editors and referees in a way that is hard to figure out.  I feel that the publication process has gotten more opaque in the dozen or so years I've been involved with it: at the beginning of my career, whenever a paper got rejected, I would come away with a clear reason for why.  I have had some truly head-scratching rejections in recent years...along with acceptances in prestigious journals, which is good for me but even more confusing in this regard.

I think that the best advice I can give to you is to find a trusted senior mentor -- someone who understands the content of your work and the sociology of your field -- and really go over your situation with them in minute detail.  I would certainly go over all of the referee reports in detail.  If any one rejection feels especially unexplained to you, please realize that you certainly can ask the editor for more information, if you do so extremely politely and while making clear in advance that you are not arguing but rather asking for more information.  (I have experienced the wildest of extremes in response to such inquiries: I have gotten no response at all, and I have also gotten "I'm so sorry, your paper was rejected by mistake!  Would you like to resubmit it?" Seriously.)  

Without this specific information, we can just guess on top of your guesses.  But that's interesting to me, so I'll play along.  First, here is one piece of information that probably should have come before the end of your question:


  My goal is to just get these papers published; at my current institution, the publication of these two papers alone would be enough to satisfy all of the research requirement for tenure.


This is pretty key.  Before I read it, I was going to respond to your worries that you are publishing work in a very narrow field that is (perhaps!) being viewed by the same small set of people as unpublishable by suggesting that you make an effort to branch out a bit in your work.  This is what everyone eventually does anyway.  But if your goal really is to only publish the work you've already written up, that's quite different.  In particular you talk about how you have a very high teaching load (33 credits per year = more than five courses per semester?!?  I hope I got that calculation wrong) and thus don't have time to do research.  Well, you may have zero time for research during the academic year, but if you need two publications for tenure you can spend entire summers working on publishing these papers...and you probably should, if you think your tenure depends on it.  I note though that you said you needed only these two publications; you didn't say that you wanted to stop writing and publishing papers after that.  

Okay, now to examine your reasons:


  They are just not up to shape, because I don't have the time to focus on polishing them.


As I said, I think you should make the time, but that doesn't invalidate the suggestion that a lack of polish is why the papers haven't been accepted as yet.  This is where you should get help from mentors and colleagues and look at the referee reports.  Based on the information given, this explanation sounds a bit unlikely to me: you've already published five papers in a very similar field, so you know what you're doing here.  Math research papers really should not be rejected for lack of polish; they should be rejected when the writing is so bad that the referees can't make sense of them, or when the writing is bad and the results are also not up to snuff.  


  The referees hate me. This may seem silly, but...


Actually you sold that explanation pretty well to me: if you had a terrible falling out with a key person in a tiny field and one other known past referee has told you that you're going in a wrong direction (to weigh in on comments about that: yeah, I think that's a nasty thing to say to someone), then it may well be that your work is getting evaluated by the same small number of people again and again and they just don't like it.  If that is the case, I would again recommend branching out a bit...perhaps even without starting a new project.  Most mathematical work can be cast in various ways and in order to appeal to various people.  I find it hard to believe that there are really on the order of 4 people in the world who are interested, but even if that's true your job is then to interest other people.  You could try for instance rewriting one of your papers to be a lot more explanatory and appeal to a much more general mathematical audience, and then send it to a not-too-exalted generalist journal.  If you do this, there's a very good chance that an editor who is "out of the loop" is not going to send the paper to the same small, sour group of people.


  Chance. The reviewers of the first papers happened to be interested in those topics, and the reviewers of the current papers have not been interested.


Yes, it certainly could just be chance, and in my experience chance plays a distressingly large role in contemporary mathematical publishing.  However, even if it is just chance, you can still probably improve the situation by changing it in some way.

Good luck.