Lack of team support in post-PhD Mathematics career I received my PhD in mathematics about ten years ago. Despite my preference,  desire, and attempts, I was not offered any postdoctoral positions. Part of the reason was (I think) that my dissertation topic was outdated, as opposed to being trendy. Since then, I have accepted a couple of tenure-track positions in teaching-oriented institutions, the second at a two-year Community College and in two years from now I will most probably be tenured. However, I constantly struggle with existential questions. I am not where I wanted to be. Granted, I have a job, but if my goal was to have a job, I was totally capable of aiming at a higher-paying job, instead of mathematics, when I was deciding what to study at college. As a teenager what intrigued me about mathematics (and still does) was its beauty and the opportunity it provided me for being creative through research and continuous learning. However, since my PhD the processes of research and learning have been very slow, despite my continuous attempts. For a long time I used to think the reason was lack of time due to heavy teaching load. While that may have contributed to the problem, I now think a bigger reason is lack of team support. There is a disconnect between my job and my research. I don't have people (graduate students, researchers, etc.) around me to whom I can talk about my research, everyday or every week. Consequently, the ratio of outcome to the time that I spend on research and learning new things is very small. So what are my questions?


Ideally I would have liked to be part of a team, a lab if you like, consisting of one or two senior mathematicians (lab leaders), research assistants (mathematicians like myself who are not young enough to be postdocs, but still need to develop their research plans), postdocs, and graduate students. My question is, does such a model exist anywhere?
Are there any options for mathematicians who really are not after a secure job and prefer to work as research assistants (paid less), rather than tenured "calculus teaching" faculty who do not meet age requirements to be considered for postdoctoral positions?
Suppose someone (e.g., myself) didn't know better and did his dissertation on a topic that was not trendy and because of that couldn't secure a postdoctoral position. Is that the end of their career as a research mathematician? Are there any other options? What would you suggest?
This model is extremely common in Europe (for example, one of my coauthors in Germany is in a group with her and another senior professor, two junior mathematicians on essentially very long term second postdocs, and about 7 or 8 young postdocs and 7 or 8 graduate students), but there is no institutional framework for it in the US.  I know of pairs of tenure-line faculty at the same institution who are close enough to essentially run a joint research group, but outside of a few departments, they are unlikely to have more than one postdoc at a time, so "research group" mostly means them and their graduate students.
Not really.  Essentially postdoctoral positions are the only mechanism that exists in the US system for paying people who are not already performing at a pretty high level to do research.  You could still apply to these, though as I think you've already guessed, the odds aren't great this long post-Ph.D.  I think the other possibility would be obtaining a long term, non-tenure-track lectureship at a research university (for example, my department hired someone to such a job this year) where there are senior faculty who might work with you.  There would still be high teaching expectations (though maybe not worse than a TT community college position), and any research involvement would be "on your own time," but presumably that's largely the case now.
I suppose this depends on how you define "career as a research mathematician."  The answer is obviously not "absolutely no," look at Yitang Zhang.  Of course, that's a very extreme example (Zhang didn't even have a teaching position in mathematics for several years), but I think it does show that if you do something undeniably great, at whatever age, the community is open to it.  On the other hand, this is like asking "If I can't get on a Division I college team, does that mean my career as a basketball player is over?"   Obviously, no one can stop you from playing basketball, but if you mean doing so professionally, the odds are slim. If you're really that good, then maybe an NBA team will notice and hire you.  But you're not going to be in the place that they look for potential new players, and you'll be missing out on how most people develop their skills.  So it's a tough row to hoe.  


I think in terms of what's realistic: if what you're missing is being around other people doing mathematics, getting ideas, etc. and you're not worried about security then I would look into finding a permanent (or at least renewable) but non-tenure-track teaching position at a larger research department.  It will certainly have some downsides, but I think it's the closest feasible approximation to what you're looking for.