Life as a financially independent academic I hope this is not to much rambling, but there is a concrete question in the end. I really like being a mathematician. I like the constant learning, I like working on problems at research-level and I even like teaching university students. I did my PhD and definitely had fun, but during the time, I have also seen aspects of academia that I do not like, most of which fall under "publish or perish", namely the constant struggle for grants, publications and the all-elusive tenured positions.

I'm obviously not alone in this, but let's assume that I have some money, not enough to be overly rich, but enough to keep a modest standard of living without ever having to work. (The details of this are not part of the question, just assume that salary will not be much of a job-motivation.)

Now this got me thinking. I would never want to "retire" at an early age, but on the hypothetical, what if I do not join the fight for tenure and instead effectively tenure myself?

I know that there are many people without university affiliation, who occasionally publish some papers, but are there any who do this full time?   There would be enough time to talk with collaborators, write articles I am actually interested in, visit guest lectures at the local universities and  the occasional conference on a holiday. Apart from teaching, I would do what everyone else is doing, just on my own. But would it work the same? A possible doubt is, would many people even be interested in working with me? I'm a reasonably competent mathematician but would probably be ineligible for many sources of funding, couldn't offer a big name or any return visits, and would be less motivated to get joint work published in the best possible journal. To put the question succinct:

Is life as a full-time "independent researcher" feasible?

What are other pros and cons? Are there precedents (apart from bored 18th century noblemen)?
I think this is completely feasible for a mathematician and maybe several other fields. No academic affiliation is needed to attend academic conferences or to publish papers. What you need is some means of collaboration to help firm up your ideas, nothing more, really. For that you need to meet people and to exchange ideas. 

What would be frowned on, however, is replacing a paid faculty member with a volunteer. Most people do need that paycheck and it seems somehow wrong to deny someone that opportunity. You aren't suggesting that, of course, so I see no issue. 

The easy way to achieve it, assuming that you live near to some large university (preferably) with a large faculty, is just to go and introduce yourself. You might need an invitation first, arranged by email, campus security being what it is today. If you can interest a member of the faculty (or a few) to talk to you about common interests you could probably get a standing invitation to participate in research seminars and such. If your ideas are sound, people will be happy to work with you. 

Since internet communication is easy these days you could even use a conference to meet people and then work remotely. Or use your Alma Mater as a source of contacts. 

One possible negative, however, is the level of commitment you are willing to make. If you are working on joint research, you need to do your part so that others aren't let down. I don't see that as a big issue in mathematics, but it might be in some fields.