Returning to academia after long break with new research topics I am currently out of academia but I would like to return to university and start as a 
postdoc. 

It is more than 5 years ago when I finished my PhD in mathematics (algebraic geometry) 
at a very weak British university. Despite my efforts, I was not able to find a postdoc 
position. From the beginning of my PhD, I did not like my research. Furthermore, the area
my PhD thesis belongs to is quite isolated. I made one publication in a journal which
is rather at a medium level.

Now I have a very interesting research topic I can work on together with another 
professor, but in a completely different field. It lies in the area of mathematical 
physics. I must add that there is no financial support for this project, so 
I have to invest a lot of my personal time into it. My questions are: 
Does it make sense to start working on my research topic and later at some point 
(for example, when I have enough results with one publication) start to apply for a 
postdoc position? 
Is it reallistic to get a postdoc position a few years after the PhD but with a research 
topic different from the PhD thesis?

I am concerned because of the folllowing things: my main concern is my age. I am nearly 
40 years old. There are universities which take this issue into consideration although 
officially they would never admit it. In particular, German universities do so. Secondly,
my PhD degree from a weak university does not favor an application for postdoc positions 
not to mention a permanent position. Thirdly, I expect that universities will reject 
postdocs who have such a big gap in their CV.

Here is some background: I did my undergraduate degree in Germany with an excellent grade.
Unfortunately, I could not find a PhD position in Germany, so I applied to some British
universities. At that time, I did not know that the British university system is 
completely different from the German one. So it happened, that I arrived at a weak 
university. I was not able to leave it because I could not get any recommendation letters
to start new applications. Although my PhD advisor knew that I do not fit into this 
university he did not warn me. Moreover, he was telling me a lot of lies to make me stay 
there. It was unbearable. I would have left this university if I could.

After my PhD, he did not help me to find a postdoc position. The only thing that he
did was telling me that there are no postdoc positions and that students who graduated
from strong universities like Cambridge have better chances on the academic market. 
This was very frustrating.

I applied to many universities, even outside Europe but not a single application was 
successful. Even my applications for different kinds of fellowships failed. It was very
difficult (and still is) for me to figure out what are the important things for a 
succesfull application for someone who is in my situation.

Any advice is welcome.
I have been in a very similar situation for quite some time. Sadly, I don't think there is a way out, or rather 'back in' - without the right connections. Unlike me, you seem to have at least one connection, the professor you mention.

You have not given any details of that possible project (and how it would be financed). You should be able to talk to that professor and evaluate the chances for a (later) postdoc position from there. Again, this critically depends on his connections.

If you are really passionate for this research topic, I think you should take a shot. 
If you are OK with not being a researcher, an ordinary job would of course be the safer route.