Is my advisor right, am I a turn-off? I'm doing a Phd in math. Unfortunately it didn't go very smoothly. I'm in the fifth year now, and a Phd is not in sight. 

This is due to several things, personal problems (I suffered from a depression for several years), but also my advisor and me just not getting along very well. 

I know I should have left him years ago, and I tried, but I'm still stuck in this situation. That's my background. 

Now today I had a depressing conversation with my advisor. He offered me to continue my Phd with him, but at the same time he also said it would be "highly unlikey" that another advisor would accept someone like me with this difficult background, as there are "plenty of talented people". 

So I would like to know how other people (preferably people who advise) would rate my situation. Is my advisor right? Is it too late for me to change? Is my background such a turn-off?
It sounds like your advisor is right, and your situation is a turn-off.  He wants you to succeed to improve his own track record, as he is at least by reputation partially responsible for your success or failure.  And other potential advisors don't have that motivation. 

Since you're both in this together, and since you have so much to lose relative to him, I think you need to come to a plan together, and you need to listen to him very carefully and be as cooperative as you can muster yourself to be. He wants you to complete. He wouldn't have had that blunt conversation if he just wanted you to flounder. You need to now cast the past into the past and move forward with whatever you choose to do.