"The money isn't considered yours in the first place."
Sort of Correct. Every argument i have ever seen that people say "don't you dare touch SSI", they are saying it is the earned money of the workers - which they are correct. But you are also correct that once the government lays their hands on it, it is really no longer yours. And that is the problem - it should be ! You earned it !  You should be able to take what you earned and pass it to your family or heirs. That is the entire arguing point of the republicans. They want to (eventually) dump SSI for real retirement accounts that YOU own and control.
http://www.ontheissues.org/celeb/Republican_Party_Social_Security.htm

Here is a chart to describe and show how SSI has been working
https://www.mercatus.org/publication/how-many-workers-support-one-social-security-retiree
And as Nancy pointed out, we have yet to see the baby boomers all retire, further inverting the ratio.
This is why it will eventually fail.