Is it worth it for me to try at this point? Basically, I came into this university on a scholarship, doing their most prestigious undergraduate degree. For a variety of reasons including but not limited to depression and anxiety, I basically failed half my courses in my first year. Last semester, I took a 6 month break.

Right now, I have been kicked from that degree and have lost my scholarship, but I am really enjoying my schoolwork again. A lot of the problems that were plaguing me the past 1.5 years have been largely resolved. I have missed struggling with a difficult problem and/or the feeling I get when I finally understand something. Originally my passion was space and wanted to do something in that area (such as astrophysics). It was (maybe still is) a dream of mine to go into this field at a higher level after undergrad. When I came to university, I was ambitious and wanted to get into some Ivy League school for graduate school, or some top school for whichever field I chose.

I had lost that dream, but now I have found it again. I was wondering if it was worth pursuing and putting all my effort into it, or is it too late for me already with my terrible background?
I think this is a simple answer.

Stop thinking and start doing.

If you want a graduate degree at an Ivy League school and a successful academic career (perhaps even in the field of astrophysics), then pull your socks up, set definite, measurable and realistic goals and start achieving them. The path will not be easy but always remember that you are not the first and neither will you be the last.

Note that a graduate degree at an Ivy League school and a successful academic career are not necessarily mutually inclusive. To do one does not mean that you will do the other and neither does it mean that there is a causal relationship between the two.

Personal anecdote:

I had a rather forgettable start to my academic career as well. I recovered enough to get myself admitted to an "Ivy League" graduate program. I am still on my journey and I dare say the destination is years away. Not everyone is a rising star. Some folks are late bloomers. 

I reiterate again. To achieve your dreams, just keep trying sincerely and honestly. It will happen in due time.