Astrophysics after Aerospace Engineering? I'm a second year aerospace engineering undergraduate student. 
I like this course because it might get me to the space industry; but it is not as closely tied to the subject of space sciences as astronomy/astrophysics courses are.

Of course I don't want to abandon my course, so I was wondering how difficult it is to get a graduate degree in astrophysics after I'm done with aerospace. I know I'd be lacking some knowledge but I feel like the trade-off is worth it.

I can consider abandoning my course and going to physics instead, but that's a last resort.

Any advice is highly appreciated. Thank you.
It's doable. Won't be easy, but doable. Get as much background in physics as you can during your undergraduate studies. You probably won't be able to take e.g. Quantum Field Theory, but it doesn't sound like you're aiming for these theoretical branches of astrophysics anyway.
Example from the MIT Physics PhD webpage:

Must I have a degree in physics in order to apply to this graduate program?
Prior to starting their Physics doctoral studies here, our successful applicants generally hold a Bachelor of Science degree in Physics or have taken many Physics classes, if they have majored in another discipline. The most common other majors are astronomy, engineering, mathematics and chemistry. Bachelor of Science degrees may be 3-year or 4-year degrees, depending on the education structure of the country they are in.