Switching into Math/Physics This is going to be long as I believe that the context is important.

I have always been fairly good at maths and physics (won state wide prizes in the math contests and even won a gold medal in the national physics olympiad.) I live in USA/UK/AUS (trying to hide my identity). 

However, after going through grade 12 I got a spot in medicine and happily went along with it because most of my friends were high achievers and I thought why not? I got a fair bit of recognition for getting into medicine (I admit I enjoyed the ego stroking) but luckily medical school is a graduate course (M.D), so I still had undergrad to go. I initially enrolled as a double major - math and physics but within the first week switched to a pre-med track as I thought that it would ease the transition into medicine. 

Here's where things take a turn for the worse. 
By the end of second year I absolutely hate medicine and I realise that I truly do love maths and physics. I had luckily taken a couple of the first year mathematics modules  - multivariate calculus and intro to linear algebra and two 2nd year course as well - intro to DEs and intro to probability. 

So I talk to my course office and beg them to let me transfer into a mathematics track, but I am only able to do courses where I have the prerequisites satisfied  and hence now (end of 3rd year) - have done, in addition to the courses mentioned above, courses on: 
nonlinear dynamics, intro to stochastic processes, a modelling course and a course on systems of coupled dynamical systems. 

However, my true passion had always been physics and not only have I not done any meaningful university physics, but also haven't learnt a lot of mathematics - analysis, algebra and geometry. 

I am thinking of going on to do a masters and try as hard as possible to steer myself back onto the path of physics and mathematics, but I haven't done a lot of the necessary coursework. Furthermore, my family is low income (another factor pushing me into medicine (no parental pressure) but the thought of a good income was tempting) and so I don't think I can afford to take a couple years to essentially redo a bachelor's degree. 

My question is: am I screwed? I am more than willing, and passionate, to learn these subjects by myself - through books. In fact that's how I've learnt most of the things I have to date. But I realise that universities want to see another university giving a student their stamp of approval saying - this student has successfully learnt (insert subject.) I truly believe most of the time these stamps are meaningless, but in the case of admissions it truly is everything. 

Please help, any advice is much appreciated.
Maybe you could head for something a little in between like chemical physics or biophysics or chem engineering where some of the courses to date are useful?
My advice is to push for getting your 4 year degree (assuming US) on time.  Just see bad things when people delay too much from the switching.  If you can switch majors while still getting done on time, fine.  Otherwise, do some masters or the like.  Perhaps after working.
You have the wrong idea about the math needed for physics. Analysis (as opposed to "calculus"), algebra (of the abstract sort), and advanced geometry are all really pretty marginal for someone working on undergrad 'zoics.  Read up on Feynman or just talk to physicists (not math types).  You need to be very strong at calculus, high school algebra, and ODEs/PDES.  That's probably your biggest gap right now--diffyQs.  NOT the more theoretical math courses that people talk about on the Internetz.  
Getting through a "math methods" book would be good.  If you learn this book from start to finish, that will put you in great stead:  https://www.amazon.com/gp/product/B005G14K86/ref=dbs_a_def_rwt_hsch_vapi_tkin_p1_i0 (get a used earlier editions, don't spend $100, I have the 5th.)  Many physicists like Mary Boas (little easier but similar).  The traditional math methods for physics is Arfken Weber, but while it has a few harder topics, it's really a miserable grabbag and doesn't teach well.  I like Kryezing better for self study.)
Finally, I would caution you on the income aspect of physics (or math).  Unless you are a real superstar (which can be rare), you will face huge competition for academic jobs.  These forums are littered with people struggling to find jobs or unhappy with their advisor situation (and the relative subservience required in Ph.D. relationship).  Getting a certified job like a physician is a great move financially and in terms of societal prestige.