PhD or physics as a hobby? I completed my undergraduate degree in physics in a good university (outside the US). Because of financial commitments, I couldn't leave for grad school straight away. So I worked for a couple of years as a research assistant at the same university after I graduated. I did very well, got three papers out in around three years and picked up a master's by research for that. I loved my time working in physics but the research assistant job was a temp job at best and I eventually had to leave it.  

Now I work at a well known engineering company. The job is relaxed and spare time is plenty and I'm being paid well. There are good opportunities for growth because the company is about to expand in a big way. I also love having enough time for hobbies, fitness and just life in general. 

I also applied to PhD programs in the US and was accepted into several good ones. The reason I applied is primarily because I miss physics and I am interested in being really good at core physics. I have this feeling that my education is unfinished in some way. I want to be that guy who has a deep understanding of fundamental physics. That guy who can easily answer questions on the physics stacks. Maybe I'd even like to write a textbook someday. I liked research when I did it but to be honest, I'm not sure I have that same fierce passion for it as I do for learning. In addition to this, I know I'm not cut out for the academic tenure track - Even if I were smart enough (and this is not impostor syndrome or anything, I am self aware enough know that I am good but not great), I have no interest in chasing low paid postdocs and living with the general instability of post PhD life. In five years, when I complete my PhD, I will likely end up in a very similar job to what I have now.

With this in mind, should I simply stay in my regular job and do physics as a side hobby or actually do the PhD? 

TL;DR Is it worth doing a PhD if learning and mastery of physics over research is the primary motivation and sacrifice a great career for it?
One possible way is to do both, in a way.

To help explain, I am drawing on my own current experience -  I earned my PhD in Physics back in 2014, but I still work in a different field. However, I continue research in Physics (and am still getting published) as an Adjunct researcher to the university where I completed the PhD. (Of course, you should check with the university if it is possible to be an Adjunct).

Completing a PhD can provide a foundation for you to continue research in an area of interest as an Adjunct (especially if you published while completing the PhD). Analytical and research skills developed in the PhD could potentially greatly benefit your work in your job (as I have found in my situation).

As in your case, my job allows a considerable amount of spare time, where I have found plenty of time to continue to do research, exercise and engage in my hobbies in my time.

Depending on your time constraints, there is no reason to give up either.