3 years into my phd program, want to shift to theoretical physics I joined a phd in electrical engg 2 years back. But requiring physics, I did log of physics courses and I have now developed a deep interest in mathematical physics and condensed matter. But that has lead me to undergoing coursework in maths dept as i had already finished lot of basic/masters level courses in physics and which gave me a feeling that doing math courses thoroughly first is the right way to do any physics. But my supervisor now is terribly disappointed with me. Also I am confused about shifting university as I am already 2years + into a graduate program. Also there is a feeling that once am through the coursework , I might find some problem interesting to electrical engineers that I might solve with the new skills. But a few professors and my current supervisor are discouraging me saying it is a vague plan. So what should I do, any suggestions?
You can find many areas in microelectronics which are closely related to the condensed matter physics (i.e: Electronic band structure, semiconductor, Conductors,  Superconductor, Ferroelectric, etc). My recommendation is to find a specific field in condensed matter that is also interesting for your supervisors, and pursue your PhD.

In your case, moving to another department is just burning two years.