My primary care doc moved to a new office.  She's about half an hour further away from me, but I decided to stay with her anyway.
Yes it is definitely difficult to change doctors.
She's pretty great, I can't think of many people I'd drive that far to see haha
I definitely understand.  My doctor is not super close either.