Is it considered rude to address a PhD holder as sir or miss? I suspect I might have offended some of my professors/lecturers and tutors by calling them sir or miss despite them holding a PhD.

Edit (info provided in comment): I'm from Malaysia. We have our own variation of English called Manglish (Malaysian English). So ma'am instead of miss then for female college educators?
I'm a male PhD holder from a private university in Malaysia. This answer is based on my personal experiences.

It is very common for me and my male colleagues to be addressed as 'Sir', and I have not personally come across any (local) academic staff for whom this has been a problem. Generally, 'Sir' is considered suitably respectful, especially as it's pretty often used as a translation for the Malay 'Tuan' or various Chinese dialect honorifics which could alternatively be translated to 'teacher' or 'professor'.

This is slightly different in universities with significant foreign teaching staff (I also have experience with 2 such examples), where some staff take more notice of these titles, but even in those situations I've not yet seen any academic staff take offense. It's easy enough to introduce ourselves to students with something like "Hi, my name is So-and-so and I'd like to be addressed as Dr. So".

That being said, please do not use 'Miss'. That's only partially acceptable in very Chinese-centric institutions, and not at all acceptable outside those. 'Ma'am' is much better, even though not as well known locally.

Finally, considering our local culture, there's probably a hard age limit above which you'd wish to be very careful how you address a teaching staff. Generally speaking once the academic staff is mid-40s or older (or old enough to be your parent) you should be more careful in including titles. The younger staff just starting out generally do not mind.