I'm admitted to a PhD program in mathematics, but only achieved a pass (no distinction) in masters. Will this affect my reputation in the PhD program? I'm a maths student who did his BSc at a top 10 university in England and got solid first. 
After this, I decided to go to Cambridge to do Part III of the Mathematical tripos (a Masters). This was a HUGE step up for me and I felt like a fish out of water. It didn't help that I had personal problems going on in my life around the time of exams, and this ended up having an effect on my results: I got a pass grade.

The next step for me is a PhD and I have already been admitted to a program. My worry is that I don't feel I am good enough to do this. Yes, I got the place but it seems that everyone else I know who's doing a PhD ended up getting a distinction in their masters. 

My question is this: how much does the Masters grade matter for fellow PhD students and supervisors? Are students with 'only' a pass looked down upon?
I am a former 'pass' Part III student. I am going to be blunt here, as I feel emotionally attached to your case and I know why you may feel frustrated (it may be due to the typical atmosphere that part III students are supplied). Part III is just a course that people want to show-off in their CVs (only one final exam at the end of the year, busy professors, for no particular reason unhealthily competitive students). Nobody has ever asked me what my grades were in Part III. Never. I am a Ph.D., did postdocs and am finalizing a good academic job.

All that matters is your hard work in your research. Forget about the scores in Part III. If research is what you want to do in the coming years, just do it without thinking about part III results. You may end up proving that the Part III degree doesn't make any difference in anyone's life for better or worse!