How to tell my MS advisor I've changed my mind about doing a PhD with him? I did pretty much all my important bachelor and master courses within one working group, which maybe was not that efficient, but it offers a very nice and comfortable environment and colleagues.

Currently I am about half way done with my masters thesis in a similar working group within my university, which I had not much contact with before. They work very efficiently, and I am catching up and learning quickly, yet I do not feel too comfortable around them. Both working groups basically do very similar work, but use different programs and approaches.

I'd like to point out that there is no conflict because of me "switching groups" and there are no shared projects between them.

Some time ago, I got offered a PhD position in my new working group, and my supervisor encourages me to get my master thesis done even more quickly, so I can fill his PhD position. I signalized him that I would be interested and ever since then he is very convinced that I will stay. On the other hand, I also know about possible upcoming positions in my old group as well as in other groups/places, which look very intesting to me. I fear that my masters thesis quality is suffering, which would probably make it harder for me to apply for other positions.

I feel hardly pressured to tell him that I am hesitating to accept his offer and seriously thinking about going back to my old group or somewhere else, but I don't dare to do so. He has no idea about this.

The question is: what is an appropriate way to tell him, and is there a good timing for this? Am I putting his guidance for my master thesis at risk here?
Two points: 

First, just because your new supervisor offers you a PhD position does not mean that you are in any way obliged to take it. That said, don't hide the fact that you are going to consider sending applications out to a variety of programs. The thing that will cause hard feelings is if your advisor thinks you are going to join his or her group and turns other people away, only to have you leave. If your supervisor is a professional, he or she will recognize that you pursuing other opportunities is not personal. You have to do the best you can for your career, and if that means taking a better position elsewhere, then you should do so. (Note that you can also get outside offers and use these to negotiate better pay as a PhD student, but that's a different issue, and somewhat more complicated politically.)

Second, don't under any circumstances work for a group that makes you be slow and do poor work. Academic hiring is cutthroat competitive, so if you want to actually get a job, you're going to have to produce a lot of very good work, and you're only going to do that if you're surrounded by other people who are trying to do the same who push one another.