Can I ask letter of recommendation to my undergraduate thesis supervisor for PhD in other lab I am doing my bachelor's thesis (4th-year 1st semester) in one of the top 5 research group in my field in the world (but not a renowned university). I want to do direct PhD (also called Masters track PhD) after this and my supervisor is extremely impressed by my work and wants to convince me to do it in his group.

He also offered me one more funded semester (4th-year 2nd semester) of research internship which I have accepted.

But, I have friends in some other labs and also, I want to explore some other options like doing a PhD in the top research group as well as a renowned university. Hence, I have decided to apply for direct PhD to some other labs too (top 5 and also university wise top 5 universities). Is it okay to ask my bachelor's thesis supervisor to write a recommendation letter for me? How can I ask him to write a strong letter without disappointing him?
Yes, it's perfectly fine. He should be encouraging you to seek other opportunities as well as staying in his own lab. It take a very jealous and vindictive person to refuse to write a reference for a good student applying to other places. If he is such a person, it would not be a good idea to stay working with him anyway.