How can I tell a professor that I don't want to join his group? (science major) I am a incoming first year grad student this fall. I have contacted prof A during when I was applying and talked to him during the visit. He strongly encouraged me to join his group.

After the visit, I emailed students in Prof A's group and other current grad students. I started feeling that his group atmosphere is not the best for me, and I don't want to join him anymore. 

He is really nice and recently offered me an educational opportunity that is only shared by current students. I am grateful and I really want to take that course.

I feel guilty right now b/c he is so nice to me. But I don't want to spend my next 5 years in a group I don't like. 

Is there any way that I can let the prof know I am not interested in his group anymore? Do I have to give up the course he offered? I am not assigned to prof A so I have the freedom to choose.
It sounds like you have not made any sort of commitment, so if that is the case then there should be absolutely no hard feelings by simply writing a polite email saying that you have decided not to do research with him.

If you wish to elucidate the reason, just say you don't feel that you'd fit in well with the group. He'll understand. You should not feel obligated to come up with an elaborate explanation.

It's hard to tell without more information on the course you speak of. If it's an ad-hoc, informal course taught by the professor himself then I can see how the investment of time may only be worthwhile for his own students. If it's a proper course that you pay fees for and earn credits, then perhaps it is reasonable to ask if you can still do it.