Agreed.  And theoretically there is a slippery slope here.   The Bible also says that I'm married women are not to have children so if one came in wanting a birthday cake for her son as he can refuse that too?  The Bible also says that you need to honor the Sabbath so if somebody from his church wants a cake but they haven't been going in a while if you can he refuse them?   One can imagine a religious justification for just about any form of disagreeable behavior. 

That being said I still prefer to let the invisible hand of the market do it's work.  Who would want a cake from someone that didn't want to serve them?