This is my opinion, I don't think that Mecca generally or the Kaaba in particular should really be restricted to Muslims because it suggests they have power. But there's a big difference between a religious organization deliberately and willingly letting tourists in and one which doesn't want them to come in. If you don't want tourists poking around Notre Dame or St Peter's during Mass,tell the Church you don't want them to be let in