They are not terrified of brown outsiders.  They are terrified of the displacement and marginalization of their cultures values and mores by immigrant groups with radically different values and mores.  Racism has nothing to do with it.  Culture does. There is also a rational fear of international violent salifism for lack of a better word which has proven adapt at inspiring terror in Europe and enforcing speech codes with respect to any discussion of Islam and Islamic belief and theology (as Demonstrated in the Danish Cartoon incident)
I would submit to well meaning intellectuals that if you want to reassure the public than you need to prove that censorship and fear are not the case, and talk about Mohammed and Islam with objectivity instead of phony reverence. Let us see if it is safe out there.