Ironically, it's extremely ok to make this association at all. Immigrants get called lots of stuff (rapists, criminals, scum, etc), but if I said "don't say criminal, it might be insensitive to immigrants", I would hopefully be cheered because it's a fitting