It's used for people who are born here but not indigenous.  Being settled is the opposite of visiting.

If you don't like settler, then come up with another word.

I don't have a problem being called settler, and I don't see what's derogatory about it.