Some things merit a knee-jerk reaction. Like using the N word, or a Heil Hitler salute. Doesn't require much thought to know it's offensive. I don't think "colonial" is quite as offensive as those, and yea, if you have a building with white pillars, and it's called Colonial Apartments, and that's the extent of any references to colonialism, maybe that's not such a big deal. But throwing in "plantation" on a drink menu, Lloyds of London (they made some of their wealth insuring slave cargoes) and Churchill brunch menu, and it starts to reek of "ah, the old days." Even if you "didn't mean it that way" you have to see that it can easily be taken that way and change it!