"Saffron Colonial" is not an "overt expresson of racism" anymore than the "Victorian" homes that dot the neighborhood are overt expressions of sexual repression.  It's a word that denotes a time and a place, nothing more.