That's the thing about memories of the old days - they're often not as true about the way things were for everybody as we recall.

Of course, it depends on just how old you are, but by the early 60s, that "I don't notice skin colour' thing was already dated and widely recognized as dismissing people's roots and life experiences.  At its worst, it was the thinking behind residential schools. 

The belief that it's important to celebrate our common humanity AND our differences has been around for quite a few decades now.