The counterproductivity from the use of a symbol seems to depend on the motives of the user. If the motive is to upset a lot of people, then a Confederate flag might be working. If the purpose is to make a racist statement, then it might be working. If the purpose is to say, "I'm born and raised in the South," then it might be counterproductive.
As to the swastika, I've seen it displayed numerous times but in a context that indicates it's not a Nazi symbol and would be thusly interpreted by most people.
In any case, if I were out and about and saw someone waving a Confederate flag, if the situation allowed, I think I would engage the individual in conversation and try to find out exactly what it was they were attempting to convey by their actions rather than assume they were purposely being offensive.