Some context according to only half the parties involved. 

It seems pretty apparent, based on the "Chinaman" comment, that the old guy started the whole thing by being a racist. Amber lamps certainly initiated the physicality, but lets not praise an old racist for his patience simply because he didn't swing first.