If you watch it carefully, the big guy makes a strategic withdrawal after the two black guys start standing up to him.  He realized that the whole shock and awe thing had worn off and it was time to go.

Intimidation only works as long as the other party lets it. 

edit: it's obvious that the argument had  got to the point of either fighting or walking, he walked which was a smart move.