If you go to any white supremacist websites you'd be surprised that they at least claim not to hate black people or other non whites.  They say (rightly) that no one chooses their race and it wouldn't be logical to hate someone for something that they can't control.  They do believe that blacks are dumb, inferior, more eager to violence and crime and they believe that if the Jews didn't exist to fool everyone into desegregation all those problems would go away.  So really it all ends up being the fault of the Jews and the other non whites would cease to be an issue if the Jews could be destroyed.