he was not without justification nervous of the general crowd of refugees from the cape , who regarded the boys with simmering anger as representatives of the `` kaffirs '' they blamed for the destruction of their homes .