But the passage which angered him most was that which told him that he ought to have taken his family back to town. 
But he was angry at being told that he ought to have taken his family back to town. 