`` the people inside out-number us just now , and we do n't want to give them any reason to come swarming out of that place -and if you pull a man 's walls down , it usually makes him very grouchy . ''