and then we should have to get back , '' he added , and sighed a small sigh ; he ought have known that it could be no easy matter to build up a fortune , or else everyone would have done it already before now .